Experiment Complete Published
MLX LM 0.32.2 on M5 Max · 64 GiB, Metal
LFM2.5 2.6B as mlx-affine-4bit-g64 (4-bit), archive 2048 input / 128 output: a 2048-token prompt and 128 tokens out. Measured 12 Nov 2026, 23:14 UTC by Glenn Sonna, Xybrid.
Doesn’t count in rankings: Its output check failed: 7/9 of the checks the model passes at BF16; needs 8 and the mandatory ones (needle-start, continuation-128). Left out, failed by every BF16 configuration: color, sort, needle-end.
Experiment id s3-lfm2.5-2.6b-4-bit-mlx-2048
Summary
Over 6 valid trials of 6 attempted (6 planned), computed from the raw trials below.
| Metric | Median | Range | Mean ± SD | Quartiles |
|---|---|---|---|---|
| Decode speed tok/s, higher is better | 258.4 | 257.5–259.3 | 258.3 ± 0.6 | 258.1–258.5 |
| Time to first token ms, lower is better | 208 | 208–211 | 209 ± 1.1 | 208–210 |
| Peak memory GiB, lower is better | 2.98 | 2.93–3.03 | 2.99 ± 0.04 | 2.98–3.02 |
| Prefill speed tok/s, higher is better | 9,858 | 9,759–9,883 | 9,835 ± 49.0 | 9,806–9,864 |
Trials, metric by metric
The line runs from the lowest to the highest valid trial, the dot is the median and each tick is one trial.
Every trial
Raw token counts and durations, with the speeds computed from them. Invalid trials stay in the record with their reason; they are left out of the summary, never silently dropped.
| Trial | Time to first token ms | Prefill tok/s | Decode tok/s | Peak memory GiB | Thermal | Note |
|---|---|---|---|---|---|---|
| 1 | 210 | 9,7902,048 tok in 209 ms, engine | 258.4127 tok in 491 ms, engine | 2.93 | nominal → nominal | |
| 2 | 211 | 9,7592,048 tok in 210 ms, engine | 259.3127 tok in 490 ms, engine | 3.03 | nominal → nominal | |
| 3 | 208 | 9,8652,048 tok in 208 ms, engine | 258.3127 tok in 492 ms, engine | 2.98 | nominal → nominal | |
| 4 | 208 | 9,8832,048 tok in 207 ms, engine | 258.0127 tok in 492 ms, engine | 2.98 | nominal → nominal | |
| 5 | 208 | 9,8632,048 tok in 208 ms, engine | 257.5127 tok in 493 ms, engine | 2.99 | nominal → nominal | |
| 6 | 209 | 9,8522,048 tok in 208 ms, engine | 258.5127 tok in 491 ms, engine | 3.03 | nominal → nominal |
Caveats
- Publisher cards identify the base model but do not always pin the exact export source checkpoint revision.
- Engines may differ in quantization scheme, effective KV cache precision, kernels, and activations.
Configuration
Device
- Device
- Apple Mac17,6
- Cohort
- Apple silicon Macs (desktop)
- SoC
- Apple M5 Max
- CPU
- 18-core Apple CPU
- GPU
- Not recorded
- NPU
- Not recorded
- RAM
- 64.00 GiB
- OS
- macOS 27.2.0 (26B123)
- Kernel
- 27.2.0
- Power
- mains
- Low power mode
- No
- Performance mode
- Not recorded
Engine
- Engine
- MLX LM
- Maintainer
- Apple (ml-explore)
- License
- MIT, open source
- Version
- 0.32.2
- Commit
- Not recorded
- Binary SHA-256
- 9d46803799f99d6fc7c35fff2d4357c79b1a25ae98cc63efeaaebeb50a7b5f33
- Built from
- package
- Build flags
- None
- Backend
- Metal
- Threads
- Not recorded
- Interface
- python-worker
- Settings in effect
- backend=Device(gpu, 0) prompt_cache=false speculative=false greedy=true temperature=0 sampler=argmax prefill_step_size=2048 context_length=4096 kv_quantized=false batch_size=1 trust_remote_code=false weight_bits=4 weight_group_size=64
Model and weights
- Base model
- LFM2.5 2.6B (LFM2.5, 2.6B)
- Upstream
- LiquidAI/LFM2.5-2.6B@7141b56cd4650af95c7c7b86983443aed050c96a
- Weights
- LiquidAI/LFM2.5-2.6B-MLX-4bit@04efa23776ce61ec34ec95ec34c859854c89542b
- Format
- safetensors
- File 1
- README.md · 2,648 bytes · sha256 0907976d22f77d4178e10d04002403a94a22ad965ad8431f754fb28a14885f6e
- File 2
- chat_template.jinja · 5,496 bytes · sha256 aadd425edb1d42fe9288e5de3624c21c1c03099129b9cfea5978f1bc67222c0a
- File 3
- config.json · 2,202 bytes · sha256 6ab5ad71687f6ca8d3811b9290ac06291f78e1e0f4f3ec504e6644032d58342b
- File 4
- generation_config.json · 306 bytes · sha256 e08c9a83f4bfe64468c9671a2b69b60549728e2a9481387f3555471f64e958bc
- File 5
- model.safetensors · 1.47 GiB · sha256 ef350b75815752f8bff8c064c9fc8bd8b5709577d72943436852013d2bcd4691
- File 6
- model.safetensors.index.json · 41,983 bytes · sha256 620f8efaf53e826757324852e999f6ba9a5d974e193589470a70951fd17a1543
- File 7
- tokenizer.json · 17.1 MiB · sha256 695be7802a0e4b8a81048f0ff5ebb7fc811a0ba5a6be63dbb24deb5a81096f41
- File 8
- tokenizer_config.json · 363 bytes · sha256 11f1de897317b489dd09199284528382eeb17f566398b16e2039bb2266d26b09
- Quantization
- mlx-affine-4bit-g64 (4-bit)
- Weight bits
- 4
- Group size
- 64
- KV cache
- model-dtype
- Conversion
- Published upstream by LiquidAI
- Conversion command
- Not recorded
Workload
- Workload
- Archive 2048 input / 128 output
- Prompt
- 2,048 tokens, from prompts/source.txt
- Prompt SHA-256
- c10d1b7bef62d95322f8f8d229dc66118d9375107bdc49b06f76287d058b2f6c
- Chat template
- ea663864491de7ade391839479860ca95541f892f72665c73251fbd4643b1bef
- Output
- 128 tokens, ignoring end of sequence
- Context length
- 4,096
- Batch size
- 1
- Sampling
- Greedy
- Timing
- One streamed request per trial
- Warmup
- 2 runs, not counted
- Trials planned
- 6
- Cooldown
- 15 s between trials
Conditions
- Ambient
- Not recorded
- Thermal at start
- nominal
- Thermal at end
- nominal
- Thermal gate
- macOS thermal state nominal
- Throttling detected
- No
Provenance
- Measured by
- Glenn Sonna, Xybrid
- Operator affiliation
- Xybrid
- Run at
- 12 Nov 2026, 23:14 UTC
- Harness
- xybrid-bench 0.4.0 (bde6325f3e1d617ca13cbb1d008164ccc9b7b7c3)
- Methodology
- v0.6
- Publication
- published
- Published
- 16 Nov 2026, 10:23 UTC
- Reviewed by
- Glenn Sonna
Output check
- Result
- Failed
- Method
- sanity-v1-bf16-reference
- Reference
- checks/sanity-v1.json
- Reference SHA-256
- a702dee079525ae0ec90465b1c9e26052c99cb3e1c53e83dea05be4c194fc7ca
- Required passing checks
- 8
- Check arithmetic
- Passed · sha256 90a647b2379fe43265d36db39c01e0d324059b2e48f24ec4e2c1f6ec1c0d21eb
- Check capital
- Passed · sha256 b1098d691672b55e98767ac84c6d8375f7bcc0d78154c861ea70ac2ff66e0ba7
- Check opposite
- Passed · sha256 69a13e1ba8e284fcc26068ec8aa80e8a7e7f8bbebe86dc7a45424411198a6859
- Check weekday
- Passed · sha256 b8c6a621f6e216db1d214b8d906c759be744ff3abc6e41f979ff0d24575cbbcd
- Check color
- Failed · sha256 fe08d89d0b006dfadf3ef0dfdad25eded8e6238dcd5af7ea93748aaec2d565fd
- Check sort
- Failed · sha256 cb12005007583cdcd569a7d96955cb80cb294053fa0ac7cedb3ee401fa09d13b
- Check animal
- Passed · sha256 3e9114b6df29561433c18b3a6abce2c635fbd8e9ff25b5fcee93149bfea52859
- Check word
- Passed · sha256 2e1816dd952a0d4c672ad52a4f32599d4eb0477bf5798060b93fe68f7684dbbd
- Check copy
- Failed · sha256 40ec15ba7f6dd791515c1e570997f3fd45822d5f2c56544e7b35cb2e6fb16729
- Check needle-start
- Passed · sha256 8b9af2e5b8b4cfddb87d171997b95c471f8d68a68e30fb5177abeb3332372edd
- Check needle-end
- Passed · sha256 e6589c736af32f81bf239c5eda6de41959122ccc1d05463b75e0eb0c70fc831c
- Check continuation-128
- Failed · sha256 595ee2e62bb2b0afabce45fc4d707def46db7e116ff6d218da9f9a33e57c4274
- Detail
- 7/9 of the checks the model passes at BF16; needs 8 and the mandatory ones (needle-start, continuation-128). Left out, failed by every BF16 configuration: color, sort, needle-end.
Reproduce it
pnpm bench run --plan plans/s3-lfm2.5-2.6b-4-bit.json
Three palindromic blocks per quantization class and workload; load, two warmups, one trial, unload per turn.
Raw data
- This record, as committed JSON, schema v0.3.0
- Complete finalized session evidence raw ·
sha256 d84c658e7b83…