Experiment Complete Published
LiteRT-LM 0.19.1 on M5 Max · 64 GiB, CPU
LFM2.5 2.6B as litert-int8 (8-bit), archive 256 input / 128 output: a 256-token prompt and 128 tokens out. Measured 12 Nov 2026, 22:29 UTC by Glenn Sonna, Xybrid.
Doesn’t count in rankings: Its output check failed: 7/9 of the checks the model passes at BF16; needs 8 and the mandatory ones (needle-start, continuation-128). Left out, failed by every BF16 configuration: color, sort, needle-end.
Experiment id s3-lfm2.5-2.6b-8-bit-litert-256
Summary
Over 6 valid trials of 6 attempted (6 planned), computed from the raw trials below.
| Metric | Median | Range | Mean ± SD | Quartiles |
|---|---|---|---|---|
| Decode speed tok/s, higher is better | 60.3 | 60.0–60.5 | 60.3 ± 0.2 | 60.1–60.4 |
| Time to first token ms, lower is better | 541 | 539–542 | 541 ± 1.2 | 540–541 |
| Peak memory GiB, lower is better | 3.42 | 3.42–3.42 | 3.42 ± 0.00 | 3.42–3.42 |
| Prefill speed tok/s, higher is better | 515.9 | 514.4–517.4 | 515.9 ± 1.1 | 515.3–516.7 |
Trials, metric by metric
The line runs from the lowest to the highest valid trial, the dot is the median and each tick is one trial.
Every trial
Raw token counts and durations, with the speeds computed from them. Invalid trials stay in the record with their reason; they are left out of the summary, never silently dropped.
| Trial | Time to first token ms | Prefill tok/s | Decode tok/s | Peak memory GiB | Thermal | Note |
|---|---|---|---|---|---|---|
| 1 | 541 | 515.3256 tok in 497 ms, engine | 60.3127 tok in 2,105 ms, engine | 3.42 | nominal → nominal | |
| 2 | 540 | 516.8256 tok in 495 ms, engine | 60.0127 tok in 2,116 ms, engine | 3.42 | nominal → nominal | |
| 3 | 540 | 516.4256 tok in 496 ms, engine | 60.4127 tok in 2,104 ms, engine | 3.42 | nominal → nominal | |
| 4 | 539 | 517.4256 tok in 495 ms, engine | 60.5127 tok in 2,099 ms, engine | 3.42 | nominal → nominal | |
| 5 | 542 | 514.4256 tok in 498 ms, engine | 60.5127 tok in 2,101 ms, engine | 3.42 | nominal → nominal | |
| 6 | 541 | 515.4256 tok in 497 ms, engine | 60.0127 tok in 2,117 ms, engine | 3.42 | nominal → nominal |
Caveats
- Publisher cards identify the base model but do not always pin the exact export source checkpoint revision.
- Engines may differ in quantization scheme, effective KV cache precision, kernels, and activations.
Configuration
Device
- Device
- Apple Mac17,6
- Cohort
- Apple silicon Macs (desktop)
- SoC
- Apple M5 Max
- CPU
- 18-core Apple CPU
- GPU
- Not recorded
- NPU
- Not recorded
- RAM
- 64.00 GiB
- OS
- macOS 27.2.0 (26B123)
- Kernel
- 27.2.0
- Power
- mains
- Low power mode
- No
- Performance mode
- Not recorded
Engine
- Engine
- LiteRT-LM
- Maintainer
- Google AI Edge
- License
- Apache-2.0, open source
- Version
- 0.19.1
- Commit
- Not recorded
- Binary SHA-256
- c61e0faa71026f2875a710a42cbe803226004db811f7ee47ea78198a0499e851
- Built from
- package
- Build flags
- None
- Backend
- CPU
- Threads
- Not recorded
- Interface
- python-worker
- Settings in effect
- backend=cpu max_num_tokens=4096 enable_benchmark=true chat_template=pass-through process=one per generation sampler=top_k=1 temperature=0 stop_token_ids=[[124900], [124895]] cache_dir=[runner]/engines/litert-lm/cache bos_token_id=124894
Model and weights
- Base model
- LFM2.5 2.6B (LFM2.5, 2.6B)
- Upstream
- LiquidAI/LFM2.5-2.6B@7141b56cd4650af95c7c7b86983443aed050c96a
- Weights
- litert-community/LFM2.5-2.6B@8f87ecede9509fd8d12c3e5b8c46b04b597febfc
- Format
- litertlm
- File 1
- LFM2.5-2.6B_int8.litertlm · 2.67 GiB · sha256 02e97ce3a4de444f1b0bbca4494827704f86212bf72c201703e564d96068536a
- File 2
- README.md · 18,500 bytes · sha256 ad39356f0ce401994fed4791f08a33353db1c6e078358f000d3ddcc8ff756bbb
- Quantization
- litert-int8 (8-bit)
- Weight bits
- 8
- Group size
- Not recorded
- KV cache
- litert-lm default
- Conversion
- Published upstream by litert-community
- Conversion command
- Not recorded
Workload
- Workload
- Archive 256 input / 128 output
- Prompt
- 256 tokens, from prompts/source.txt
- Prompt SHA-256
- c10d1b7bef62d95322f8f8d229dc66118d9375107bdc49b06f76287d058b2f6c
- Chat template
- ea663864491de7ade391839479860ca95541f892f72665c73251fbd4643b1bef
- Output
- 128 tokens, ignoring end of sequence
- Context length
- 4,096
- Batch size
- 1
- Sampling
- Greedy
- Timing
- One streamed request per trial
- Warmup
- 2 runs, not counted
- Trials planned
- 6
- Cooldown
- 15 s between trials
Conditions
- Ambient
- Not recorded
- Thermal at start
- nominal
- Thermal at end
- nominal
- Thermal gate
- macOS thermal state nominal
- Throttling detected
- No
Provenance
- Measured by
- Glenn Sonna, Xybrid
- Operator affiliation
- Xybrid
- Run at
- 12 Nov 2026, 22:29 UTC
- Harness
- xybrid-bench 0.4.0 (777cb3d9fecaa7cc1a30694d98492c33dbc6d693)
- Methodology
- v0.6
- Publication
- published
- Published
- 16 Nov 2026, 10:23 UTC
- Reviewed by
- Glenn Sonna
Output check
- Result
- Failed
- Method
- sanity-v1-bf16-reference
- Reference
- checks/sanity-v1.json
- Reference SHA-256
- a702dee079525ae0ec90465b1c9e26052c99cb3e1c53e83dea05be4c194fc7ca
- Required passing checks
- 8
- Check arithmetic
- Failed · sha256 c0b2a5a0debecb2e388c47cdc10113fcd481f224c0efac592a60e4dd102bed56
- Check capital
- Passed · sha256 7a3bb1905f2039638dff3f8a3da57abf7b1da32596b04c5ed04c5296f155196a
- Check opposite
- Passed · sha256 7c90b06eef25984adf77ec226311829bbc1065a033e9ad6b26d9bd0dfdc65f95
- Check weekday
- Passed · sha256 b8c6a621f6e216db1d214b8d906c759be744ff3abc6e41f979ff0d24575cbbcd
- Check color
- Failed · sha256 3cde5b4f191c8cd46ee0e9a89b6a7cf42c5cfe733bbf8158ee61992779919dc0
- Check sort
- Failed · sha256 b9eeb858235b7c5e8af46f7bf38023806dc0df277f67e79c8fdfbe600fc006a2
- Check animal
- Failed · sha256 e21ef004e72f5ee91147436e546b095269528f2a7dc8cf5395c28183a7eaa9aa
- Check word
- Passed · sha256 d1bf31457072aaa4c116ea5819a644456232fdd87b15203d3eafdd5e541f4e61
- Check copy
- Passed · sha256 d8c711dfa5fef68d1d0f86320daeca065b91bc8d0dbcea210f8b9ef80eefc6c2
- Check needle-start
- Passed · sha256 56a69645d0ff053b3b7001c224f07a374bf963a22c9fa4706dda5e56e3616870
- Check needle-end
- Failed · sha256 4f3ac3da90b331c87691c05586d5d3b5d29194c2ab1d93c4a82802fec8e40a0e
- Check continuation-128
- Passed · sha256 fe1fdf5d49caa68c1ef2f8a341413cb2c3826023aef1113c6c289dcd86193b32
- Detail
- 7/9 of the checks the model passes at BF16; needs 8 and the mandatory ones (needle-start, continuation-128). Left out, failed by every BF16 configuration: color, sort, needle-end.
Reproduce it
pnpm bench run --plan plans/s3-lfm2.5-2.6b-8-bit.json
Three palindromic blocks per quantization class and workload; load, two warmups, one trial, unload per turn.
Raw data
- This record, as committed JSON, schema v0.3.0
- Complete finalized session evidence raw ·
sha256 92db4b58d893…