Experiment Failed Published
LiteRT-LM 0.19.0 on Pixel 8 · 8 GB, CPU
LFM2.5 2.6B as litert-int8 (8-bit), archive 2048 input / 128 output: a 2048-token prompt and 128 tokens out. Measured 13 Nov 2026, 02:34 UTC by Glenn Sonna, Xybrid.
Doesn’t count in rankings: No trial was valid.
Experiment id s4-pixel8-lfm2.5-2.6b-8-bit-litert-2048
Errors
- trial: No trial was valid.
- trial: No trial was valid.
- trial: No trial was valid.
- trial: No trial was valid.
- trial: No trial was valid.
- trial: No trial was valid.
Summary
Over 0 valid trials of 6 attempted (6 planned), computed from the raw trials below.
| Metric | Median | Range | Mean ± SD | Quartiles |
|---|---|---|---|---|
| Decode speed tok/s, higher is better | Not measured | |||
| Time to first token ms, lower is better | Not measured | |||
| Peak memory GiB, lower is better | Not measured | |||
| Prefill speed tok/s, higher is better | Not measured | |||
Trials, metric by metric
The line runs from the lowest to the highest valid trial, the dot is the median and each tick is one trial.
Every trial
Raw token counts and durations, with the speeds computed from them. Invalid trials stay in the record with their reason; they are left out of the summary, never silently dropped.
| Trial | Time to first token ms | Prefill tok/s | Decode tok/s | Peak memory GiB | Thermal | Note |
|---|---|---|---|---|---|---|
| 1Invalid | not recorded | 93.22,048 tok in 21,970 ms, engine | 7.3127 tok in 17,377 ms, engine | not recorded | 36.9 °C, nominal → 39.8 °C, light | The measurement interval lacks nominal thermal evidence or reported throttling. The device reported thermal throttling during the trial. |
| 2Invalid | not recorded | 91.62,048 tok in 22,349 ms, engine | 7.3127 tok in 17,363 ms, engine | not recorded | 37.0 °C, nominal → 39.7 °C, light | The measurement interval lacks nominal thermal evidence or reported throttling. The device reported thermal throttling during the trial. |
| 3Invalid | not recorded | 92.42,048 tok in 22,167 ms, engine | 7.4127 tok in 17,272 ms, engine | not recorded | 37.0 °C, nominal → 39.7 °C, light | The measurement interval lacks nominal thermal evidence or reported throttling. The device reported thermal throttling during the trial. |
| 4Invalid | not recorded | 93.12,048 tok in 21,995 ms, engine | 7.4127 tok in 17,275 ms, engine | not recorded | 37.0 °C, nominal → 39.6 °C, light | The measurement interval lacks nominal thermal evidence or reported throttling. The device reported thermal throttling during the trial. |
| 5Invalid | not recorded | 93.42,048 tok in 21,921 ms, engine | 7.6127 tok in 16,720 ms, engine | not recorded | 37.0 °C, nominal → 39.8 °C, light | The measurement interval lacks nominal thermal evidence or reported throttling. The device reported thermal throttling during the trial. |
| 6Invalid | not recorded | 91.82,048 tok in 22,299 ms, engine | 7.3127 tok in 17,508 ms, engine | not recorded | 36.9 °C, nominal → 39.9 °C, light | The measurement interval lacks nominal thermal evidence or reported throttling. The device reported thermal throttling during the trial. |
Caveats
- Publisher cards identify the base model but do not always pin the exact export source checkpoint revision.
- Engines may differ in quantization scheme, effective KV cache precision, kernels, and activations.
Configuration
Device
- Device
- Google Pixel 8
- Cohort
- Android phones (mobile)
- SoC
- Google Tensor G3
- CPU
- 9-core CPU (1× Cortex-X3, 4× Cortex-A715, 4× Cortex-A510)
- GPU
- Arm Mali-G715
- NPU
- Not recorded
- RAM
- 8.00 GiB
- OS
- Android 16 (CP1A.260505.005)
- Kernel
- 6.1.145-android14-11-gfa1d6308d1fe-ab14691759
- Power
- mains
- Low power mode
- No
- Wakefulness
- awake
Engine
- Engine
- LiteRT-LM
- Maintainer
- Google AI Edge
- License
- Apache-2.0, open source
- Version
- 0.19.0
- Commit
- Not recorded
- Binary SHA-256
- 8e199c6b3fb438cb94943567a279f9e1f994d57c3e1f4667084fc8ae2d06a6eb
- Built from
- release
- Build flags
- None
- Backend
- CPU
- Threads
- Not recorded
- Interface
- cli
- Settings in effect
- backend=cpu process=one per generation chat_template=none (session API) sampler=top_k=1 temperature=0 launch.max_num_tokens=4096 launch.num_cpu_threads=engine default launch.cache_dir=/data/local/tmp/xybrid/litert-cache launch.benchmark=true launch.benchmark_prefill_tokens=0 launch.benchmark_decode_tokens=0 launch.async=false
Model and weights
- Base model
- LFM2.5 2.6B (LFM2.5, 2.6B)
- Upstream
- LiquidAI/LFM2.5-2.6B@7141b56cd4650af95c7c7b86983443aed050c96a
- Weights
- litert-community/LFM2.5-2.6B@8f87ecede9509fd8d12c3e5b8c46b04b597febfc
- Format
- litertlm
- File 1
- LFM2.5-2.6B_int8.litertlm · 2.67 GiB · sha256 02e97ce3a4de444f1b0bbca4494827704f86212bf72c201703e564d96068536a
- File 2
- README.md · 18,500 bytes · sha256 ad39356f0ce401994fed4791f08a33353db1c6e078358f000d3ddcc8ff756bbb
- Quantization
- litert-int8 (8-bit)
- Weight bits
- 8
- Group size
- Not recorded
- KV cache
- litert-lm default
- Conversion
- Published upstream by litert-community
- Conversion command
- Not recorded
Workload
- Workload
- Archive 2048 input / 128 output
- Prompt
- 2,048 tokens, from prompts/source.txt
- Prompt SHA-256
- c10d1b7bef62d95322f8f8d229dc66118d9375107bdc49b06f76287d058b2f6c
- Chat template
- ea663864491de7ade391839479860ca95541f892f72665c73251fbd4643b1bef
- Output
- 128 tokens, ignoring end of sequence
- Context length
- 4,096
- Batch size
- 1
- Sampling
- Greedy
- Timing
- One streamed request per trial
- Warmup
- 2 runs, not counted
- Trials planned
- 6
- Cooldown
- 60 s between trials
Conditions
- Ambient
- Not recorded
- Thermal at start
- 36.8 °C, nominal
- Thermal at end
- 39.9 °C, light
- Thermal gate
- Android thermal status none (0)
- Throttling detected
- Yes
Provenance
- Measured by
- Glenn Sonna, Xybrid
- Operator affiliation
- Xybrid
- Run at
- 13 Nov 2026, 02:34 UTC
- Harness
- xybrid-bench 0.4.0 (3d9b4e6edd5a0d4463e6585c07504f7db77c3efb)
- Methodology
- v0.6
- Publication
- published
- Published
- 16 Nov 2026, 10:23 UTC
- Reviewed by
- Glenn Sonna
Output check
- Result
- Failed
- Method
- sanity-v1-bf16-reference
- Reference
- checks/sanity-v1.json
- Reference SHA-256
- a702dee079525ae0ec90465b1c9e26052c99cb3e1c53e83dea05be4c194fc7ca
- Required passing checks
- 8
- Check arithmetic
- Failed · sha256 c0b2a5a0debecb2e388c47cdc10113fcd481f224c0efac592a60e4dd102bed56
- Check capital
- Passed · sha256 7a3bb1905f2039638dff3f8a3da57abf7b1da32596b04c5ed04c5296f155196a
- Check opposite
- Passed · sha256 7c90b06eef25984adf77ec226311829bbc1065a033e9ad6b26d9bd0dfdc65f95
- Check weekday
- Passed · sha256 03db656ee680fb1053be9c98631722488bb10269075159dbae5b000f0359dae2
- Check color
- Failed · sha256 6e750f3fc99c2d2843d6acf7e58e66220c77cd0b7acd35364ac2c7ad08794a62
- Check sort
- Failed · sha256 34123c4eaed1c7f1bcb8c1bfafbe9f3449b07dd06dbbb83a54e653a1ba2c0b60
- Check animal
- Failed · sha256 aaf458b3b76ec5162e6ac43bee3ea73000738a3f4fe5c0316852d2f99886fce3
- Check word
- Passed · sha256 d1bf31457072aaa4c116ea5819a644456232fdd87b15203d3eafdd5e541f4e61
- Check copy
- Passed · sha256 d8c711dfa5fef68d1d0f86320daeca065b91bc8d0dbcea210f8b9ef80eefc6c2
- Check needle-start
- Passed · sha256 56a69645d0ff053b3b7001c224f07a374bf963a22c9fa4706dda5e56e3616870
- Check needle-end
- Failed · sha256 4f3ac3da90b331c87691c05586d5d3b5d29194c2ab1d93c4a82802fec8e40a0e
- Check continuation-128
- Passed · sha256 fe1fdf5d49caa68c1ef2f8a341413cb2c3826023aef1113c6c289dcd86193b32
- Detail
- 7/9 of the checks the model passes at BF16; needs 8 and the mandatory ones (needle-start, continuation-128). Left out, failed by every BF16 configuration: color, sort, needle-end.
Reproduce it
pnpm bench run --plan plans/s4-pixel8-lfm2.5-2.6b-8-bit.json
Three palindromic blocks per quantization class and workload; load, two warmups, one trial, unload per turn.
Raw data
- This record, as committed JSON, schema v0.3.0
- Complete finalized session evidence raw ·
sha256 d0d56122afe9…