Experiment Partial Published
LiteRT-LM 0.19.0 on Pixel 8 · 8 GB, CPU
LFM2.5 2.6B as litert-int4 (4-bit), archive 2048 input / 128 output: a 2048-token prompt and 128 tokens out. Measured 13 Nov 2026, 04:28 UTC by Glenn Sonna, Xybrid.
Doesn’t count in rankings: Partial: 3 of 6 planned trials were valid.
Experiment id s4-pixel8-lfm2.5-2.6b-4-bit-litert-2048
Errors
- trial: No trial was valid.
- trial: No trial was valid.
- trial: No trial was valid.
Summary
Over 3 valid trials of 6 attempted (6 planned), computed from the raw trials below.
| Metric | Median | Range | Mean ± SD | Quartiles |
|---|---|---|---|---|
| Decode speed tok/s, higher is better | 9.9 | 9.7–11.2 | 10.2 ± 0.8 | 9.8–10.5 |
| Time to first token ms, lower is better | Not measured | |||
| Peak memory GiB, lower is better | Not measured | |||
| Prefill speed tok/s, higher is better | 45.8 | 45.3–46.1 | 45.8 ± 0.4 | 45.6–46.0 |
Trials, metric by metric
The line runs from the lowest to the highest valid trial, the dot is the median and each tick is one trial.
Every trial
Raw token counts and durations, with the speeds computed from them. Invalid trials stay in the record with their reason; they are left out of the summary, never silently dropped.
| Trial | Time to first token ms | Prefill tok/s | Decode tok/s | Peak memory GiB | Thermal | Note |
|---|---|---|---|---|---|---|
| 1 | not recorded | 45.32,048 tok in 45,226 ms, engine | 9.7127 tok in 13,158 ms, engine | not recorded | 38.7 °C, nominal → 38.7 °C, nominal | |
| 2 | not recorded | 46.12,048 tok in 44,403 ms, engine | 9.9127 tok in 12,872 ms, engine | not recorded | 38.6 °C, nominal → 38.8 °C, nominal | |
| 3 | not recorded | 45.82,048 tok in 44,673 ms, engine | 11.2127 tok in 11,380 ms, engine | not recorded | 36.5 °C, nominal → 39.1 °C, nominal | |
| 4Invalid | not recorded | 48.62,048 tok in 42,132 ms, engine | 11.2127 tok in 11,362 ms, engine | not recorded | 36.6 °C, nominal → 39.5 °C, light | The measurement interval lacks nominal thermal evidence or reported throttling. The device reported thermal throttling during the trial. |
| 5Invalid | not recorded | 48.32,048 tok in 42,430 ms, engine | 11.2127 tok in 11,304 ms, engine | not recorded | 36.6 °C, nominal → 39.5 °C, light | The measurement interval lacks nominal thermal evidence or reported throttling. The device reported thermal throttling during the trial. |
| 6Invalid | not recorded | 48.12,048 tok in 42,553 ms, engine | 11.2127 tok in 11,295 ms, engine | not recorded | 36.7 °C, nominal → 39.5 °C, light | The measurement interval lacks nominal thermal evidence or reported throttling. The device reported thermal throttling during the trial. |
Caveats
- Publisher cards identify the base model but do not always pin the exact export source checkpoint revision.
- Engines may differ in quantization scheme, effective KV cache precision, kernels, and activations.
Configuration
Device
- Device
- Google Pixel 8
- Cohort
- Android phones (mobile)
- SoC
- Google Tensor G3
- CPU
- 9-core CPU (1× Cortex-X3, 4× Cortex-A715, 4× Cortex-A510)
- GPU
- Arm Mali-G715
- NPU
- Not recorded
- RAM
- 8.00 GiB
- OS
- Android 16 (CP1A.260505.005)
- Kernel
- 6.1.145-android14-11-gfa1d6308d1fe-ab14691759
- Power
- mains
- Low power mode
- No
- Wakefulness
- awake
Engine
- Engine
- LiteRT-LM
- Maintainer
- Google AI Edge
- License
- Apache-2.0, open source
- Version
- 0.19.0
- Commit
- Not recorded
- Binary SHA-256
- 8e199c6b3fb438cb94943567a279f9e1f994d57c3e1f4667084fc8ae2d06a6eb
- Built from
- release
- Build flags
- None
- Backend
- CPU
- Threads
- Not recorded
- Interface
- cli
- Settings in effect
- backend=cpu process=one per generation chat_template=none (session API) sampler=top_k=1 temperature=0 launch.max_num_tokens=4096 launch.num_cpu_threads=engine default launch.cache_dir=/data/local/tmp/xybrid/litert-cache launch.benchmark=true launch.benchmark_prefill_tokens=0 launch.benchmark_decode_tokens=0 launch.async=false
Model and weights
- Base model
- LFM2.5 2.6B (LFM2.5, 2.6B)
- Upstream
- LiquidAI/LFM2.5-2.6B@7141b56cd4650af95c7c7b86983443aed050c96a
- Weights
- litert-community/LFM2.5-2.6B@8f87ecede9509fd8d12c3e5b8c46b04b597febfc
- Format
- litertlm
- File 1
- LFM2.5-2.6B_int4.litertlm · 1.55 GiB · sha256 7e2194cd81cd82bcbdd6029b3509d99f49c1782afbdfc98c0e4b29b9cc04e6a6
- File 2
- README.md · 18,500 bytes · sha256 ad39356f0ce401994fed4791f08a33353db1c6e078358f000d3ddcc8ff756bbb
- Quantization
- litert-int4 (4-bit)
- Weight bits
- 4
- Group size
- Not recorded
- KV cache
- litert-lm default
- Conversion
- Published upstream by litert-community
- Conversion command
- Not recorded
Workload
- Workload
- Archive 2048 input / 128 output
- Prompt
- 2,048 tokens, from prompts/source.txt
- Prompt SHA-256
- c10d1b7bef62d95322f8f8d229dc66118d9375107bdc49b06f76287d058b2f6c
- Chat template
- ea663864491de7ade391839479860ca95541f892f72665c73251fbd4643b1bef
- Output
- 128 tokens, ignoring end of sequence
- Context length
- 4,096
- Batch size
- 1
- Sampling
- Greedy
- Timing
- One streamed request per trial
- Warmup
- 2 runs, not counted
- Trials planned
- 6
- Cooldown
- 60 s between trials
Conditions
- Ambient
- Not recorded
- Thermal at start
- 36.3 °C, nominal
- Thermal at end
- 39.5 °C, light
- Thermal gate
- Android thermal status none (0)
- Throttling detected
- Yes
Provenance
- Measured by
- Glenn Sonna, Xybrid
- Operator affiliation
- Xybrid
- Run at
- 13 Nov 2026, 04:28 UTC
- Harness
- xybrid-bench 0.4.0 (14b0290a7d4391e5dbe45d6bc492b0715990af88)
- Methodology
- v0.6
- Publication
- published
- Published
- 16 Nov 2026, 10:23 UTC
- Reviewed by
- Glenn Sonna
Output check
- Result
- Passed
- Method
- sanity-v1-bf16-reference
- Reference
- checks/sanity-v1.json
- Reference SHA-256
- a702dee079525ae0ec90465b1c9e26052c99cb3e1c53e83dea05be4c194fc7ca
- Required passing checks
- 8
- Check arithmetic
- Passed · sha256 90a647b2379fe43265d36db39c01e0d324059b2e48f24ec4e2c1f6ec1c0d21eb
- Check capital
- Passed · sha256 7a3bb1905f2039638dff3f8a3da57abf7b1da32596b04c5ed04c5296f155196a
- Check opposite
- Passed · sha256 69a13e1ba8e284fcc26068ec8aa80e8a7e7f8bbebe86dc7a45424411198a6859
- Check weekday
- Passed · sha256 b8c6a621f6e216db1d214b8d906c759be744ff3abc6e41f979ff0d24575cbbcd
- Check color
- Failed · sha256 e7cb21de52f71fbf3756d3e7fbeec92e8b5fa123b1a21476d1fd59911201f7d1
- Check sort
- Failed · sha256 15f349e050f6fd4d9c4f893f73af531ddb5d7b6242c4516df6401d3df0f653f3
- Check animal
- Failed · sha256 aaf458b3b76ec5162e6ac43bee3ea73000738a3f4fe5c0316852d2f99886fce3
- Check word
- Passed · sha256 2e1816dd952a0d4c672ad52a4f32599d4eb0477bf5798060b93fe68f7684dbbd
- Check copy
- Passed · sha256 d8c711dfa5fef68d1d0f86320daeca065b91bc8d0dbcea210f8b9ef80eefc6c2
- Check needle-start
- Passed · sha256 56a69645d0ff053b3b7001c224f07a374bf963a22c9fa4706dda5e56e3616870
- Check needle-end
- Passed · sha256 34b7b9e17d75628c346994d8be48566a2dd011ba8533dc75fc33a579ee705365
- Check continuation-128
- Passed · sha256 fe1fdf5d49caa68c1ef2f8a341413cb2c3826023aef1113c6c289dcd86193b32
- Detail
- 8/9 of the checks the model passes at BF16; needs 8 and the mandatory ones (needle-start, continuation-128). Left out, failed by every BF16 configuration: color, sort, needle-end.
Reproduce it
pnpm bench run --plan plans/s4-pixel8-lfm2.5-2.6b-4-bit.json
Three palindromic blocks per quantization class and workload; load, two warmups, one trial, unload per turn.
Raw data
- This record, as committed JSON, schema v0.3.0
- Complete finalized session evidence raw ·
sha256 34ec477ebcf6…