Rankings

Explore benchmarks

Filter by model, hardware, workload and quantization. Every score links to the comparisons and experiments behind it.

Clear filters

Top Engines

Not enough eligible results to rank.

Coverage

Speed score per device, out of 100: each device ranked on its own.
DesktopMobile
M5 Max 64 GiB macOS 27 Apple M5 MaxOnePlus 13 16 GB Android 16 Qualcomm Snapdragon 8 Elite (SM8750)Pixel 8 8 GB Android 16 Google Tensor G3iPhone 12 4 GB iOS 27 Apple A14 Bionic
llama.cpp93 out of 100, in 4 of 4 comparisonsNo rival yet No other engine has a result in the same comparison yet.Failed Failed in 2 comparisons. No trial was valid.Unsupported Unsupported in 2 comparisons. Partial: 2 of 6 planned trials were valid.
MLX LM100 out of 100, in 2 of 4 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
Ollama98 out of 100, in 4 of 4 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
Rapid-MLX100 out of 100, in 2 of 4 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
LiteRT-LMNo rival yet No other engine has a result in the same comparison yet.No rival yet No other engine has a result in the same comparison yet.No rival yet No other engine has a result in the same comparison yet.Not measured on this device.
uzuUnsupported Unsupported in 4 comparisons. uzu can't run the frozen prompts: Error: uzu counted 23 input tokens, not the frozen prompt's 26 (measured in qualification).Not measured on this device.Not measured on this device.Unsupported Unsupported in 4 comparisons. uzu can't run the frozen prompts: Error: uzu counted 23 input tokens, not the frozen prompt's 26 (measured in qualification).
MagnitudeUnsupported Unsupported in 6 comparisons. Magnitude's chat API renders LFM2.5 2.6B's own template, which always opens a thinking block, so it streams reasoning and can't run the closed-think prompts (measured in qualification).Not measured on this device.Not measured on this device.Not measured on this device.
ZMLUnsupported Unsupported in 2 comparisons. zml can't run the frozen prompts: Error: LLMD streamed reasoning before its answer; the harness can't time it apart from the answer (measured in qualification).Not measured on this device.Not measured on this device.Not measured on this device.

Speed

Speed, LFM2.5 2.6B on OnePlus 13 · 16 GB, 4-bit · Archive 256 input / 128 output · Android 16
RankEngineDecode and TTFT
Not rankedLiteRT-LM, approximate comparison, see details Google AI Edge23.8 tok/s TTFT not measured
Not rankedllama.cpp ggml-orgNot counted
details

LFM2.5 2.6B on OnePlus 13 · 16 GB · 4-bit · Archive 256 input / 128 output · Android 16

Medians with the model warmed; model loading is excluded. Each table uses one matched comparison.

LiteRT-LM: † Approximate comparison. The experiments use different effective KV cache precision: litert-lm default, f16. The experiments here use different 4-bit schemes: litert-int4, Q4_K_M.

llama.cpp: Its output check failed: 8/9 of the checks the model passes at BF16; needs 8 and the mandatory ones (needle-start, continuation-128). Left out, failed by every BF16 configuration: color, sort, needle-end. † Approximate comparison. The experiments use different effective KV cache precision: litert-lm default, f16. The experiments here use different 4-bit schemes: litert-int4, Q4_K_M.

Memory usage by engine

Memory usage by engine, LFM2.5 2.6B on OnePlus 13 · 16 GB, 4-bit · Archive 256 input / 128 output · Android 16
RankEngineMemory
Not rankedLiteRT-LM Google AI EdgeNot measured
Not rankedllama.cpp ggml-orgNot counted
details

LFM2.5 2.6B on OnePlus 13 · 16 GB · 4-bit · Archive 256 input / 128 output · Android 16

Memory ranks require comparable, calibrated measurements. Unavailable measurements remain unranked.

LiteRT-LM: No peak memory measured. † Approximate comparison. The experiments use different effective KV cache precision: litert-lm default, f16. The experiments here use different 4-bit schemes: litert-int4, Q4_K_M.

llama.cpp: Its output check failed: 8/9 of the checks the model passes at BF16; needs 8 and the mandatory ones (needle-start, continuation-128). Left out, failed by every BF16 configuration: color, sort, needle-end. † Approximate comparison. The experiments use different effective KV cache precision: litert-lm default, f16. The experiments here use different 4-bit schemes: litert-int4, Q4_K_M.

Every comparison

Comparisons in this view: each engine's score in each comparison, out of 100, or why it has none.
ComparisonLiteRT-LMllama.cpp
LFM2.5 2.6B on OnePlus 13 · 16 GB 4-bit · Archive 2048 input / 128 output · Android 16 Not counted: fewer than two scored enginesMeasured, approximateNot counted
LFM2.5 2.6B on OnePlus 13 · 16 GB 4-bit · Archive 256 input / 128 output · Android 16 Not counted: fewer than two scored enginesMeasured, approximateNot counted
LFM2.5 2.6B on OnePlus 13 · 16 GB 8-bit · Archive 2048 input / 128 output · Android 16 Not counted: fewer than two scored enginesNot countedMeasured, approximate
LFM2.5 2.6B on OnePlus 13 · 16 GB 8-bit · Archive 256 input / 128 output · Android 16 Not counted: fewer than two scored enginesNot countedMeasured, approximate
LFM2.5 2.6B on Pixel 8 · 8 GB 4-bit · Archive 2048 input / 128 output · Android 16 Not counted: fewer than two scored enginesNot countedFailed
LFM2.5 2.6B on Pixel 8 · 8 GB 4-bit · Archive 256 input / 128 output · Android 16 Not counted: fewer than two scored enginesMeasured, approximateNot counted
LFM2.5 2.6B on Pixel 8 · 8 GB 8-bit · Archive 2048 input / 128 output · Android 16 Not counted: fewer than two scored enginesFailedFailed
LFM2.5 2.6B on Pixel 8 · 8 GB 8-bit · Archive 256 input / 128 output · Android 16 Not counted: fewer than two scored enginesNot countedNot counted

Each engine’s Speed score in each matched comparison, out of 100, or why it has none. How rankings work