Rankings

Explore benchmarks

Filter by model, hardware, workload and quantization. Every score links to the comparisons and experiments behind it.

Clear filters

Top Engines

Top Engines, Mobile, Speed score.
#EngineSpeed score out of 100
1uzu, an approximate comparison, see details Mirai 100 /100 Score
2llama.cpp, an approximate comparison, see details ggml-org 69 /100 Score
Not ranked in this view
Not rankedExecuTorch PyTorch (Meta) Not counted
details

Speed Decode speed: tokens generated per second.

Score: 100 is the best result in every comparison an engine entered; 80 is 80% of the best, on average. Only complete experiments with a passed output check count. How rankings work

† uzu: approximate comparison. The experiments use different effective KV cache precision: f16, uzu default. The experiments here use different 8-bit schemes: Q8_0, mirai-l.

† llama.cpp: approximate comparison. The experiments use different effective KV cache precision: f16, uzu default. The experiments here use different 8-bit schemes: Q8_0, mirai-l.

ExecuTorch: Partial: 4 of 6 planned trials were valid.

Coverage

Speed score per device, out of 100: each device ranked on its own.
DesktopMobile
M5 Max 64 GiB macOS 27 Apple M5 MaxOnePlus 13 16 GB Android 16 Qualcomm Snapdragon 8 Elite (SM8750)Pixel 8 8 GB Android 16 Google Tensor G3iPhone 12 4 GB iOS 27 Apple A14 Bionic
llama.cpp85 out of 100, in 11 of 11 comparisons100 out of 100, in 3 of 3 comparisons81 out of 100, in 1 of 1 comparisons69 out of 100, in 1 of 1 comparisons
LiteRT-LMNot ranked. Results in 2 of 11 comparisons; a rank needs 6.81 out of 100, in 2 of 3 comparisons100 out of 100, in 1 of 1 comparisonsNot measured on this device.
uzuNot ranked. Results in 3 of 11 comparisons; a rank needs 6.Not measured on this device.Not measured on this device.100 out of 100, in 1 of 1 comparisons
Magnitude93 out of 100, in 6 of 11 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
MLX LM97 out of 100, in 9 of 11 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
Ollama89 out of 100, in 11 of 11 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
Rapid-MLX97 out of 100, in 6 of 11 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
ExecuTorchNot ranked. Results in 1 of 11 comparisons; a rank needs 6.Not ranked. Results in 1 of 3 comparisons; a rank needs 2.Not counted Partial: 2 of 6 planned trials were valid.Not counted Partial: 4 of 6 planned trials were valid.
ZMLNot ranked. Results in 2 of 11 comparisons; a rank needs 6.Not measured on this device.Not measured on this device.Not measured on this device.

Speed

TTFT ms

Generation tok/s

Speed, LFM2.5 350M on iPhone 12 · 4 GB, 8-bit · Archive 2048 input / 128 output · iOS 27
RankEngineDecode and TTFT
1uzu, approximate comparison, see details Mirai70.2 tok/s 2,356 ms TTFT
2llama.cpp, approximate comparison, see details ggml-org48.3 tok/s 2,210 ms TTFT
details

LFM2.5 350M on iPhone 12 · 4 GB · 8-bit · Archive 2048 input / 128 output · iOS 27

Medians with the model warmed; model loading is excluded. Each table uses one matched comparison.

uzu: † Approximate comparison. The experiments use different effective KV cache precision: f16, uzu default. The experiments here use different 8-bit schemes: Q8_0, mirai-l.

llama.cpp: † Approximate comparison. The experiments use different effective KV cache precision: f16, uzu default. The experiments here use different 8-bit schemes: Q8_0, mirai-l.

Memory usage by engine

Memory usage by engine, LFM2.5 350M on iPhone 12 · 4 GB, 8-bit · Archive 2048 input / 128 output · iOS 27
RankEngineMemory
Not rankedllama.cpp ggml-orgNot measured
Not rankeduzu MiraiNot measured
details

LFM2.5 350M on iPhone 12 · 4 GB · 8-bit · Archive 2048 input / 128 output · iOS 27

Memory ranks require comparable, calibrated measurements. Unavailable measurements remain unranked.

llama.cpp: No peak memory measured. † Approximate comparison. The experiments use different effective KV cache precision: f16, uzu default. The experiments here use different 8-bit schemes: Q8_0, mirai-l.

uzu: No peak memory measured. † Approximate comparison. The experiments use different effective KV cache precision: f16, uzu default. The experiments here use different 8-bit schemes: Q8_0, mirai-l.

Every comparison

Comparisons in this view: each engine's score in each comparison, out of 100, or why it has none.
Comparisonuzullama.cppExecuTorch
LFM2.5 350M on iPhone 12 · 4 GB 8-bit · Archive 2048 input / 128 output · iOS 27 100, approximate69, approximateNot run
LFM2.5 1.2B Instruct on iPhone 12 · 4 GB 4-bit · Archive 2048 input / 128 output · iOS 27 Not counted: fewer than two scored enginesNot countedMeasured, approximateNot counted
LFM2.5 1.2B Instruct on iPhone 12 · 4 GB 8-bit · Archive 2048 input / 128 output · iOS 27 Not counted: fewer than two scored enginesNot countedMeasured, approximateNot run
LFM2.5 1.2B Instruct on iPhone 12 · 4 GB bf16 · Archive 2048 input / 128 output · iOS 27 Not counted: fewer than two scored enginesNot runUnsupportedNot run
LFM2.5 2.6B on iPhone 12 · 4 GB 4-bit · Archive 2048 input / 128 output · iOS 27 Not counted: fewer than two scored enginesUnsupportedNot countedNot run
LFM2.5 2.6B on iPhone 12 · 4 GB 8-bit · Archive 2048 input / 128 output · iOS 27 Not counted: fewer than two scored enginesUnsupportedUnsupportedNot run
LFM2.5 350M on iPhone 12 · 4 GB 4-bit · Archive 2048 input / 128 output · iOS 27 Not counted: fewer than two scored enginesFailedMeasured, approximateNot counted
LFM2.5 350M on iPhone 12 · 4 GB bf16 · Archive 2048 input / 128 output · iOS 27 Not counted: fewer than two scored enginesNot runMeasuredNot run

Each engine’s Speed score in each matched comparison, out of 100, or why it has none. How rankings work