Rankings

Explore benchmarks

Filter by model, hardware, workload and quantization. Every score links to the comparisons and experiments behind it.

Clear filters

Top Engines

Top Engines, Desktop, Speed score.
#EngineSpeed score out of 100
197 /100 Score
295 /100 Score
395 /100 Score
480 /100 Score
576 /100 Score
Not ranked in this view
Not rankedLiteRT-LM Google AI Edge Too few results
Not rankeduzu Mirai Too few results
Not rankedZML ZML Too few results
Not rankedExecuTorch PyTorch (Meta) Not counted
details

Speed Decode speed: tokens generated per second.

Score: 100 is the best result in every comparison an engine entered; 80 is 80% of the best, on average. Only complete experiments with a passed output check count. How rankings work

The score combines counted comparisons. Select one comparison to see its measured values.

† Rapid-MLX: approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its weights are 4-bit in another scheme: mlx-affine-4bit-g64, where most use Q4_K_M. The experiments use different effective KV cache precision: litert-lm default, f16, magnitude default, model-dtype, uzu default. Its weights are 8-bit in another scheme: mlx-affine-8bit-g64, where most use Q8_0. The experiments use different effective KV cache precision: f16, magnitude default, model-dtype, zml default.

† Magnitude: approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its phases were timed by the harness clock around the engine API, so they include transport that engine counters leave out. The experiments use different effective KV cache precision: litert-lm default, f16, magnitude default, model-dtype, uzu default. The experiments use different effective KV cache precision: f16, magnitude default, model-dtype, zml default.

† MLX LM: approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its weights are 4-bit in another scheme: mlx-affine-4bit-g64, where most use Q4_K_M. The experiments use different effective KV cache precision: litert-lm default, f16, magnitude default, model-dtype, uzu default. Its weights are 8-bit in another scheme: mlx-affine-8bit-g64, where most use Q8_0. The experiments use different effective KV cache precision: f16, magnitude default, model-dtype, zml default.

† Ollama: approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. The experiments use different effective KV cache precision: litert-lm default, f16, magnitude default, model-dtype, uzu default. The experiments use different effective KV cache precision: f16, magnitude default, model-dtype, zml default.

† llama.cpp: approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. The experiments use different effective KV cache precision: litert-lm default, f16, magnitude default, model-dtype, uzu default. The experiments use different effective KV cache precision: f16, magnitude default, model-dtype, zml default.

LiteRT-LM: Results in 1 of 3 comparisons; a rank needs 2.

uzu: Results in 1 of 3 comparisons; a rank needs 2.

ZML: Results in 1 of 3 comparisons; a rank needs 2.

ExecuTorch: Its output check failed: 6/8 of the checks the model passes at BF16; needs 7 and the mandatory ones (needle-start, needle-end). Left out, failed by every BF16 configuration: arithmetic, sort, word, continuation-128.

Coverage

Speed score per device, out of 100: each device ranked on its own.
DesktopMobile
M5 Max 64 GiB macOS 27 Apple M5 MaxOnePlus 13 16 GB Android 16 Qualcomm Snapdragon 8 Elite (SM8750)Pixel 8 8 GB Android 16 Google Tensor G3iPhone 12 4 GB iOS 27 Apple A14 Bionic
llama.cpp76 out of 100, in 3 of 3 comparisons100 out of 100, in 1 of 1 comparisons81 out of 100, in 1 of 1 comparisons69 out of 100, in 1 of 1 comparisons
LiteRT-LMNot ranked. Results in 1 of 3 comparisons; a rank needs 2.76 out of 100, in 1 of 1 comparisons100 out of 100, in 1 of 1 comparisonsNot measured on this device.
uzuNot ranked. Results in 1 of 3 comparisons; a rank needs 2.Not measured on this device.Not measured on this device.100 out of 100, in 1 of 1 comparisons
Magnitude95 out of 100, in 3 of 3 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
MLX LM95 out of 100, in 3 of 3 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
Ollama80 out of 100, in 3 of 3 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
Rapid-MLX97 out of 100, in 3 of 3 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
ExecuTorchNot counted Its output check failed: 6/8 of the checks the model passes at BF16; needs 7 and the mandatory ones (needle-start, needle-end). Left out, failed by every BF16 configuration: arithmetic, sort, word, continuation-128.Not counted Its output check failed: 5/8 of the checks the model passes at BF16; needs 7 and the mandatory ones (needle-start, needle-end). Left out, failed by every BF16 configuration: arithmetic, sort, word, continuation-128.Not counted Its output check failed: 5/8 of the checks the model passes at BF16; needs 7 and the mandatory ones (needle-start, needle-end). Left out, failed by every BF16 configuration: arithmetic, sort, word, continuation-128.Not counted Its output check failed: 5/8 of the checks the model passes at BF16; needs 7 and the mandatory ones (needle-start, needle-end). Left out, failed by every BF16 configuration: arithmetic, sort, word, continuation-128.
ZMLNot ranked. Results in 1 of 3 comparisons; a rank needs 2.Not measured on this device.Not measured on this device.Not measured on this device.

Speed

TTFT ms

Generation tok/s

Speed, LFM2.5 350M on M5 Max · 64 GiB, 4-bit · Archive 2048 input / 128 output · macOS 27
RankEngineDecode and TTFT
1Rapid-MLX, approximate comparison, see details Rapid-MLX contributors946.8 tok/s 57.6 ms TTFT
2MLX LM, approximate comparison, see details Apple (ml-explore)897.8 tok/s 56.2 ms TTFT
3Magnitude, approximate comparison, see details Magnitude884.9 tok/s 46.5 ms TTFT
4Ollama, approximate comparison, see details Ollama700.6 tok/s 64.1 ms TTFT
5llama.cpp, approximate comparison, see details ggml-org650.0 tok/s 65.2 ms TTFT
Not rankedExecuTorch PyTorch (Meta)Not counted
Not rankedLiteRT-LM Google AI EdgeNot counted
Not rankeduzu MiraiFailed
details

LFM2.5 350M on M5 Max · 64 GiB · 4-bit · Archive 2048 input / 128 output · macOS 27

Medians with the model warmed; model loading is excluded. Each table uses one matched comparison.

Rapid-MLX: † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its weights are 4-bit in another scheme: mlx-affine-4bit-g64, where most use Q4_K_M.

MLX LM: † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its weights are 4-bit in another scheme: mlx-affine-4bit-g64, where most use Q4_K_M.

Magnitude: † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its phases were timed by the harness clock around the engine API, so they include transport that engine counters leave out.

Ollama: † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype.

llama.cpp: † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype.

ExecuTorch: Its output check failed: 6/8 of the checks the model passes at BF16; needs 7 and the mandatory ones (needle-start, needle-end). Left out, failed by every BF16 configuration: arithmetic, sort, word, continuation-128. † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its weights are 4-bit in another scheme: mlx-4w-g64, where most use Q4_K_M.

LiteRT-LM: Its output check failed: 6/8 of the checks the model passes at BF16; needs 7 and the mandatory ones (needle-start, needle-end). Left out, failed by every BF16 configuration: arithmetic, sort, word, continuation-128. † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its weights are 4-bit in another scheme: litert-int4, where most use Q4_K_M.

uzu: No trial was valid.

Memory usage by engine

details

LFM2.5 350M on M5 Max · 64 GiB · 4-bit · Archive 2048 input / 128 output · macOS 27

Memory ranks require comparable, calibrated measurements. Unavailable measurements remain unranked.

Ollama: † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype.

llama.cpp: † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype.

Magnitude: † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its phases were timed by the harness clock around the engine API, so they include transport that engine counters leave out.

MLX LM: † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its weights are 4-bit in another scheme: mlx-affine-4bit-g64, where most use Q4_K_M.

Rapid-MLX: † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its weights are 4-bit in another scheme: mlx-affine-4bit-g64, where most use Q4_K_M.

ExecuTorch: Its output check failed: 6/8 of the checks the model passes at BF16; needs 7 and the mandatory ones (needle-start, needle-end). Left out, failed by every BF16 configuration: arithmetic, sort, word, continuation-128. † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its weights are 4-bit in another scheme: mlx-4w-g64, where most use Q4_K_M.

LiteRT-LM: Its output check failed: 6/8 of the checks the model passes at BF16; needs 7 and the mandatory ones (needle-start, needle-end). Left out, failed by every BF16 configuration: arithmetic, sort, word, continuation-128. † Approximate comparison. The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype. Its weights are 4-bit in another scheme: litert-int4, where most use Q4_K_M.

uzu: No trial was valid.

Every comparison

Comparisons in this view: each engine's score in each comparison, out of 100, or why it has none.
ComparisonRapid-MLXMagnitudeMLX LMOllamallama.cppLiteRT-LMuzuZMLExecuTorch
LFM2.5 350M on M5 Max · 64 GiB 4-bit · Archive 2048 input / 128 output · macOS 27 100, approximate93, approximate95, approximate74, approximate69, approximateNot countedFailedNot runNot counted
LFM2.5 350M on M5 Max · 64 GiB 8-bit · Archive 2048 input / 128 output · macOS 27 92, approximate96, approximate90, approximate77, approximate76, approximate26, approximate100, approximateNot runNot run
LFM2.5 350M on M5 Max · 64 GiB bf16 · Archive 2048 input / 128 output · macOS 27 100, approximate96, approximate99, approximate88, approximate83, approximateNot runNot run49, approximateNot run

Each engine’s Speed score in each matched comparison, out of 100, or why it has none. How rankings work