Explore benchmarks
Filter by model, hardware, workload and quantization. Every score links to the comparisons and experiments behind it.
Top Engines
Overall needs mobile results; each platform is ranked on its own until then.
Desktop
| # | Engine | Speed score out of 100 |
|---|---|---|
| 1 | MLX LM†, an approximate comparison, see details | 98 /100 Score |
| ||
| 2 | Ollama†, an approximate comparison, see details | 88 /100 Score |
| ||
| 3 | llama.cpp†, an approximate comparison, see details | 86 /100 Score |
| ||
| Not ranked in this view | ||
| Not ranked | Magnitude | Unsupported |
| Not ranked | uzu | Unsupported |
details
Speed Decode speed: tokens generated per second.
Score: 100 is the best result in every comparison an engine entered; 80 is 80% of the best, on average. Only complete experiments with a passed output check count. How rankings work
The score combines counted comparisons. Select one comparison to see its measured values.
† MLX LM: approximate comparison. The experiments use different effective KV cache precision: f16, model-dtype. Its weights are 4-bit in another scheme: mlx-affine-4bit-g64, where most use Q4_K_M. Its weights are 8-bit in another scheme: mlx-affine-8bit-g64, where most use Q8_0.
† Ollama: approximate comparison. The experiments use different effective KV cache precision: f16, model-dtype.
† llama.cpp: approximate comparison. The experiments use different effective KV cache precision: f16, model-dtype.
Magnitude: Unsupported in 2 comparisons. Magnitude 0.2.17 reports this GGUF as Unsupported (magnitude catalog inspect)
uzu: Unsupported in 2 comparisons. uzu 0.6.17 renders its own Qwen3.8 chat template: 67 input tokens for the 27-token canonical check prompt, so it can't run the frozen prompts (measured in qualification).
Coverage
| Desktop | |
|---|---|
| M5 Max 64 GiB macOS 27 Apple M5 Max | |
| llama.cpp | 862/2 out of 100, in 2 of 2 comparisons |
| MLX LM | 982/2 out of 100, in 2 of 2 comparisons |
| Ollama | 882/2 out of 100, in 2 of 2 comparisons |
| Magnitude | Unsupported Unsupported in 2 comparisons. Magnitude 0.2.17 reports this GGUF as Unsupported (magnitude catalog inspect) |
| uzu | Unsupported Unsupported in 2 comparisons. uzu 0.6.17 renders its own Qwen3.8 chat template: 67 input tokens for the 27-token canonical check prompt, so it can't run the frozen prompts (measured in qualification). |
Speed
| Rank | Engine | Decode and TTFT |
|---|---|---|
| 1 | MLX LM†, approximate comparison, see details | 33.4 tok/s 350 ms TTFT |
| 2 | Ollama†, approximate comparison, see details | 26.0 tok/s 394 ms TTFT |
| 3 | llama.cpp†, approximate comparison, see details | 25.1 tok/s 421 ms TTFT |
| Not ranked | Magnitude | Unsupported |
| Not ranked | uzu | Unsupported |
details
Qwen3.8 27B on M5 Max · 64 GiB · 4-bit · Archive 256 input / 128 output · macOS 27
Medians with the model warmed; model loading is excluded. Each table uses one matched comparison.
MLX LM: † Approximate comparison. The experiments use different effective KV cache precision: f16, model-dtype. Its weights are 4-bit in another scheme: mlx-affine-4bit-g64, where most use Q4_K_M.
Ollama: † Approximate comparison. The experiments use different effective KV cache precision: f16, model-dtype.
llama.cpp: † Approximate comparison. The experiments use different effective KV cache precision: f16, model-dtype.
Magnitude: Magnitude 0.2.17 reports this GGUF as Unsupported (magnitude catalog inspect)
uzu: uzu 0.6.17 renders its own Qwen3.8 chat template: 67 input tokens for the 27-token canonical check prompt, so it can't run the frozen prompts (measured in qualification).
Memory usage by engine
| Rank | Engine | Memory |
|---|---|---|
| 1 | MLX LM†, approximate comparison, see details | 15.48 GiB |
| 2 | Ollama†, approximate comparison, see details | 18.90 GiB |
| 3 | llama.cpp†, approximate comparison, see details | 18.90 GiB |
| Not ranked | Magnitude | Unsupported |
| Not ranked | uzu | Unsupported |
details
Qwen3.8 27B on M5 Max · 64 GiB · 4-bit · Archive 256 input / 128 output · macOS 27
Memory ranks require comparable, calibrated measurements. Unavailable measurements remain unranked.
MLX LM: † Approximate comparison. The experiments use different effective KV cache precision: f16, model-dtype. Its weights are 4-bit in another scheme: mlx-affine-4bit-g64, where most use Q4_K_M.
Ollama: † Approximate comparison. The experiments use different effective KV cache precision: f16, model-dtype.
llama.cpp: † Approximate comparison. The experiments use different effective KV cache precision: f16, model-dtype.
Magnitude: Magnitude 0.2.17 reports this GGUF as Unsupported (magnitude catalog inspect)
uzu: uzu 0.6.17 renders its own Qwen3.8 chat template: 67 input tokens for the 27-token canonical check prompt, so it can't run the frozen prompts (measured in qualification).
Every comparison
| Comparison | llama.cpp | Magnitude | MLX LM | Ollama | uzu |
|---|---|---|---|---|---|
| Qwen3.8 27B on M5 Max · 64 GiB 4-bit · Archive 256 input / 128 output · macOS 27 | 75†, approximate | Unsupported | 100†, approximate | 78†, approximate | Unsupported |
| Qwen3.8 27B on M5 Max · 64 GiB 8-bit · Archive 256 input / 128 output · macOS 27 | 98†, approximate | Unsupported | 97†, approximate | 100†, approximate | Unsupported |
Each engine’s Speed score in each matched comparison, out of 100, or why it has none. How rankings work