On-device AI leaderboard
Top Engines
Desktop: 3 engines ranked
+ 6 engines with incomplete coverage
Coverage
| Desktop | Mobile | |||
|---|---|---|---|---|
| M5 Max 64 GiB macOS 27 Apple M5 Max | OnePlus 13 16 GB Android 16 Qualcomm Snapdragon 8 Elite (SM8750) | Pixel 8 8 GB Android 16 Google Tensor G3 | iPhone 12 4 GB iOS 27 Apple A14 Bionic | |
| llama.cpp | 8522/22 out of 100, in 22 of 22 comparisons | 976/6 out of 100, in 6 of 6 comparisons | 864/4 out of 100, in 4 of 4 comparisons | 695/5 out of 100, in 5 of 5 comparisons |
| LiteRT-LM | 4/22Not ranked. Results in 4 of 22 comparisons; a rank needs 11. | 784/6 out of 100, in 4 of 6 comparisons | 983/4 out of 100, in 3 of 4 comparisons | Not measured on this device. |
| uzu | 7/22Not ranked. Results in 7 of 22 comparisons; a rank needs 11. | Not measured on this device. | Not measured on this device. | 1005/5 out of 100, in 5 of 5 comparisons |
| Magnitude | 9112/22 out of 100, in 12 of 22 comparisons | Not measured on this device. | Not measured on this device. | Not measured on this device. |
| MLX LM | 9818/22 out of 100, in 18 of 22 comparisons | Not measured on this device. | Not measured on this device. | Not measured on this device. |
| Ollama | 8822/22 out of 100, in 22 of 22 comparisons | Not measured on this device. | Not measured on this device. | Not measured on this device. |
| Rapid-MLX | 9812/22 out of 100, in 12 of 22 comparisons | Not measured on this device. | Not measured on this device. | Not measured on this device. |
| ExecuTorch | 2/22Not ranked. Results in 2 of 22 comparisons; a rank needs 11. | 2/6Not ranked. Results in 2 of 6 comparisons; a rank needs 3. | 1/4Not ranked. Results in 1 of 4 comparisons; a rank needs 2. | 1/5Not ranked. Results in 1 of 5 comparisons; a rank needs 3. |
| ZML | 4/22Not ranked. Results in 4 of 22 comparisons; a rank needs 11. | Not measured on this device. | Not measured on this device. | Not measured on this device. |
Performance
Performance scope: Desktop.
LFM2.5 1.2B Instruct on M5 Max · 64 GiB · 8-bit · Archive 256 input / 128 output · macOS 27Speed
- Rank 1: Magnitude 25.5 ms, range 25.3 to 25.6; open the experiment and its raw trials
- Rank 2: Ollama 30.5 ms, range 24.1 to 32.3; open the experiment and its raw trials
- Rank 3: MLX LM 30.7 ms, range 30.6 to 31.2; open the experiment and its raw trials
- Rank 4: Rapid-MLX 33.5 ms, range 31.3 to 37.3; open the experiment and its raw trials
- Rank 5: llama.cpp 34.5 ms, range 34.0 to 36.6; open the experiment and its raw trials
- Rank 6: uzu 75.1 ms, range 74.3 to 76.5; open the experiment and its raw trials
- Rank 7: LiteRT-LM 277 ms, range 276 to 278; open the experiment and its raw trials
Lowest Memory
- Rank 1: Ollama 1.41 GiB, range 1.41 to 1.43; open the experiment and its raw trials
- Rank 2: llama.cpp 1.41 GiB, range 1.41 to 1.43; open the experiment and its raw trials
- Rank 3: MLX LM 1.44 GiB, range 1.43 to 1.45; open the experiment and its raw trials
- Rank 4: uzu 1.46 GiB, range 1.46 to 1.52; open the experiment and its raw trials
- Rank 5: Rapid-MLX 1.69 GiB, range 1.68 to 1.70; open the experiment and its raw trials
- Rank 6: LiteRT-LM 1.73 GiB, range 1.73 to 1.73; open the experiment and its raw trials
- Rank 7: Magnitude 1.90 GiB, range 1.89 to 1.90; open the experiment and its raw trials
Performance Breakdowns
Speed score by model, out of 100
| LFM2.5 350M | LFM2.5 1.2B Instruct | LFM2.5 2.6B | Qwen3.8 27B | Ternary Bonsai 2 27B | |
|---|---|---|---|---|---|
| MLX LM | 97 | 97 | 100 | 98 | 99 |
| Rapid-MLX | 98 | 96 | 100 | No score. Not measured on this model. | No score. Not measured on this model. |
| Magnitude | 92 | 91 | No score. Unsupported in 6 comparisons. Magnitude's chat API renders LFM2.5 2.6B's own template, which always opens a thinking block, so it streams reasoning and can't run the closed-think prompts (measured in qualification). | No score. Unsupported in 4 comparisons. Magnitude 0.2.17 reports this GGUF as Unsupported (magnitude catalog inspect) | No score. Unsupported in 2 comparisons. Magnitude 0.2.17 reports this GGUF as Unsupported (magnitude catalog inspect) |
| Ollama | 78 | 90 | 98 | 88 | 100 |
| llama.cpp | 74 | 87 | 93 | 86 | 96 |
| uzu | 96 | 100 | No score. Unsupported in 4 comparisons. uzu can't run the frozen prompts: Error: uzu counted 23 input tokens, not the frozen prompt's 26 (measured in qualification). | No score. Unsupported in 4 comparisons. uzu 0.6.17 renders its own Qwen3.8 chat template: 67 input tokens for the 27-token canonical check prompt, so it can't run the frozen prompts (measured in qualification). | No score. Not measured on this model. |