On-device AI leaderboard

Top Engines

Mobile: 0 engines ranked

+ 3 engines with incomplete coverage

Coverage

Speed score per device, out of 100: each device ranked on its own.
DesktopMobile
M5 Max 64 GiB macOS 27 Apple M5 MaxOnePlus 13 16 GB Android 16 Qualcomm Snapdragon 8 Elite (SM8750)Pixel 8 8 GB Android 16 Google Tensor G3iPhone 12 4 GB iOS 27 Apple A14 Bionic
llama.cpp85 out of 100, in 22 of 22 comparisons97 out of 100, in 6 of 6 comparisons86 out of 100, in 4 of 4 comparisons69 out of 100, in 5 of 5 comparisons
LiteRT-LMNot ranked. Results in 4 of 22 comparisons; a rank needs 11.78 out of 100, in 4 of 6 comparisons98 out of 100, in 3 of 4 comparisonsNot measured on this device.
uzuNot ranked. Results in 7 of 22 comparisons; a rank needs 11.Not measured on this device.Not measured on this device.100 out of 100, in 5 of 5 comparisons
Magnitude91 out of 100, in 12 of 22 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
MLX LM98 out of 100, in 18 of 22 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
Ollama88 out of 100, in 22 of 22 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
Rapid-MLX98 out of 100, in 12 of 22 comparisonsNot measured on this device.Not measured on this device.Not measured on this device.
ExecuTorchNot ranked. Results in 2 of 22 comparisons; a rank needs 11.Not ranked. Results in 2 of 6 comparisons; a rank needs 3.Not ranked. Results in 1 of 4 comparisons; a rank needs 2.Not ranked. Results in 1 of 5 comparisons; a rank needs 3.
ZMLNot ranked. Results in 4 of 22 comparisons; a rank needs 11.Not measured on this device.Not measured on this device.Not measured on this device.

Performance

Performance scope: Mobile.

LFM2.5 1.2B Instruct on iPhone 12 · 4 GB · 4-bit · Archive 256 input / 128 output · iOS 27

Speed

Response time, ms

Response time, ms. Medians, ms.
EngineMedian Range
llama.cpp987 984 to 993
uzu1,073 1,055 to 1,117
ExecuTorch2,033 1,990 to 2,372
  1. Rank 1: llama.cpp 987 ms, range 984 to 993; open the experiment and its raw trials
  2. Rank 2: uzu 1,073 ms, range 1,055 to 1,117; open the experiment and its raw trials
  3. Rank 3: ExecuTorch 2,033 ms, range 1,990 to 2,372; open the experiment and its raw trials

Performance Breakdowns

Speed score by model, out of 100

Speed score by model, out of 100: 100 is the fastest engine on that model.
LFM2.5 350MLFM2.5 1.2B Instruct
llama.cpp7790
ExecuTorchNo score. Its output check failed: 5/8 of the checks the model passes at BF16; needs 7 and the mandatory ones (needle-start, needle-end). Left out, failed by every BF16 configuration: arithmetic, sort, word, continuation-128.98
LiteRT-LM82No score. Results in 3 of 8 comparisons; a rank needs 4.

Explore Benchmarks

Open the explorer