Rankings

Engine

ExecuTorch

PyTorch (Meta) · BSD-3-Clause, open source · Homepage

Top Engines

Desktop
Not ranked Results in 0 of 4 comparisons; a rank needs all 4.
Mobile
Not ranked Results in 0 of 12 comparisons; a rank needs all 12.
details

Approximate comparison: The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype, uzu default. Its weights are 4-bit in another scheme: mlx-4w-g64, where most use Q4_K_M. The experiments use different effective KV cache precision: fp32, f16, uzu default. The experiments here use different 4-bit schemes: 8da4w, Q4_K_M, mirai-m. The experiments use different effective KV cache precision: fp32, litert-lm default, f16. The experiments here use different 4-bit schemes: 8da4w, litert-int4, Q4_K_M.

How rankings work

Supporting experiments

ExecuTorch: every experiment, grouped by matched comparison. Medians over valid trials.
ConfigurationStatusGenerationtok/sFirst tokenmsPrefilltok/sMemoryGiB
LFM2.5 1.2B Instruct on M5 Max · 64 GiB 4-bit · Archive 256 input / 128 output · macOS 27 Speed score 87, approximate comparison
LFM2.5 1.2B Instruct on M5 Max · 64 GiB, 4-bit · Archive 256 input / 128 output · macOS 27: 1.6.0 · Metal mlx-4w-g64 · 6/6 validCounted 472.1range470.4–475.752.3range51.6–52.54,923range4,830–5,020Not measured
LFM2.5 1.2B Instruct on M5 Max · 64 GiB, 4-bit · Archive 256 input / 128 output · macOS 27: 1.6.0 · CPU 8da4w · 6/6 validNot this engine’s entry 158.8range156.6–161.8262range261–272977.1range941.2–980.8Not measured
LFM2.5 350M on M5 Max · 64 GiB 4-bit · Archive 256 input / 128 output · macOS 27
LFM2.5 350M on M5 Max · 64 GiB, 4-bit · Archive 256 input / 128 output · macOS 27: 1.6.0 · Metal mlx-4w-g64 · 6/6 validCheck failed 786.4range779.1–788.823.4range23.1–23.511,130range10,667–11,130Not measured
LFM2.5 350M on M5 Max · 64 GiB, 4-bit · Archive 256 input / 128 output · macOS 27: 1.6.0 · CPU 8da4w · 6/6 validCheck failed 335.5range334.2–340.585.8range85.7–86.72,977range2,943–3,012Not measured
LFM2.5 1.2B Instruct on M5 Max · 64 GiB 4-bit · Archive 2048 input / 128 output · macOS 27 Speed score 86, approximate comparison
LFM2.5 1.2B Instruct on M5 Max · 64 GiB, 4-bit · Archive 2048 input / 128 output · macOS 27: 1.6.0 · Metal mlx-4w-g64 · 6/6 validCounted 456.8range445.6–458.5332range330–3326,178range6,169–6,206Not measured
LFM2.5 1.2B Instruct on M5 Max · 64 GiB, 4-bit · Archive 2048 input / 128 output · macOS 27: 1.6.0 · CPU 8da4w · 6/6 validNot this engine’s entry 121.6range116.8–128.52,405range2,357–3,052851.6range671.3–869.3Not measured
LFM2.5 350M on M5 Max · 64 GiB 4-bit · Archive 2048 input / 128 output · macOS 27
LFM2.5 350M on M5 Max · 64 GiB, 4-bit · Archive 2048 input / 128 output · macOS 27: 1.6.0 · Metal mlx-4w-g64 · 6/6 validCheck failed 738.4range725.7–747.1107range107–10719,140range19,140–19,321Not measured
LFM2.5 350M on M5 Max · 64 GiB, 4-bit · Archive 2048 input / 128 output · macOS 27: 1.6.0 · CPU 8da4w · 6/6 validCheck failed 246.8range237.4–248.5720range719–7252,844range2,829–2,848Not measured
LFM2.5 1.2B Instruct on iPhone 12 · 4 GB 4-bit · Archive 256 input / 128 output · iOS 27 Speed score 94, approximate comparison
LFM2.5 1.2B Instruct on iPhone 12 · 4 GB, 4-bit · Archive 256 input / 128 output · iOS 27: 1.6.0 · CPU 8da4w · 6/6 validCounted 42.5range40.4–42.82,033range1,990–2,372125.9range107.9–128.6Not measured
LFM2.5 350M on iPhone 12 · 4 GB 4-bit · Archive 256 input / 128 output · iOS 27
LFM2.5 350M on iPhone 12 · 4 GB, 4-bit · Archive 256 input / 128 output · iOS 27: 1.6.0 · CPU 8da4w · 6/6 validCheck failed 121.9range110.3–122.8481range469–582532.2range439.9–547.0Not measured
LFM2.5 1.2B Instruct on OnePlus 13 · 16 GB 4-bit · Archive 256 input / 128 output · Android 16 Speed score 100, approximate comparison
LFM2.5 1.2B Instruct on OnePlus 13 · 16 GB, 4-bit · Archive 256 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 validCounted 76.3range75.6–76.6Not measured559.0range545.8–566.4Not measured
LFM2.5 1.2B Instruct on Pixel 8 · 8 GB 4-bit · Archive 256 input / 128 output · Android 16 Speed score 100, approximate comparison
LFM2.5 1.2B Instruct on Pixel 8 · 8 GB, 4-bit · Archive 256 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 validCounted 31.7range29.4–32.3Not measured156.3range147.2–165.4Not measured
LFM2.5 1.2B Instruct on OnePlus 13 · 16 GB 4-bit · Archive 2048 input / 128 output · Android 16 Speed score 100, approximate comparison
LFM2.5 1.2B Instruct on OnePlus 13 · 16 GB, 4-bit · Archive 2048 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 validCounted 52.1range51.4–53.3Not measured480.0range474.8–549.4Not measured
LFM2.5 350M on OnePlus 13 · 16 GB 4-bit · Archive 256 input / 128 output · Android 16 Not counted
LFM2.5 350M on OnePlus 13 · 16 GB, 4-bit · Archive 256 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 validCheck failed 203.0range199.7–204.5Not measured1,524range1,471–1,662Not measured
LFM2.5 350M on Pixel 8 · 8 GB 4-bit · Archive 256 input / 128 output · Android 16 Not counted
LFM2.5 350M on Pixel 8 · 8 GB, 4-bit · Archive 256 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 validCheck failed 97.5range85.7–101.4Not measured329.1range293.2–365.2Not measured
LFM2.5 1.2B Instruct on iPhone 12 · 4 GB 4-bit · Archive 2048 input / 128 output · iOS 27 Not counted
LFM2.5 1.2B Instruct on iPhone 12 · 4 GB, 4-bit · Archive 2048 input / 128 output · iOS 27: 1.6.0 · CPU 8da4w · 4/6 validPartial 32.2range26.9–34.419,336range16,021–19,605105.9range104.5–127.8Not measured
LFM2.5 350M on iPhone 12 · 4 GB 4-bit · Archive 2048 input / 128 output · iOS 27 Not counted
LFM2.5 350M on iPhone 12 · 4 GB, 4-bit · Archive 2048 input / 128 output · iOS 27: 1.6.0 · CPU 8da4w · 6/6 validCheck failed 83.5range81.0–84.04,731range4,723–5,197432.8range394.1–433.6Not measured
LFM2.5 350M on OnePlus 13 · 16 GB 4-bit · Archive 2048 input / 128 output · Android 16 Not counted
LFM2.5 350M on OnePlus 13 · 16 GB, 4-bit · Archive 2048 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 validCheck failed 121.4range119.2–122.2Not measured1,510range1,503–1,527Not measured
LFM2.5 1.2B Instruct on Pixel 8 · 8 GB 4-bit · Archive 2048 input / 128 output · Android 16 Not counted
LFM2.5 1.2B Instruct on Pixel 8 · 8 GB, 4-bit · Archive 2048 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 2/6 validPartial 19.7range19.5–19.9Not measured213.7range212.3–215.1Not measured
LFM2.5 350M on Pixel 8 · 8 GB 4-bit · Archive 2048 input / 128 output · Android 16 Not counted
LFM2.5 350M on Pixel 8 · 8 GB, 4-bit · Archive 2048 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 validCheck failed 53.0range52.7–53.7Not measured665.9range631.1–675.7Not measured