Rankings
Engine
ExecuTorch
PyTorch (Meta) · BSD-3-Clause, open source · Homepage
Top Engines
- Desktop
- Not ranked
- Mobile
- Not ranked
details
Approximate comparison: The experiments use different effective KV cache precision: bf16, fp32, litert-lm default, f16, magnitude default, model-dtype, uzu default. Its weights are 4-bit in another scheme: mlx-4w-g64, where most use Q4_K_M. The experiments use different effective KV cache precision: fp32, f16, uzu default. The experiments here use different 4-bit schemes: 8da4w, Q4_K_M, mirai-m. The experiments use different effective KV cache precision: fp32, litert-lm default, f16. The experiments here use different 4-bit schemes: 8da4w, litert-int4, Q4_K_M.
Supporting experiments
| Configuration | Status | Generationtok/s | First tokenms | Prefilltok/s | MemoryGiB |
|---|---|---|---|---|---|
| LFM2.5 1.2B Instruct on M5 Max · 64 GiB 4-bit · Archive 256 input / 128 output · macOS 27 Speed score 87, approximate comparison | |||||
| LFM2.5 1.2B Instruct on M5 Max · 64 GiB, 4-bit · Archive 256 input / 128 output · macOS 27: 1.6.0 · Metal mlx-4w-g64 · 6/6 valid | Counted | 472.1range470.4–475.7 | 52.3range51.6–52.5 | 4,923range4,830–5,020 | Not measured |
| LFM2.5 1.2B Instruct on M5 Max · 64 GiB, 4-bit · Archive 256 input / 128 output · macOS 27: 1.6.0 · CPU 8da4w · 6/6 valid | Not this engine’s entry | 158.8range156.6–161.8 | 262range261–272 | 977.1range941.2–980.8 | Not measured |
| LFM2.5 350M on M5 Max · 64 GiB 4-bit · Archive 256 input / 128 output · macOS 27 | |||||
| LFM2.5 350M on M5 Max · 64 GiB, 4-bit · Archive 256 input / 128 output · macOS 27: 1.6.0 · Metal mlx-4w-g64 · 6/6 valid | Check failed | 786.4range779.1–788.8 | 23.4range23.1–23.5 | 11,130range10,667–11,130 | Not measured |
| LFM2.5 350M on M5 Max · 64 GiB, 4-bit · Archive 256 input / 128 output · macOS 27: 1.6.0 · CPU 8da4w · 6/6 valid | Check failed | 335.5range334.2–340.5 | 85.8range85.7–86.7 | 2,977range2,943–3,012 | Not measured |
| LFM2.5 1.2B Instruct on M5 Max · 64 GiB 4-bit · Archive 2048 input / 128 output · macOS 27 Speed score 86, approximate comparison | |||||
| LFM2.5 1.2B Instruct on M5 Max · 64 GiB, 4-bit · Archive 2048 input / 128 output · macOS 27: 1.6.0 · Metal mlx-4w-g64 · 6/6 valid | Counted | 456.8range445.6–458.5 | 332range330–332 | 6,178range6,169–6,206 | Not measured |
| LFM2.5 1.2B Instruct on M5 Max · 64 GiB, 4-bit · Archive 2048 input / 128 output · macOS 27: 1.6.0 · CPU 8da4w · 6/6 valid | Not this engine’s entry | 121.6range116.8–128.5 | 2,405range2,357–3,052 | 851.6range671.3–869.3 | Not measured |
| LFM2.5 350M on M5 Max · 64 GiB 4-bit · Archive 2048 input / 128 output · macOS 27 | |||||
| LFM2.5 350M on M5 Max · 64 GiB, 4-bit · Archive 2048 input / 128 output · macOS 27: 1.6.0 · Metal mlx-4w-g64 · 6/6 valid | Check failed | 738.4range725.7–747.1 | 107range107–107 | 19,140range19,140–19,321 | Not measured |
| LFM2.5 350M on M5 Max · 64 GiB, 4-bit · Archive 2048 input / 128 output · macOS 27: 1.6.0 · CPU 8da4w · 6/6 valid | Check failed | 246.8range237.4–248.5 | 720range719–725 | 2,844range2,829–2,848 | Not measured |
| LFM2.5 1.2B Instruct on iPhone 12 · 4 GB 4-bit · Archive 256 input / 128 output · iOS 27 Speed score 94, approximate comparison | |||||
| LFM2.5 1.2B Instruct on iPhone 12 · 4 GB, 4-bit · Archive 256 input / 128 output · iOS 27: 1.6.0 · CPU 8da4w · 6/6 valid | Counted | 42.5range40.4–42.8 | 2,033range1,990–2,372 | 125.9range107.9–128.6 | Not measured |
| LFM2.5 350M on iPhone 12 · 4 GB 4-bit · Archive 256 input / 128 output · iOS 27 | |||||
| LFM2.5 350M on iPhone 12 · 4 GB, 4-bit · Archive 256 input / 128 output · iOS 27: 1.6.0 · CPU 8da4w · 6/6 valid | Check failed | 121.9range110.3–122.8 | 481range469–582 | 532.2range439.9–547.0 | Not measured |
| LFM2.5 1.2B Instruct on OnePlus 13 · 16 GB 4-bit · Archive 256 input / 128 output · Android 16 Speed score 100, approximate comparison | |||||
| LFM2.5 1.2B Instruct on OnePlus 13 · 16 GB, 4-bit · Archive 256 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 valid | Counted | 76.3range75.6–76.6 | Not measured | 559.0range545.8–566.4 | Not measured |
| LFM2.5 1.2B Instruct on Pixel 8 · 8 GB 4-bit · Archive 256 input / 128 output · Android 16 Speed score 100, approximate comparison | |||||
| LFM2.5 1.2B Instruct on Pixel 8 · 8 GB, 4-bit · Archive 256 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 valid | Counted | 31.7range29.4–32.3 | Not measured | 156.3range147.2–165.4 | Not measured |
| LFM2.5 1.2B Instruct on OnePlus 13 · 16 GB 4-bit · Archive 2048 input / 128 output · Android 16 Speed score 100, approximate comparison | |||||
| LFM2.5 1.2B Instruct on OnePlus 13 · 16 GB, 4-bit · Archive 2048 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 valid | Counted | 52.1range51.4–53.3 | Not measured | 480.0range474.8–549.4 | Not measured |
| LFM2.5 350M on OnePlus 13 · 16 GB 4-bit · Archive 256 input / 128 output · Android 16 Not counted | |||||
| LFM2.5 350M on OnePlus 13 · 16 GB, 4-bit · Archive 256 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 valid | Check failed | 203.0range199.7–204.5 | Not measured | 1,524range1,471–1,662 | Not measured |
| LFM2.5 350M on Pixel 8 · 8 GB 4-bit · Archive 256 input / 128 output · Android 16 Not counted | |||||
| LFM2.5 350M on Pixel 8 · 8 GB, 4-bit · Archive 256 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 valid | Check failed | 97.5range85.7–101.4 | Not measured | 329.1range293.2–365.2 | Not measured |
| LFM2.5 1.2B Instruct on iPhone 12 · 4 GB 4-bit · Archive 2048 input / 128 output · iOS 27 Not counted | |||||
| LFM2.5 1.2B Instruct on iPhone 12 · 4 GB, 4-bit · Archive 2048 input / 128 output · iOS 27: 1.6.0 · CPU 8da4w · 4/6 valid | Partial | 32.2range26.9–34.4 | 19,336range16,021–19,605 | 105.9range104.5–127.8 | Not measured |
| LFM2.5 350M on iPhone 12 · 4 GB 4-bit · Archive 2048 input / 128 output · iOS 27 Not counted | |||||
| LFM2.5 350M on iPhone 12 · 4 GB, 4-bit · Archive 2048 input / 128 output · iOS 27: 1.6.0 · CPU 8da4w · 6/6 valid | Check failed | 83.5range81.0–84.0 | 4,731range4,723–5,197 | 432.8range394.1–433.6 | Not measured |
| LFM2.5 350M on OnePlus 13 · 16 GB 4-bit · Archive 2048 input / 128 output · Android 16 Not counted | |||||
| LFM2.5 350M on OnePlus 13 · 16 GB, 4-bit · Archive 2048 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 valid | Check failed | 121.4range119.2–122.2 | Not measured | 1,510range1,503–1,527 | Not measured |
| LFM2.5 1.2B Instruct on Pixel 8 · 8 GB 4-bit · Archive 2048 input / 128 output · Android 16 Not counted | |||||
| LFM2.5 1.2B Instruct on Pixel 8 · 8 GB, 4-bit · Archive 2048 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 2/6 valid | Partial | 19.7range19.5–19.9 | Not measured | 213.7range212.3–215.1 | Not measured |
| LFM2.5 350M on Pixel 8 · 8 GB 4-bit · Archive 2048 input / 128 output · Android 16 Not counted | |||||
| LFM2.5 350M on Pixel 8 · 8 GB, 4-bit · Archive 2048 input / 128 output · Android 16: 1.6.0 · CPU 8da4w · 6/6 valid | Check failed | 53.0range52.7–53.7 | Not measured | 665.9range631.1–675.7 | Not measured |