This is how Xybrid Bench measures on-device AI and how its leaderboard ranks what it measured. It is published before any result, so anyone can check how a rank will be produced before they see one. It is a draft: it will change as the first experiments are measured and reviewed, and every change is versioned (section 13).
1. Scope and status
The leaderboard has four tracks, each a perspective on the same experiments: Engines, Decision models, Real-time voice and Vision. Version 0.6 ranks one of them, Engines: which inference engine runs a model best on a device, measured as time to first token, prefill and decode speed, and peak memory while generating text with a large language model.
The other tracks are announced with what they will rank. Each gets its own sections here, and its own experiments, before it ranks anything.
The leaderboard shows an empty state wherever no experiment is published with its raw data. It never shows placeholder or estimated numbers.
2. Conflict of interest
Who runs this. Xybrid runs this benchmark. Xybrid builds an on-device AI SDK and its own inference engine, zzz, a closed-source engine that competes with the engines measured here. That is a conflict of interest, and every page says so where it matters.
zzz is not part of this version. No zzz experiment is measured or published under it. The engine catalog does not list zzz, and an experiment that names an engine missing from the catalog fails validation.
When zzz is added, these rules take effect, and they are part of the methodology from now on:
- The methodology for a comparison is published before its results.
- Every experiment is published: wins, losses, failures and unsupported configurations alike. A comparison is never published without the experiments it lost.
- Every engine gets the same tuning effort: each runs its maintainers’ documented recommended configuration for the device, and any extra tuning applied to one engine is applied to all, and recorded.
- Engine maintainers are invited to review their engine’s configuration, and their corrections are applied and logged (section 13).
- Every correction goes in a public changelog (section 13).
- The SHA-256 of the zzz binary measured is published with every zzz experiment, with a statement of how to obtain the same binary.
- Every page that shows zzz says that Xybrid builds it.
The leaderboard marks engines Xybrid builds through the engine catalog (affiliation: xybrid), not experiment by experiment, so the disclosure cannot be left off a single page or a single row.
3. Definitions
- Experiment: one engine, one model artifact (the exact weight files), one device and one workload, measured over several trials. One experiment is one JSON file.
- Trial: one measured generation: the prompt goes in, the output tokens come out. Every attempted trial is recorded, including invalid ones.
- Comparison: the experiments that can be compared directly, because they share the device, the operating system and its major version, the power state, the base model, the quantization class and the workload (section 7). Comparisons are worked out from the experiments; nobody picks them.
- Hardware cohort: devices of one platform and a similar hardware class, such as Apple silicon laptops or Android flagships. A cohort groups devices for navigation and filtering; experiments are only ever compared on the same device.
- Platform: desktop, mobile or embedded. Every cohort has one. The leaderboard’s platform views are Overall, Desktop and Mobile; embedded devices appear in Overall.
- Track: a perspective that turns experiments into a ranking: what a row is, what can be filtered, and what each objective means.
- Objective: what a ranking rewards: Balanced, Speed or Memory, defined per track (section 10).
4. Raw measurements and computed results
An experiment’s file holds what was set up and what was measured: the configuration, and for every trial its token counts, durations, memory reading and conditions. It never holds a rate, a summary, a score or a rank.
Everything else is computed from those raw values when the leaderboard is built: tokens per second, each experiment’s summary statistics, the comparisons, the scores and the ranks. The computation is versioned with this methodology. A change in how a score is computed is a new methodology version; it never edits a measurement, and the raw records stay byte for byte as published.
5. Metrics
Each metric is computed per trial from raw values and summarized over the valid trials of an experiment.
- Time to first token (milliseconds, lower is better): from the moment the harness submits the prompt to the moment it receives the first generated token. It includes prompt processing and the first decode step. It excludes model loading, which happens before the first trial.
- Prefill speed (tokens per second, higher is better): prompt tokens divided by the time spent processing the prompt, up to the first generated token.
- Decode speed (tokens per second, higher is better): generated tokens after the first, divided by the time from the first generated token to the last. The first token is excluded because its time belongs to prefill.
- Peak memory (bytes, shown in GiB, lower is better): the highest memory the engine’s process held during the trial.
Timing source. Each phase records its token count, its duration, and whether that duration comes from the engine’s own counters (engine) or from the harness’s clock around the engine’s API (harness). Formal measurements use an engine’s own counters when it reports them; an engine that doesn’t is timed by the harness clock from its streamed tokens, which includes the engine API’s transport, and its comparisons are approximate (section 7). Session 1 (methodology 0.3) required native counters. llama.cpp b12110 uses predicted_n − 1 over predicted_ms; its duration already excludes the first output token. MLX LM 0.32.2 uses generation_tokens − 1 over the duration recovered as generation_tokens / generation_tps. LiteRT-LM 0.19.1 reports a count and a rate per phase: prefill uses last_prefill_token_count over count / rate, and decode uses last_decode_token_count − 1 over last_decode_token_count / rate, since its decode count includes the first output token. On Android, LiteRT-LM 0.19.0’s litert_lm_advanced_main prints each phase’s count and duration: prefill uses its prefill turn’s count over its duration, and decode its decode turn’s count − 1 over its duration, for the same reason. On a Mac, from 0.5, each LiteRT-LM generation runs in a worker process of its own, which loads the engine before the harness submits the prompt. A reused engine keeps the previous sequence’s state (LiteRT-LM issue #3165), so in session 2, whose worker kept one engine for a turn’s warmups and trial, each output depended on the generations before it. Its prefill is therefore a just-loaded engine’s. In session 2, a turn’s first generation of LFM2.5 1.2B 8-bit prefilled 256 tokens at about 1,015 tokens per second and the later ones at about 1,430; 2,048 tokens ran at about 1,385 against 1,510. Decode was the same. ExecuTorch 1.6.0 reports millisecond timestamps: prefill uses num_prompt_tokens over prompt_eval_end − inference_start, and decode uses num_generated_tokens, which already excludes the first token, over inference_end − first_token. uzu 0.6.17 reports counts and rates like LiteRT-LM: tokens_count_input over tokens_count_input / prefill_tokens_per_second, and tokens_count_output − 1 over tokens_count_output / generate_tokens_per_second. Rapid-MLX 1.0.2 runs in process (its HTTP server takes neither token IDs nor ignore_eos) and reports its scheduler’s monotonic timings: prefill uses the prompt count over time_to_first_token_ms, the request’s arrival to its first token, and decode uses completion_tokens − 1 over mean_itl_ms × (completion_tokens − 1), the first token to the last. It runs one sequence with its prefix cache, speculative decoding and request-coalescing window off; that window would otherwise hold a lone request for about 8 ms before scheduling it. Ollama 0.40.13 passes on its llama-server’s counters, prompt_eval_count over prompt_eval_duration and eval_count − 1 over eval_duration; because its prompt_eval_count includes cached tokens, the llama-server log must also show the whole prompt evaluated. Magnitude 0.2.17 and ZML LLMD 20261106.0 are timed by the harness: prefill is the time to the first streamed content, and decode is completion_tokens − 1 over the first to the last streamed content. Native counters and Node monotonic receipt times are retained. Buffered first-token delivery leaves TTFT null and first-text timing diagnostic; ordinary transport overhead does not invalidate a trial. Speeds are computed as tokens divided by seconds, so a recorded rate can never disagree with its tokens and time.
On a phone (from 0.5), the engine runs on the phone and the harness on the Mac it is plugged into. The harness reaches llama-server through adb forward, so its time to first token includes the USB link. Prefill and decode use the engine’s own counters, timed on the phone. LiteRT-LM runs one process per generation, because a reused engine keeps the previous sequence’s state (LiteRT-LM issue #3165). It streams nothing to the harness, so its time to first token is null.
On an iPhone (from 0.5), llama.cpp, uzu and ExecuTorch run inside one app, the Xybrid Bench app, which the harness launches with Xcode’s devicectl from the Mac the iPhone is wired to. Each generation is one launch of the app, as LiteRT-LM is one process on Android: the app loads the engine, generates once, writes its result and exits, so a turn’s two warmups and its trial are three launches, and the first launch after the app is installed (which compiles the engines’ Metal shaders) is a warmup. The engines keep their Mac contracts: llama.cpp is given the frozen token IDs through its C API, with every end-of-generation token masked on a fixed-length run, and its phases come from llama_perf_context (n_p_eval over t_p_eval_ms, n_eval = N − 1 over t_eval_ms); uzu and ExecuTorch report as on the Mac. The app times each token as it receives it, on its own clock, so an iPhone’s time to first token runs from the engine call to the first token inside the app, without any transport.
Measuring memory differs by operating system, and an experiment records the method it used:
- macOS and iOS: the process’s physical footprint (
phys_footprint), sampled during the trial, plus the resident pages of the process’s mappings of its model files (macOS, from methodology 0.4). - Linux and Android: the resident set size high-water mark (
VmHWM) of the engine process, or of the app process when the engine runs inside it. - Windows: the process’s peak working set.
On a phone (from 0.5), peak memory is the engine process’s VmHWM. llama-server’s is reset at the start of each generation. LiteRT-LM’s process lives for one generation, so its peak includes loading the model. Neither is comparable until the engine is calibrated on the phone, and no phone calibration exists yet: phone memory is a diagnostic. On an iPhone it is the app process’s lifetime peak physical footprint (the kernel ledger’s phys_footprint peak), which includes loading the model; like footprint on the Mac, it leaves out weights an engine maps from its files (llama.cpp), so it is a diagnostic too. A model that exceeds the memory iOS gives the app ends the app: that generation is an error, and the result records it.
On the Mac (methodology 0.4), peak memory is the highest physical footprint, sampled every 50 ms during the measured generation, plus the resident pages of the process’s mappings of its model files, read just before and just after the generation (the larger reading). Footprint counts what the process allocates, Metal buffers included, but not clean file-backed pages. An engine that maps its weights from the model files (llama.cpp) holds them in those pages, while one that loads them into its own memory (MLX LM) holds them in footprint; the two terms count the weights once either way. Footprint alone would have shown llama.cpp at about a tenth of MLX LM’s memory for the same 1.2B BF16 weights. Model files are matched by device and inode. An engine that keeps its own copy of the model reports the files it loads, and that copy must match the pinned hashes. The residency walk runs outside the generation because it takes 15 ms or more.
A value is comparable only when the engine is calibrated on the device (bench/calibration/). Calibration is one documented run with the exact helper build that compares the measured parts with the engine’s own memory accounting (llama.cpp’s buffer sizes, MLX LM’s peak memory) and establishes whether the engine maps or loads its weights. Every trial must then show the weights once, where the calibration says: for an engine that maps them, at least 90% of the model files resident in its mappings; for one that loads them, at most 10%. Otherwise, or without calibration, the trial’s memory is not comparable. It stays in the evidence with its reason and is never estimated, and the experiment has no ranked memory. Session 1 (0.3) measured footprint alone, uncalibrated, and has none. The verifier recomputes every comparable value from the retained samples and readings. Cumulative process peaks and native engine memory counters remain diagnostics. The same interval retains its worst thermal state even when memory is unavailable. Unknown thermal/throttling evidence invalidates a formal trial.
6. Workloads and sampling
The first session uses Liquid LFM2.5 1.2B Instruct, BF16 and 4-bit, at 256 input / 128 output and 2048 input / 128 output, context 4096. llama.cpp b12110 and MLX LM 0.32.2 use pinned official Liquid exports. The canonical tokenizer revision is b9d816ab510f9a28dca37cb182e9da8b0226e13c.
A workload fixes everything two experiments must share to be compared: the prompt and its length, the output length, the context length, the batch size, the sampling and how the phases are timed. The CC0 source text and construction algorithm are hashed. One pinned canonical tokenizer and chat template freeze exact prompt token IDs and their SHA-256. The complete BOS, roles and assistant envelope are preserved, and only user content is truncated. From 0.4, prompts are prepared per model with its own tokenizer and chat template; when no cut at a token boundary reaches the exact length (LFM2.5 2.6B’s 2048-token needles), the same text is cut by characters. LFM2.5 2.6B is a pure reasoning model: its template always opens a thinking block and has no switch to turn it off, so its prompts close the block right after the template (</think>) for every engine alike, as Qwen3.8’s template does with thinking off. Without that, the frozen checks’ short answers can’t fit their output budget. Both engines consume those IDs directly, without an additional BOS or chat template. Defined vocabulary IDs must match; explicitly UNUSED padded GGUF rows are verified separately. Every experiment repeats its workload’s values, and validation checks that they match, so the workload’s id is the guarantee.
- Greedy decoding: temperature 0 and argmax sampling. Fresh request state, no prompt cache reuse, no speculation and one sequence.
- Fixed output length: exactly 128 output tokens with the shared EOS/EOG union, IDs 2 and 7, masked identically. Correctness checks restore normal unmasked stopping on that same union.
- Batch size 1: one request at a time, as an app on a phone would make.
- The same context length for every engine on a workload.
Validation also checks the measured phase counts: prefill must count the workload’s prompt tokens, and decode must count the requested output tokens minus the first token. An experiment explicitly allowing early end-of-sequence may record fewer decode tokens, but never more. Engine-reported phase counters must agree with the generation’s actual token counts.
7. Fairness rules
A comparison is only as fair as its weakest row. Every comparison follows these rules, and the leaderboard checks them in code:
- Same device, same operating system, same power state. Experiments on different devices, different OS major versions, or one on battery and one on external power are never in the same comparison. Mains and USB count as external power. For each source, low power mode on, off and unknown are separate states; enabling low power mode never merges battery and external-power experiments.
- Checkpoint provenance. Every artifact names a pinned publisher export and declared base model. The verifier checks publisher cards, architecture and vocabulary compatibility. Liquid’s current export cards do not pin their source checkpoint commit, so exact source-weight identity remains unverified and is disclosed on every configuration result. BF16 does not establish numerical identity. Engines need different file formats; the conversion tool, version and command are recorded for every artifact that wasn’t published upstream.
- The closest quantization class. Weights are grouped into classes (for example 4-bit, 8-bit, bf16). Experiments in different classes are never in the same comparison. Within a class, an experiment whose scheme differs from the one most of the comparison uses (for example
Q4_K_Magainst an MLX 4-bit format with group size 64) is compared approximately, and the difference is stated next to its numbers. - Each engine’s recommended configuration: the backend, thread count and flags its maintainers document for the device. The settings in effect are recorded as the engine reports them. On a phone, each engine runs at its own default thread count. The count in effect is recorded from the engine: llama.cpp’s
n_threadsfrom its log, LiteRT-LM’s from its executor settings. - Three blocks per quantization class and workload. Each block visits the group’s configurations in plan order, then in reverse, so each gets two turns a block and a linear drift weighs on all of them alike; with two engines this is ABBA. Each turn loads its engine, runs two unmeasured warmups, measures one trial and unloads; six trials per configuration. An engine that runs one process per generation (LiteRT-LM, section 5) starts and loads that process before each warmup and trial, outside the measured window, so its warmups don’t warm the trial’s engine. Session 1 (0.3) ran exactly llama.cpp as A and MLX LM as B. An experiment is complete only when all six formal trials are valid and there are no errors. Fixtures retain their illustrative five-trial schedules.
- Controlled conditions. At least 15 seconds between turns, AC power, low power mode off and nominal macOS thermal state. The actual hardware/OS is reconciled with the frozen device catalog. Each rerun receives a new session ID.
- Phones (from 0.5). The phone stays plugged into the Mac that runs the harness, by USB, for the whole session. Before the first turn, adb must reach it over USB, and it must report AC or USB power, not a wireless charger or a dock. Battery saver is off, and the phone is kept awake while plugged in. Each trial starts at thermal status 0, the nominal state of Android’s thermal service. Turns are at least 60 seconds apart. On a Pixel 8, one turn warmed the phone’s skin by about 4 °C. Sixty seconds brought it back; fifteen let it build up from turn to turn. Speed varied as much with a 15-second pause as with a 240-second one, so the pause is there for the heat, not for speed. Each plan states its pause, and the verifier checks every gap against it. A phone’s model, core count and OS build must match its catalog entry and plan exactly. Its chip must be the one the catalog names. Its usable RAM (
MemTotal, after what firmware reserves) must be at most an eighth below the catalog’s size. - iPhones (from 0.5). The iPhone stays wired to the Mac that runs the harness, by cable, for the whole session, and charges from it:
devicectlmust reach it over the wire, and the app must read its battery as charging or full. Low Power Mode is off. The iPhone is unlocked with Auto-Lock off, so the app runs in the foreground: each trial starts with the app active. Each trial starts at the nominal thermal state iOS reports (ProcessInfo.thermalState); fair, serious and critical count as throttling. Turns are at least 60 seconds apart, as on Android phones. Every engine runs at its default thread count. An iPhone’s product type (iPhone13,2 for the iPhone 12), core count and iOS build must match its catalog entry and plan exactly. iOS names no chip; the Metal device’s name (Apple A14 GPU) must name the catalog’s. Its RAM (hw.memsize) must be at most an eighth below the catalog’s size. - A thermal gate. Before each trial, the harness waits until the device is back within the gate stated in the experiment (for example a SoC temperature or the OS’s thermal state). A trial during which the device throttles is recorded as invalid, with its reason; so is a formal trial with a reading in no known state, which counts as unknown, never nominal.
- Outliers are never removed silently. A trial is excluded only when it is invalid for a recorded reason (throttling, an error, a wrong output). It stays in the record and on the experiment’s page.
Within a comparison, a different scheme in the same quantization class, a different effective KV cache precision, a different memory measurement method or a different thermal gate makes an experiment an approximate comparison: it is kept and ranked, and the difference is stated. Phases timed by the harness clock always do, even when most experiments in the comparison are harness-timed: they include the engine API’s transport, which differs from engine to engine. Anything else that differs puts the experiments in different comparisons, and nothing compares them directly.
8. Quality validation
The frozen sanity-v1 suite contains nine short known-answer checks, two mandatory exact 2048-token needle checks and one mandatory check covering 128-token continuation. In session 1 (0.3), an experiment needs at least 11/12 and every mandatory item. From 0.5, an experiment’s check is against its model’s BF16 reference: the items some BF16 configuration of the model passes, on any engine and device. The check asks whether the engine reproduces the model’s behavior, so an item every BF16 configuration fails is the model’s limit, not the engine’s, and is left out. The experiment needs all but one of the remaining items and every remaining mandatory item. LFM2.5 350M can’t continue the count at BF16, and every BF16 configuration misses the same four items, so its experiments need 7 of the other 8. A model with no checked BF16 configuration gets the 0.3 rule. So does a model whose BF16 configurations pass fewer than two items between them, since all but one of those would be none. A reference is only as independent as the engines behind it: LFM2.5 2.6B’s rests on llama.cpp and Ollama, which load the same GGUF file. The verifier recomputes every check from the retained outputs under the current rule; when a new BF16 session moves a reference, bench quality restamp recomputes the model’s other records. In 0.4 the bar was the best configuration of the experiment’s own session, so a check could fail for an item the model itself can’t do. 0.5 changed only this rule, so session 2’s records, measured under 0.4, moved to 0.5 together. Before the formal plan is frozen, every engine in it runs the suite on its reference artifact for the model, the highest-precision one it runs in the plan, without an engine error. Its answers are recorded, and each experiment’s own check decides whether it ranks. Session 1 (0.3) required both BF16 configurations at 12/12. Check prompts, native generations, complete outputs and suite bytes/hashes remain in the evidence. The evaluator recomputes match results from those outputs; it never trusts an engine’s pass flag. Uppercase and exact-copy tasks are case sensitive. The continuation check requires all 128 output tokens and the frozen counting prefix; it is a sanity check, not a general reasoning evaluation.
Only experiments whose output check passed are ranked. Failed or unchecked outcomes remain visible with their reason. Quantized configurations need not generate byte-identical outputs to one another. Every valid timed trial retains its output SHA-256.
9. Statistics and uncertainty
For every metric, an experiment’s summary over its valid trials holds: the number of trials, the median, the mean, the sample standard deviation (n − 1), the minimum, the maximum, and the 25th and 75th percentiles. Percentiles use linear interpolation between order statistics (Hyndman and Fan’s type 7, the default in R and NumPy). Summaries are rounded to three decimal places.
Experiment pages lead with the median and show the full range of valid trials next to it; their metric charts start at zero and show every trial. Missing values are never drawn as zero.
The overview. The leaderboard’s first page ranks first and explains second; filters live in the explorer. Its presentation rules do not change any score:
- Platform. Desktop and Mobile are offered when they have results. All devices is offered only once two engines complete both the desktop and the mobile basket (section 10, “Top Engines”), so every engine on it has results on both platforms. The explorer keeps the full Overall view. When a platform has comparisons on two or more OS families, each making up exactly one hardware cohort, an OS choice narrows Performance, Performance Breakdowns and Explore Benchmarks, and their explorer links carry that cohort as the hardware filter; Top Engines stays per platform and Coverage stays the same.
- Top Engines is the Speed objective’s ranking on the platform’s basket (section 10), shown out of 100 with each score’s range and its rank. It is not a combined score: time to first token and prefill are ranked under Speed, peak memory under Lowest Memory, and Balanced waits for comparable memory (section 5). An engine that completes the basket alone is listed where the ranking would be, unnumbered and without a score, until a second one does. Every engine short of the basket sits behind one collapsed row beneath, which starts open only when no engine completes the basket. Each of these shows its basket coverage, without an overall score: one segment per basket comparison, filled where it has a score, and the count (for example 12 / 12, or 2 / 4), which links to Coverage.
- Coverage shows each engine’s Speed score on each device: section 10’s ranking with the hardware filter set to that device, so each device is ranked on its own under the coverage rule. One column per device with comparisons, Desktop then Mobile, each platform’s label underlined in its own colour; each is headed by its OS’s mark and name, with its memory, OS versions and chip on hover; one row per engine measured on any of them. A cell is the score with the counted comparisons the engine has a score in, out of the device’s;
!with the same count for an engine short of the coverage rule there; its reason for any other engine without a score (no rival yet, unsupported, failed, not measured, not counted); a dash where it was never measured. It is the same under every platform tab, and the explorer shows it under its model, workload and quantization filters. A score belongs to its device: columns are never compared with each other. - Performance: Speed, with one tab per speed metric (Generation, the decode speed, by default; Response Time, the time to first token; Prompt Processing, the prefill speed), and Lowest Memory (peak memory). Each list shows raw medians from a single reference comparison per platform, the same one for every section: among the platform’s comparisons, counted ones first, then the most engines with a Speed score, then the shortest prompt, then by id. With an OS chosen, this reference rule runs over only that family’s comparisons on the platform. The rule never looks at the measured values. Within it, each engine is represented by its latest eligible version in its best configuration for that metric, it is ranked by its median, and a list ranks only with at least two measured engines; a tab or section with nothing to rank is left out. Under All devices, each platform keeps its own reference comparison. Raw values are never compared across comparisons. The reference comparison is named under the Performance heading and links to its page. Each section is a column chart, one column per engine in rank order from zero to the median, with the median on it and a whisker over the range of valid trials; under it, the same engines as a ranked list: rank, engine and median. A lower-is-better metric’s columns therefore rise from left to right.
- Performance Breakdowns shows one graph at a time:
- By Precision shows Generation in the reference comparison’s setup at each precision it was measured at: the comparisons with the same device, model, workload, OS, power and methodology that differ only in quantization class, widest precision first, at most four. Each is ranked on its own, as above; each engine’s column is its median there, and a precision it has no eligible result at has no column. It shows only with values at two precisions or more.
- By Model shows each engine’s per-model Speed score (section 10), one column per model with a ranking, smallest model first. A cell without a score (not measured, unsupported, failed, or short of the coverage rule) is left empty, never shown as zero; Coverage marks the last case
!instead. It is left out with fewer than two models or two engines.
- Bars and columns start at zero; the whisker on each bar, and on each column of a performance section, spans the range of valid trials. The overview shows ranks and medians; an approximate comparison (section 7) is marked ≈ on its comparison, experiment and engine pages, with the difference stated there. Engines that Xybrid builds are marked on every row that names them.
The explorer keeps every filter (model, hardware, workload, quantization), the score with each engine’s per-comparison evidence, and raw measurement tables. Its matched speed scatter shows each configuration’s median decode throughput on X and TTFT on Y; both axes include zero, throughput increases to the right, and TTFT decreases toward the top so the upper-right is better. Each engine has a page listing every experiment behind its rank, with medians and ranges, grouped by comparison.
10. Rankings
The leaderboard leads with rankings, so this section defines exactly how they are made. Rankings are computed from the raw experiments every time the site is built; none is stored or edited by hand.
What counts. An experiment counts in rankings when it is published, complete (all six formal trials valid, no errors) and its output check passed. Unsupported, failed, partial and unchecked experiments are published and shown in their comparison with their reason, but they never count, and they never count as zero.
One comparison at a time. Rankings never compare experiments across comparisons. Within a comparison, for each metric, an experiment’s ratio is its median against the best median among the comparison’s counted experiments: 1 for the best, 0.8 for 80% of it. For time and memory, where lower is better, the ratio is the best median divided by its own. Its range comes from its worst and its best valid trials, computed the same way.
Objectives. For the Engines track:
- Speed: the decode speed ratio.
- Memory: the peak memory ratio.
- Balanced: the geometric mean of the decode speed, time to first token and peak memory ratios, weighted equally.
An experiment without a metric an objective needs has no score for that objective, and says which metric is missing.
One engine, one entry. An engine is represented in a comparison by its latest version measured there, in its best configuration for the objective: the backend and settings, among that version’s counted experiments, with the highest score. Its other experiments stay in the comparison.
Which comparisons count. A comparison counts in a ranking once at least two engines have a score in it. A comparison with a single engine has nothing to compare and is listed as such.
Scores. An engine’s score in a view is the geometric mean of its scores in the counted comparisons it entered, shown out of 100: 100 means the best result in every comparison it entered. Its range is the geometric mean of the lower ends, and of the upper ends. A view with a single counted comparison also shows each engine’s own medians and ranges.
Coverage. An engine is ranked in a view only when it has scores in at least half of the view’s counted comparisons, and in at least two when the view has two or more. Otherwise it is listed under the ranking with its coverage, without a score. An engine missing from a comparison is never penalized as if it had scored zero; it is simply not ranked until it has enough results.
Ranks. An engine’s rank is one plus the number of engines with a higher score; within one comparison, one plus the number of engines with a better median. Only equal values share a rank. Ranges stay beside the values, in the data and on the bars; they never decide the order.
Overall. Overall merges platforms only once desktop and mobile each have at least one counted comparison under the view’s filters. Until then, Overall shows each platform’s own ranking separately, with the platform it is still missing. With a hardware filter, a view covers one platform, and this rule does not apply. The overview’s All devices is not this view: it ranks on the platform baskets (Top Engines, below).
Top Engines. From 0.6, the overview’s Top Engines ranks each platform on a fixed basket: a full grid of named devices, models, quantization classes and workloads that every ranked engine must complete, so every engine on a platform’s list is scored on the same comparisons. The baskets are in bench/data/baskets.json:
| Desktop | Mobile | |
|---|---|---|
| Devices | M5 Max 64 GiB (macOS 27) | iPhone 12 (iOS 27), OnePlus 13 and Pixel 8 (Android 16) |
| Models | LFM2.5 1.2B Instruct, Qwen3.8 27B | LFM2.5 350M, LFM2.5 1.2B Instruct |
| Quantization | 8-bit | 8-bit |
| Workloads | 256 and 2,048 tokens in, 128 out | the same two |
| Comparisons | 4 | 12 |
- What goes in. One model per size class the platform runs, two per platform, chosen without looking at speeds. One precision, 8-bit, where a model’s largest file is 3 to 9% bigger than its smallest across formats (15 to 33% at 4-bit, and about twice for ExecuTorch’s 4-bit files), so the basket ranks engines rather than quantization formats. Both workloads. Every device measured on the platform, listed by name with the OS major version it is ranked under, on external power as formal sessions run (section 7). Comparisons are keyed by device, OS major version and power state (section 7), so a basket cell is one comparison, and a newly catalogued device, or a run after an OS update, changes no ranking until the basket names it on purpose.
- Who is ranked. Each comparison is scored as above. An engine is ranked on a platform only with a score in every comparison of its basket; a combination nobody has measured yet is still required. A platform is ranked once two engines complete its basket. Every other engine measured on the basket’s devices is listed with the basket comparisons it has a score in, zero included.
- Scores. An engine’s score is the geometric mean of its basket scores. A basket is a full grid, so this weighs devices equally, then each device’s models, then each model’s precisions and workloads. Ratios stay against the best counted median in each comparison, whether or not that engine completes the basket, so a ranked engine can stay below 100 on every comparison.
- All devices ranks the engines that complete both baskets, on the geometric mean of their two platform scores, once there are two of them.
- Scope. Each basket reads the comparisons on its devices under the current methodology; while none of them is measured under it, their comparisons under earlier versions. The basket decides Top Engines and nothing else: an engine’s per-model score (the By Model breakdown) is the view ranking above with the model filter set, and its per-device score (Coverage) the same with the hardware filter set to the device, so both keep every measured engine and every comparison, as do the comparison pages and the explorer.
Filters and views. The platform views and the filters (model, hardware cohort or device, workload, quantization class) select whole comparisons; a ranking is always computed from the comparisons in its view. Every selection is part of the page’s address, so any view can be shared and reproduced.
11. Unsupported, failed and unknown
These are different things, and the leaderboard keeps them apart:
- Unsupported: the engine does not support the configuration (for example, no backend for the device’s NPU). It is published as a result, with the reason and, where there is one, a link to the evidence. From 0.4, the engine reports it, or the plan declares it with its reason when the engine can’t run the frozen protocol as specified (for example, a chat API that re-renders a template the prompts deviate from). Either way, the configuration is planned, run and recorded like the others. From 0.5, an engine that errors in qualification on the device, on a check or while loading (for example, the operating system ends the app that loads the model), qualifies as unsupported on that artifact: the qualification keeps its outputs, the reason is its first error, and the session records that engine’s configurations on that artifact as unsupported with that reason and measures everything else. Up to 0.4, an engine error fails qualification.
- Failed: the configuration was attempted and produced errors. The errors are published.
- Unknown: nobody has tried. Unknown is never shown as unsupported, and never as zero.
- Partial: some trials were valid but fewer than planned. Partial experiments are shown with their status next to their numbers, and are not ranked.
12. Reproducibility and raw artifacts
Schema 0.3.0, harness 0.4.0 (session 1: 0.3.0). Every measured record carries its session ID. Every experiment records what is needed to repeat it: the engine’s version, commit and binary hash, how it was built, the exact weight files with their SHA-256 hashes, the prompt’s hash, the device’s OS build, the harness version, and the commands that ran it. Each session’s finalized JSONL journal is kept with its SHA-256, which every outcome’s record names. Until an experiment is published, its journal is held in a private dataset, never in the repository, and every verification checks its bytes against that hash. A published experiment’s journal is served byte-for-byte at /leaderboard/raw/<session>.jsonl; session 1 keeps one copy per outcome at /leaderboard/raw/<id>.jsonl. Each outcome references the complete session journal, including all failures, unsupported turns, native counters, dependency manifests, actual hardware doctor, provenance report and qualification evidence. Interrupted .partial journals cannot qualify for publication. Paths and account identifiers in diagnostics are scrubbed; model and check outputs remain intact. The Node-only verifier reconciles the complete schedule, every normalized trial, actual check outputs and all session outcomes before dataset selection or a production build.
Each experiment’s record is served byte for byte as committed, at /leaderboard/experiments/<id>.json, so its hash matches the repository’s. The index of every published experiment is at /leaderboard/data.json, the computed comparisons and rankings at /leaderboard/rankings.json, and the record format at /leaderboard/schema.
13. Publication, review, corrections and versioning
- An experiment starts as a draft, produced by the harness. Publishing it is always a person’s edit, never automatic.
- A published session requires every outcome together, including failures and unsupported configurations, with common reviewers and initial publication time. Each reviewer must be a named human and must approve the complete session before publication. The operator may also review; disclose both roles in the review package and preserve their identity in the operator and reviewer fields. An experiment needs a reviewer, the time it was published, the commands that reproduce it, and a raw artifact with its hash. The site’s build refuses anything less.
- A published experiment is never edited silently. A correction adds a changelog entry with its date and reason. An experiment that turns out to be wrong is withdrawn: it keeps its page, marked as withdrawn with the reason, and leaves every comparison and ranking.
- The methodology is versioned. Every experiment states the version it was measured under, and every computed result is computed under the current version. A change that would alter results gets a new version, and experiments measured under an older version are labelled as such. A change that only recomputes results from the retained evidence, as 0.5’s output check and 0.6’s Top Engines basket do, moves every experiment it applies to at once; their measurements are unchanged. Rules for a kind of device that has no result yet are added within a version, since they change no existing result. Phones’ rules (sections 5 and 7) were added to 0.5 this way.
14. Limitations
- One device is one device. A result on one phone does not predict another phone, or the same phone on another OS version.
- Six trials establish a range, not a distribution. Ranks follow medians even where the ranges of neighbours overlap, so the ranges beside them tell how far apart two ranks really are.
- An aggregate score summarizes comparisons that differ in device, model and workload. It says how an engine does relative to the others where they met, not how fast it is in absolute terms. The comparisons behind every score are one click away.
- Engines are moving targets. Every result is tied to an engine version and commit, and goes stale as engines improve.
- Fixed-length greedy generation is a controlled workload, not a model of every app. Real apps also load models, stream partial output and share the device with other work.
- Throughput is not quality. Section 8 is a guard against fast wrong output, not a measure of how good a model’s answers are.
- Xybrid runs this benchmark and builds a competing engine. The rules in section 2 limit that conflict; they don’t remove it.