Every ML model you’ve ever deployed had the same hidden problem: the inference script.
That Python file with hardcoded tensor shapes, tokenizer paths, magic normalization constants, and platform-specific workarounds. The one that “just works” on the original developer’s machine and breaks everywhere else.
We replaced it with a JSON file.
The Problem
Here’s what typical model deployment looks like:
# inference.py — "just copy this file with the model"
import onnxruntime as ort
import numpy as np
from tokenizer import load_tokenizer # custom module, good luck finding it
session = ort.InferenceSession("model.onnx")
tokenizer = load_tokenizer("vocab.json")
def run(text):
tokens = tokenizer.encode(text)
tokens = np.array([tokens], dtype=np.int64)
# Magic: pad to 128 because the model was exported that way
if tokens.shape[1] < 128:
tokens = np.pad(tokens, ((0,0), (0, 128 - tokens.shape[1])))
outputs = session.run(None, {"input_ids": tokens})
# Magic: take argmax of last dim, skip first token, decode
predicted = np.argmax(outputs[0], axis=-1)[0][1:]
return tokenizer.decode(predicted)This script contains critical deployment knowledge:
- Input tensor name is
input_ids - Input must be
int64 - Padding to 128 is required
- Output needs argmax → skip first → decode
None of this is discoverable from the ONNX file alone. Lose the script, lose the ability to run the model.
The Solution: model_metadata.json
Every model in Xybrid ships with a model_metadata.json that declares everything needed to run it:
{
"model_id": "wav2vec2-base-960h",
"version": "1.0",
"description": "Speech recognition using CTC decoding",
"execution_template": {
"type": "SimpleMode",
"model_file": "model.onnx"
},
"preprocessing": [
{
"type": "AudioDecode",
"sample_rate": 16000,
"channels": 1
}
],
"postprocessing": [
{
"type": "CTCDecode",
"vocab_file": "vocab.json",
"blank_index": 0
}
],
"files": ["model.onnx", "vocab.json"],
"metadata": {
"task": "speech-recognition",
"architecture": "wav2vec2",
"language": "en"
}
}The runtime reads this file, builds the execution pipeline, and runs it. No inference script. No custom code. No implicit knowledge.
How It Works
The TemplateExecutor (Xybrid’s core execution engine) processes the metadata in three phases:
Input Envelope
│
▼
┌─────────────┐
│ Preprocess │ AudioDecode: WAV bytes → f32 samples at 16kHz
└──────┬──────┘
│
▼
┌─────────────┐
│ Model │ ONNX inference: samples → logits
└──────┬──────┘
│
▼
┌─────────────┐
│ Postprocess │ CTCDecode: logits → text via vocab.json
└──────┬──────┘
│
▼
Output EnvelopeIn Rust:
let metadata: ModelMetadata = serde_json::from_str(
&std::fs::read_to_string("model_metadata.json")?
)?;
let mut executor = TemplateExecutor::with_base_path("./model-dir");
let output = executor.execute(&metadata, &input)?;Three lines. Works for ASR, TTS, classification, embeddings — any model type. The metadata file is the only thing that changes.
The Schema
Execution Templates
The execution_template field defines how the model file is loaded and run:
// Standard ONNX model
{ "type": "SimpleMode", "model_file": "model.onnx" }
// GGUF model (llama.cpp)
{
"type": "Gguf",
"model_file": "model.gguf",
"context_length": 4096
}
// Whisper (Candle, autoregressive)
{
"type": "WhisperMode",
"model_file": "model.safetensors",
"tokenizer_file": "tokenizer.json",
"config_file": "config.json"
}The template type tells the executor which backend to use (ONNX Runtime, llama.cpp, Candle) and what files are needed.
Preprocessing Steps
Available preprocessing steps and what they do:
| Step | Input | Output | Config |
|---|---|---|---|
AudioDecode | WAV bytes | f32 samples | sample_rate, channels |
Phonemize | Text | Token IDs | backend, tokens_file, dict_file |
Tokenize | Text | Token IDs | vocab_file, max_length |
Example — TTS preprocessing:
{
"preprocessing": [
{
"type": "Phonemize",
"backend": "MisakiDictionary",
"tokens_file": "tokens.txt"
}
]
}This converts "Hello world" → [phoneme token IDs] using a built-in phonemizer. No external dependencies (no espeak-ng installation needed).
Postprocessing Steps
| Step | Input | Output | Config |
|---|---|---|---|
CTCDecode | Logits tensor | Text | vocab_file, blank_index |
TTSAudioEncode | f32 waveform | WAV bytes | sample_rate, apply_postprocessing |
ArgMax | Logits tensor | Class index | — |
Voice Configuration (TTS-Specific)
TTS models can declare available voices:
{
"metadata": {
"voices": {
"voices_file": "voices.bin",
"format": "Float32Array",
"embedding_dim": 256,
"voices": [
{ "id": "af_heart", "name": "Heart", "language": "en-us" },
{ "id": "af_sky", "name": "Sky", "language": "en-us" }
],
"default_voice": "af_heart"
}
}
}The runtime loads the voice embedding from voices.bin and passes it as an additional model input. Consumers just pick a voice by name.
Why Not ONNX Model Zoo Format? Or MLflow?
ONNX Model Zoo has model cards, but they’re documentation, not execution configuration. They don’t tell you how to preprocess inputs or postprocess outputs.
MLflow has model signatures, but they describe tensor shapes, not the pipeline around the model. You still need custom code for tokenization, audio decoding, etc.
model_metadata.json is execution-complete: everything from “raw user input” to “usable output” is declared. The runtime needs zero custom code per model.
| ONNX Zoo | MLflow | model_metadata.json | |
|---|---|---|---|
| Tensor shapes | Yes | Yes | Implicit |
| Preprocessing | No | No | Yes |
| Postprocessing | No | No | Yes |
| Multi-file models | No | Partial | Yes (files array) |
| Voice/variant config | No | No | Yes |
| Runtime-executable | No | Partial | Yes |
Real Examples
ASR (wav2vec2)
{
"model_id": "wav2vec2-base-960h",
"execution_template": { "type": "SimpleMode", "model_file": "model.onnx" },
"preprocessing": [
{ "type": "AudioDecode", "sample_rate": 16000, "channels": 1 }
],
"postprocessing": [
{ "type": "CTCDecode", "vocab_file": "vocab.json", "blank_index": 0 }
]
}Input: WAV audio bytes → Output: transcribed text.
TTS (Kokoro)
{
"model_id": "kokoro-82m",
"execution_template": { "type": "SimpleMode", "model_file": "model.onnx" },
"preprocessing": [
{ "type": "Phonemize", "backend": "MisakiDictionary", "tokens_file": "tokens.txt" }
],
"postprocessing": [
{ "type": "TTSAudioEncode", "sample_rate": 24000, "apply_postprocessing": true }
]
}Input: text string → Output: WAV audio bytes.
LLM (Qwen 3.5)
{
"model_id": "qwen3.5-0.8b",
"execution_template": {
"type": "Gguf",
"model_file": "Qwen3.5-0.8B-Q4_K_M.gguf",
"context_length": 4096
},
"preprocessing": [],
"postprocessing": [],
"metadata": {
"task": "text-generation",
"architecture": "qwen35",
"backend": "llamacpp"
}
}Input: text prompt → Output: generated text. No pre/postprocessing needed — the GGUF backend handles tokenization internally.
Extending the Schema
Need a new preprocessing step? Add a variant to the enum and implement the handler:
// In steps.rs
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type")]
pub enum PreprocessingStep {
AudioDecode { sample_rate: u32, channels: u16 },
Phonemize { backend: PhonemizerBackend, tokens_file: String },
Tokenize { vocab_file: String },
// Add your new step:
MelSpectrogram { n_fft: usize, hop_length: usize, n_mels: usize },
}Implement the handler, and every model that uses MelSpectrogram in its metadata will automatically work across all platforms.
The Payoff
With model_metadata.json:
- Model authors declare how their model runs, once
- Runtime developers implement preprocessing/postprocessing steps, once
- App developers write
executor.execute(&metadata, &input)— no model-specific code - New platforms (Flutter, Swift, Kotlin) get every model for free — they use the same metadata
One JSON file. Zero inference scripts. Every model is self-describing and portable.
See it in action: github.com/xybrid-ai/xybrid — browse fixtures/models/ for real model_metadata.json examples.
Thoughts on the schema design? Things we should add? Let us know in the comments.