Notes from the on-device frontier.
Technical deep-dives into on-device AI, edge inference, and building ML-powered applications that run anywhere.
On-Device AI: The Complete Guide to Running ML Models Locally
Everything you need to know about running machine learning models directly on mobile and desktop devices — privacy, latency, cost benefits, and how to get started.
Streaming LLM Tokens from Rust to Flutter in Real-Time
How we bridge real-time token streaming from a Rust inference engine to Flutter's reactive UI using flutter_rust_bridge callbacks.
How We Made ONNX Runtime 6.8x Faster on Apple Silicon with CoreML
Real benchmarks showing when Apple's Neural Engine helps (and when it hurts). Lessons from optimizing ML inference across execution providers.
Whisper Speech Recognition as a Single Rust Binary
Build a Whisper-powered transcription tool as a single Rust binary. Candle for pure-Rust inference. No runtime dependencies, works airgapped.
Why We Chose Rust Over Python for ML Inference
Not training — inference. How Rust's zero-cost abstractions, lack of GIL, and FFI story make it the better choice for shipping ML to production devices.
On-Device AI for Mobile Apps: A Developer's Guide to iOS and Android
How to add on-device AI to iOS and Android apps — comparing CoreML, ONNX Runtime, TensorFlow Lite, and llama.cpp. Practical guide with model sizes, performance benchmarks, and integration patterns.
Edge AI: Why Edge-First Should Be Your Default Architecture
On-device AI should be the starting position, not the optimization. The industry defaults to cloud out of habit, not necessity.
Private AI: How to Run AI Models Without Sending Data to the Cloud
How to run AI models privately on-device — no cloud APIs, no data leaving the device. Covers GDPR, HIPAA, and SOC 2 compliance, private LLMs, and practical implementation patterns.
Building a Cross-Platform ML Inference SDK in Rust
How we built a single Rust core that powers ML inference across CLI, Flutter, Swift, Kotlin, and Unity — and the architectural decisions that made it possible.
How to Run LLMs Locally: A Complete Guide (2026)
How to run large language models locally on your laptop, desktop, or phone — llama.cpp, Ollama, ONNX Runtime, and on-device options compared. No cloud API needed.
One Codebase, Five Platforms: Shipping ML to Flutter, Swift, Kotlin, Unity & CLI
How UniFFI, flutter_rust_bridge, and C FFI let us write ML inference once in Rust and bind it to five platform SDKs. The cross-platform FFI playbook.
Run AI Models On-Device — Zero Config, Five Minutes
CLI, Rust, Flutter, Swift, Kotlin, Unity — run 25+ ML models on-device with one command. No tensor shapes, no preprocessing scripts.
Add Text-to-Speech to Your Flutter App in 15 Minutes
A step-by-step guide to adding high-quality, on-device TTS to a Flutter app using Xybrid and the Kokoro model. No cloud APIs, no API keys, no per-request costs.
Edge AI vs Cloud AI: When to Run Models On-Device
A practical decision framework for choosing between on-device and cloud-based AI inference, with cost analysis, comparison tables, and real-world use cases.
Building a Voice Agent That Runs Entirely On-Device
A step-by-step tutorial for building an on-device voice agent using Whisper, a local LLM, and Kokoro TTS — no cloud APIs, no internet required.