Vendoring llama.cpp in a Rust Workspace (Lessons Learned)

We needed local LLM inference in our Rust SDK. The obvious choice: llama.cpp — fast, well-maintained, supports every quantization format.

The less obvious part: actually integrating it into a Rust workspace that cross-compiles to iOS, Android, macOS, Linux, and Windows.

Here’s what we learned vendoring llama.cpp into Xybrid.

---

Why Vendor?

We considered three approaches:

Approach	Pros	Cons
System dependency	Simple, always latest	Users must install it, version mismatch hell
Git submodule	Pinned version, upstream updates	Submodule pain, build integration complexity
Vendor (copy into repo)	Full control, reproducible builds	Manual updates, larger repo

We chose vendoring because:

1. Reproducible builds: CI and users get exactly the same llama.cpp version
2. Patching: We can fix cross-compilation issues without waiting for upstream
3. Feature gating: We only compile the parts we need via Cargo features

The trade-off is manual updates. When we need a new model architecture (like Qwen 3.5), we update the vendored copy.

The Build System: build.rs

llama.cpp is pure C/C++ with no build system requirement (CMake is optional). We compile it directly from build.rs:

// build.rs (simplified)
fn main() {
    let mut build = cc::Build::new();

    build
        .cpp(true)
        .file("vendor/llama.cpp/llama.cpp")
        .file("vendor/llama.cpp/ggml.c")
        .file("vendor/llama.cpp/ggml-alloc.c")
        .file("vendor/llama.cpp/ggml-backend.c")
        .include("vendor/llama.cpp/");

    // Platform-specific backends. In a build script `cfg!()` reflects the
    // HOST, not the target you're compiling for — read CARGO_CFG_TARGET_OS
    // so cross-compiles (e.g. building for iOS from a Linux CI box) pick the
    // right backend.
    let target_os = std::env::var("CARGO_CFG_TARGET_OS").unwrap();
    if target_os == "macos" || target_os == "ios" {
        build.file("vendor/llama.cpp/ggml-metal.m");
        println!("cargo:rustc-link-framework=Metal");
        println!("cargo:rustc-link-framework=MetalPerformanceShaders");
    }

    build.compile("llama");
}

The cc crate handles cross-compilation toolchains automatically — it reads CC, CXX, and target-specific env vars from Cargo.

Lesson 1: Android Needs +fp16

Our first Android build failed immediately:

error: instruction requires: fullfp16

The culprit: llama.cpp uses ARM FP16 instructions for performance. The Android NDK compiler doesn’t enable them by default on aarch64-linux-android.

Fix:

# .cargo/config.toml
[target.aarch64-linux-android]
rustflags = ["-C", "target-feature=+fp16"]

This tells LLVM to emit FP16 instructions. But wait — what about older devices without FP16 support?

Turns out, every 64-bit ARM Android device supports FP16 (it’s mandatory in ARMv8-A). The NDK is just conservative with defaults. The flag is safe.

However, the gemm-f16 crate (used by Candle, another dependency) had the same issue. We had to ensure the flag applied globally, not just to llama.cpp.

Lesson 2: Think Tags in Reasoning Models

We added Qwen 3.5 support (a reasoning model) and immediately hit a UX problem. The model outputs:

<think>
The user is asking about weather. I should provide a helpful response
about current conditions. Let me think about what information would
be most useful...
</think>

The weather today is sunny with a high of 72°F.

Users don’t want to see the <think> block. We needed to strip it — but in both batch and streaming modes.

Batch Mode (Easy)

fn strip_think_tags(text: &str) -> String {
    // Remove <think>...</think> blocks (including nested)
    let re = Regex::new(r"(?s)<think>.*?</think>s*").unwrap();
    re.replace_all(text, "").trim().to_string()
}

Streaming Mode (Hard)

In streaming mode, tokens arrive one at a time. The <think> block might span hundreds of tokens. We can’t wait for the closing </think> because that defeats the purpose of streaming.

Our approach: a state machine that buffers tokens inside think blocks:

enum ThinkState {
    Normal,
    MaybeOpening(String),  // Buffering potential "<think>"
    InsideThink,           // Suppressing tokens
    MaybeClosing(String),  // Buffering potential "</think>"
}

When we see <, we start buffering. If it becomes <think>, we enter suppression mode. If it becomes anything else, we flush the buffer as normal tokens. Same logic for </think>.

This adds zero latency to normal tokens and correctly handles:

• Partial tag matches (<th followed by inking — not a tag, flush both)
• Multiple think blocks in one response
• Think blocks that span the entire response

Lesson 3: Updating Vendored Code

When Qwen 3.5 came out, our vendored llama.cpp didn’t support its architecture. The update process:

1. Identify the minimum commit in llama.cpp that adds support
2. Copy the relevant files (we don’t vendor the entire repo — just core inference files)
3. Re-apply our patches (we maintain a small patchset for cross-compilation fixes)
4. Test on all platforms

# Our update script (simplified)
cd /tmp && git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && git checkout <commit>

# Copy core files
cp llama.cpp ggml*.c ggml*.h ggml*.m /path/to/vendor/llama.cpp/

# Re-apply patches
cd /path/to/project
git apply patches/llama-cpp-*.patch

We went from a Jan 30, 2026 snapshot to Mar 11, 2026 — specifically to get qwen35 architecture support. The diff was manageable because we only vendor the inference core, not the entire project (examples, server, tests, etc.).

Lesson 4: Metal Shader Compilation

On macOS and iOS, llama.cpp uses Metal for GPU acceleration. The Metal shaders (ggml-metal.metal) need to be compiled and accessible at runtime.

Two approaches:

1. Embed at compile time — converts the shader to a C string embedded in the binary
2. Load at runtime — reads from a file path next to the binary

We use option 1 (embed) because it’s simpler for distribution — no shader files to bundle:

// In ggml-metal.m (our patch)
#define GGML_METAL_EMBED_LIBRARY 1
// This makes the shader source a compile-time constant

The trade-off: the binary is ~200KB larger. Worth it for zero-configuration deployment.

Lesson 5: Wrapping C++ in Safe Rust

llama.cpp’s API is C-style (it provides a C header). Our Rust wrapper:

pub struct LlamaCppModel {
    ctx: *mut llama_context,
    model: *mut llama_model,
}

impl LlamaCppModel {
    pub fn load(path: &str, config: &GenerationConfig) -> Result<Self> {
        let c_path = CString::new(path)?;
        let params = unsafe { llama_model_default_params() };

        let model = unsafe { llama_load_model_from_file(c_path.as_ptr(), params) };
        if model.is_null() {
            return Err(anyhow!("Failed to load model: {}", path));
        }

        // ... context creation, error handling
        Ok(Self { ctx, model })
    }

    pub fn generate(&mut self, prompt: &str, callback: StreamingCallback) -> Result<String> {
        // Tokenize, sample, decode — calling back per token
    }
}

// Safety: LlamaCppModel owns the pointers and frees them on drop
impl Drop for LlamaCppModel {
    fn drop(&mut self) {
        unsafe {
            llama_free(self.ctx);
            llama_free_model(self.model);
        }
    }
}

// Send + Sync are safe because we don't share the context across threads
unsafe impl Send for LlamaCppModel {}

Key patterns:

• Owned pointers with Drop for cleanup — no leaks
• Result return types for every fallible operation
• Error messages that help: we include the file path, architecture hints, and point to XYBRID_LLAMACPP_VERBOSITY for debugging

Lesson 6: Error Diagnostics Matter

The worst llama.cpp error is a silent failure — the model loads but produces garbage. We added diagnostics:

Err(anyhow!(
    "Failed to load GGUF model: {}

     Architecture: {}

     Hint: Set XYBRID_LLAMACPP_VERBOSITY=2 for detailed llama.cpp output

     Hint: Ensure the GGUF file matches a supported architecture",
    path,
    detect_architecture(path).unwrap_or("unknown".into()),
))

The XYBRID_LLAMACPP_VERBOSITY env var controls llama.cpp’s internal logging, which is invaluable for debugging quantization mismatches and architecture issues.

The Feature Flag

llama.cpp is entirely behind a feature flag:

[features]
llm-llamacpp = ["dep:llama-cpp-bindings"]

When disabled, the LLM types (GenerationConfig, ChatMessage) still compile — they’re in an always-available module. Only the implementation is gated:

#[cfg(feature = "llm-llamacpp")]
mod llamacpp_backend;

// These are always available:
pub struct GenerationConfig { ... }
pub struct ChatMessage { ... }

This prevents downstream crates (Flutter bindings) from needing conditional compilation for type definitions.

Was Vendoring Worth It?

Yes. The alternatives had worse trade-offs:

• A system dependency would require users to install llama.cpp (and the right version) on every platform
• A git submodule would add CI complexity and still require patching
• Building against a pre-compiled library would lose Metal/ANE optimization on Apple platforms

The cost is ~50MB added to the repo and occasional manual updates. For reproducible, cross-platform builds with full optimization — that’s a fair trade.

---

See the implementation: github.com/xybrid-ai/xybrid

---

Vendoring C/C++ in Rust? Share your war stories in the comments.