---
title: "Streaming LLM Tokens from Rust to Flutter in Real-Time - Xybrid Blog"
description: "How we bridge real-time token streaming from a Rust inference engine to Flutter's reactive UI using flutter_rust_bridge callbacks."
url: "https://www.xybrid.ai/blog/streaming-llm-tokens-rust-to-flutter"
---

# Streaming LLM Tokens from Rust to Flutter in Real-Time

Engineering · July 7, 2026 · Glenn Sonna · 6 min read

How we bridge real-time token streaming from a Rust inference engine to Flutter's reactive UI using flutter_rust_bridge callbacks.

You're running an LLM locally. Tokens generate one at a time, each in ~20ms. You want the UI to update as each token arrives — the ChatGPT-style typing effect.

The problem: your inference engine is in Rust. Your UI is in Flutter. They're in different worlds, connected by FFI.

Here's how we solved real-time token streaming across the Rust-Flutter boundary in [Xybrid](https://github.com/xybrid-ai/xybrid).

---

## The Challenge

Token-by-token streaming has strict requirements:

1. **Low latency** — each token must reach the UI within milliseconds of generation
2. **Non-blocking** — inference runs on a background thread; the UI stays responsive
3. **Backpressure-aware** — if the UI can't keep up, we shouldn't drop tokens
4. **Clean shutdown** — cancellation must propagate from Dart to Rust cleanly

A naive approach (poll for results, batch tokens) adds visible latency. Users notice when text appears in chunks instead of flowing smoothly.

## The Architecture

```text
Rust (background thread)              Flutter (main thread)
┌──────────────────────┐              ┌──────────────────────┐
│  llama.cpp / Candle  │              │    StreamBuilder      │
│  generates token     │              │    rebuilds UI        │
│         │            │              │         ▲             │
│         ▼            │              │         │             │
│  StreamingCallback   │──── FRB ────▶│    Dart Stream       │
│  (Rust closure)      │   (async)    │    (token events)    │
└──────────────────────┘              └──────────────────────┘
```

The key insight: **flutter_rust_bridge (FRB) supports Dart `Stream` return types from Rust functions.** This means Rust can push values to Dart asynchronously — exactly what we need.

## Rust Side: The Streaming Callback

In `xybrid-core`, streaming is built around a simple callback:

```rust
pub type StreamingCallback = Box<dyn FnMut(PartialToken) -> bool + Send>;

pub struct PartialToken {
    pub text: String,
    pub is_final: bool,
}
```

The callback receives each token as it's generated. Returning `false` signals cancellation — the engine stops generating.

The executor's streaming API:

```rust
pub fn execute_streaming(
    &mut self,
    metadata: &ModelMetadata,
    input: &Envelope,
    callback: StreamingCallback,
) -> Result<Envelope> {
    // ... sets up inference, calls callback per token
}
```

## Bridge Layer: FRB Stream

flutter_rust_bridge can turn a Rust function that calls a callback into a Dart `Stream`. Here's the bridge function:

```rust
// In the FRB bridge layer
pub fn run_pipeline_streaming(
    yaml_content: String,
    input: ApiEnvelope,
    sink: StreamSink<ApiPartialToken>,
) -> Result<ApiEnvelope> {
    let callback: StreamingCallback = Box::new(move |token| {
        let api_token = ApiPartialToken {
            text: token.text.clone(),
            is_final: token.is_final,
        };
        sink.add(api_token).is_ok()
    });

    // Run inference with streaming
    let result = executor.execute_streaming(&metadata, &envelope, callback)?;
    Ok(result.into())
}
```

The `StreamSink` is FRB's mechanism for sending values from Rust to Dart. Each `sink.add()` call pushes a value to the Dart `Stream`. If the Dart side cancels (drops the stream subscription), `sink.add()` returns `Err`, and we return `false` to stop generation.

## Dart Side: Consuming the Stream

On the Flutter side, tokens arrive as a standard Dart `Stream`:

```dart
final stream = runPipelineStreaming(
  yamlContent: pipelineYaml,
  input: Envelope.text(text: userMessage),
);

String fullResponse = '';

await for (final token in stream) {
  fullResponse += token.text;
  setState(() {
    _responseText = fullResponse;
  });
}
```

That's the core pattern. Each token triggers a `setState`, which rebuilds the text widget. The result is smooth, character-by-character text appearance.

## Building a Chat UI

Here's a more complete example — a chat screen with streaming responses:

```dart
class ChatScreen extends StatefulWidget {
  const ChatScreen({super.key});

  @override
  State<ChatScreen> createState() => _ChatScreenState();
}

class _ChatScreenState extends State<ChatScreen> {
  final _controller = TextEditingController();
  final _messages = <ChatMessage>[];
  bool _isGenerating = false;
  StreamSubscription? _subscription;

  Future<void> _send() async {
    final text = _controller.text.trim();
    if (text.isEmpty || _isGenerating) return;

    _controller.clear();

    // Add user message
    setState(() {
      _messages.add(ChatMessage(role: 'user', content: text));
      _messages.add(ChatMessage(role: 'assistant', content: ''));
      _isGenerating = true;
    });

    // Stream the response
    final stream = runPipelineStreaming(
      yamlContent: _pipelineYaml,
      input: Envelope.text(text: text),
    );

    _subscription = stream.listen(
      (token) {
        setState(() {
          _messages.last.content += token.text;
        });
      },
      onDone: () {
        setState(() => _isGenerating = false);
      },
      onError: (error) {
        setState(() {
          _messages.last.content += '\n[Error: $error]';
          _isGenerating = false;
        });
      },
    );
  }

  void _cancel() {
    _subscription?.cancel(); // Propagates to Rust via sink.add() failure
    setState(() => _isGenerating = false);
  }

  @override
  void dispose() {
    _subscription?.cancel();
    _controller.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: const Text('Local LLM Chat'),
        actions: [
          if (_isGenerating)
            IconButton(
              icon: const Icon(Icons.stop),
              onPressed: _cancel,
            ),
        ],
      ),
      body: Column(
        children: [
          Expanded(
            child: ListView.builder(
              itemCount: _messages.length,
              itemBuilder: (context, index) {
                final msg = _messages[index];
                return ChatBubble(
                  role: msg.role,
                  content: msg.content,
                  isStreaming: index == _messages.length - 1 && _isGenerating,
                );
              },
            ),
          ),
          _buildInputBar(),
        ],
      ),
    );
  }
}
```

Key details:

- **Cancel button**: `_subscription.cancel()` in Dart → `sink.add()` returns error in Rust → callback returns `false` → inference stops. Clean cancellation across the FFI boundary.
- **Last message mutation**: We append tokens to the last message in the list, which triggers a rebuild of only that widget.
- **Error propagation**: Rust errors surface as stream errors in Dart.

## GenerationConfig: Controlling Output

Control the LLM's behavior with `GenerationConfig`:

```dart
final stream = runPipelineStreaming(
  yamlContent: pipelineYaml,
  input: Envelope.text(text: userMessage),
  generationConfig: GenerationConfig.creative(), // temperature=0.8, top_p=0.95
);
```

Presets:

```dart
GenerationConfig.greedy()   // temperature=0.0, deterministic
GenerationConfig.creative() // temperature=0.8, top_p=0.95
GenerationConfig(           // custom
  maxTokens: 512,
  temperature: 0.6,
  topP: 0.9,
)
```

These are **always-available types** — they compile even on platforms without LLM support, so your code doesn't need platform conditionals.

## Performance Notes

Measured with Qwen 3.5 0.8B (Q4_K_M quantization):

| Device | Tokens/sec | First Token | Notes |
|--------|-----------|-------------|-------|
| MacBook Pro M2 | ~45 tok/s | ~200ms | Metal GPU acceleration |
| iPhone 15 Pro | ~30 tok/s | ~350ms | Metal GPU |
| Pixel 8 | ~15 tok/s | ~500ms | CPU only |

The streaming overhead (FRB callback per token) adds &lt;1ms per token — negligible compared to generation time.

## Common Pitfalls

### 1. Don't Rebuild the Entire List

```dart
// Bad: rebuilds every message on each token
setState(() => _messages = [..._messages]);

// Good: mutate the last message, use keys for targeted rebuilds
setState(() => _messages.last.content += token.text);
```

### 2. Handle Reasoning Tags

Some models (like Qwen 3.5) emit `<think>...</think>` blocks for chain-of-thought reasoning. Xybrid strips these automatically in streaming mode, so you don't need to filter them in Dart.

### 3. Warmup Before First Chat

The first token has extra latency (model loading, shader compilation). Call `warmup()` during your loading screen:

```dart
final model = await Xybrid.model(modelId: 'qwen3.5-0.8b').load();
await model.warmup(); // Pre-load weights into memory
```

---

## Summary

The streaming pipeline:

1. **Rust** generates tokens via llama.cpp or Candle
2. **StreamingCallback** fires per token, pushes to FRB `StreamSink`
3. **FRB** marshals the value across the FFI boundary
4. **Dart Stream** delivers it to Flutter's reactive UI
5. **setState** triggers a rebuild, token appears on screen

Total overhead per token: &lt;1ms. Users see smooth, real-time text generation from a model running entirely on their device.

---

**Public SDK source**: [github.com/xybrid-ai/xybrid](https://github.com/xybrid-ai/xybrid) — check the Flutter binding docs for package usage.

---

*Building something with local LLMs in Flutter? Share it in the comments!*

---

Tags: flutter, rust, ai, tutorial

## Keep reading

- [Vendoring llama.cpp in a Rust Workspace (Lessons Learned)](https://www.xybrid.ai/blog/vendoring-llamacpp-in-rust-workspace) — The practical pain of embedding a C++ inference engine in a Rust monorepo — cross-compilation, Android fp16, think-tag stripping, and when to give up on upstream.
- [Building a Cross-Platform ML Inference SDK in Rust](https://www.xybrid.ai/blog/building-cross-platform-ml-inference-sdk-in-rust) — How we built a single Rust core that powers ML inference across CLI, Flutter, Swift, Kotlin, and Unity — and the architectural decisions that made it possible.
- [One Codebase, Five Platforms: Shipping ML to Flutter, Swift, Kotlin, Unity & CLI](https://www.xybrid.ai/blog/one-codebase-five-platforms-ml-flutter-swift-kotlin-unity) — How UniFFI, flutter_rust_bridge, and C FFI let us write ML inference once in Rust and bind it to five platform SDKs. The cross-platform FFI playbook.
