Streaming LLM Tokens from Rust to Flutter in Real-Time

You’re running an LLM locally. Tokens generate one at a time, each in ~20ms. You want the UI to update as each token arrives — the ChatGPT-style typing effect.

The problem: your inference engine is in Rust. Your UI is in Flutter. They’re in different worlds, connected by FFI.

Here’s how we solved real-time token streaming across the Rust-Flutter boundary in Xybrid.

---

The Challenge

Token-by-token streaming has strict requirements:

1. Low latency — each token must reach the UI within milliseconds of generation
2. Non-blocking — inference runs on a background thread; the UI stays responsive
3. Backpressure-aware — if the UI can’t keep up, we shouldn’t drop tokens
4. Clean shutdown — cancellation must propagate from Dart to Rust cleanly

A naive approach (poll for results, batch tokens) adds visible latency. Users notice when text appears in chunks instead of flowing smoothly.

The Architecture

Rust (background thread)              Flutter (main thread)
┌──────────────────────┐              ┌──────────────────────┐
│  llama.cpp / Candle  │              │    StreamBuilder      │
│  generates token     │              │    rebuilds UI        │
│         │            │              │         ▲             │
│         ▼            │              │         │             │
│  StreamingCallback   │──── FRB ────▶│    Dart Stream       │
│  (Rust closure)      │   (async)    │    (token events)    │
└──────────────────────┘              └──────────────────────┘

The key insight: flutter_rust_bridge (FRB) supports Dart Stream return types from Rust functions. This means Rust can push values to Dart asynchronously — exactly what we need.

Rust Side: The Streaming Callback

In xybrid-core, streaming is built around a simple callback:

pub type StreamingCallback = Box<dyn FnMut(PartialToken) -> bool + Send>;

pub struct PartialToken {
    pub text: String,
    pub is_final: bool,
}

The callback receives each token as it’s generated. Returning false signals cancellation — the engine stops generating.

The executor’s streaming API:

pub fn execute_streaming(
    &mut self,
    metadata: &ModelMetadata,
    input: &Envelope,
    callback: StreamingCallback,
) -> Result<Envelope> {
    // ... sets up inference, calls callback per token
}

Bridge Layer: FRB Stream

flutter_rust_bridge can turn a Rust function that calls a callback into a Dart Stream. Here’s the bridge function:

// In the FRB bridge layer
pub fn run_pipeline_streaming(
    yaml_content: String,
    input: ApiEnvelope,
    sink: StreamSink<ApiPartialToken>,
) -> Result<ApiEnvelope> {
    let callback: StreamingCallback = Box::new(move |token| {
        let api_token = ApiPartialToken {
            text: token.text.clone(),
            is_final: token.is_final,
        };
        sink.add(api_token).is_ok()
    });

    // Run inference with streaming
    let result = executor.execute_streaming(&metadata, &envelope, callback)?;
    Ok(result.into())
}

The StreamSink is FRB’s mechanism for sending values from Rust to Dart. Each sink.add() call pushes a value to the Dart Stream. If the Dart side cancels (drops the stream subscription), sink.add() returns Err, and we return false to stop generation.

Dart Side: Consuming the Stream

On the Flutter side, tokens arrive as a standard Dart Stream:

final stream = runPipelineStreaming(
  yamlContent: pipelineYaml,
  input: Envelope.text(text: userMessage),
);

String fullResponse = '';

await for (final token in stream) {
  fullResponse += token.text;
  setState(() {
    _responseText = fullResponse;
  });
}

That’s the core pattern. Each token triggers a setState, which rebuilds the text widget. The result is smooth, character-by-character text appearance.

Building a Chat UI

Here’s a more complete example — a chat screen with streaming responses:

class ChatScreen extends StatefulWidget {
  const ChatScreen({super.key});

  @override
  State<ChatScreen> createState() => _ChatScreenState();
}

class _ChatScreenState extends State<ChatScreen> {
  final _controller = TextEditingController();
  final _messages = <ChatMessage>[];
  bool _isGenerating = false;
  StreamSubscription? _subscription;

  Future<void> _send() async {
    final text = _controller.text.trim();
    if (text.isEmpty || _isGenerating) return;

    _controller.clear();

    // Add user message
    setState(() {
      _messages.add(ChatMessage(role: 'user', content: text));
      _messages.add(ChatMessage(role: 'assistant', content: ''));
      _isGenerating = true;
    });

    // Stream the response
    final stream = runPipelineStreaming(
      yamlContent: _pipelineYaml,
      input: Envelope.text(text: text),
    );

    _subscription = stream.listen(
      (token) {
        setState(() {
          _messages.last.content += token.text;
        });
      },
      onDone: () {
        setState(() => _isGenerating = false);
      },
      onError: (error) {
        setState(() {
          _messages.last.content += '
[Error: $error]';
          _isGenerating = false;
        });
      },
    );
  }

  void _cancel() {
    _subscription?.cancel(); // Propagates to Rust via sink.add() failure
    setState(() => _isGenerating = false);
  }

  @override
  void dispose() {
    _subscription?.cancel();
    _controller.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: const Text('Local LLM Chat'),
        actions: [
          if (_isGenerating)
            IconButton(
              icon: const Icon(Icons.stop),
              onPressed: _cancel,
            ),
        ],
      ),
      body: Column(
        children: [
          Expanded(
            child: ListView.builder(
              itemCount: _messages.length,
              itemBuilder: (context, index) {
                final msg = _messages[index];
                return ChatBubble(
                  role: msg.role,
                  content: msg.content,
                  isStreaming: index == _messages.length - 1 && _isGenerating,
                );
              },
            ),
          ),
          _buildInputBar(),
        ],
      ),
    );
  }
}

Key details:

• Cancel button: _subscription.cancel() in Dart → sink.add() returns error in Rust → callback returns false → inference stops. Clean cancellation across the FFI boundary.
• Last message mutation: We append tokens to the last message in the list, which triggers a rebuild of only that widget.
• Error propagation: Rust errors surface as stream errors in Dart.

GenerationConfig: Controlling Output

Control the LLM’s behavior with GenerationConfig:

final stream = runPipelineStreaming(
  yamlContent: pipelineYaml,
  input: Envelope.text(text: userMessage),
  generationConfig: GenerationConfig.creative(), // temperature=0.8, top_p=0.95
);

Presets:

GenerationConfig.greedy()   // temperature=0.0, deterministic
GenerationConfig.creative() // temperature=0.8, top_p=0.95
GenerationConfig(           // custom
  maxTokens: 512,
  temperature: 0.6,
  topP: 0.9,
)

These are always-available types — they compile even on platforms without LLM support, so your code doesn’t need platform conditionals.

Performance Notes

Measured with Qwen 3.5 0.8B (Q4_K_M quantization):

Device	Tokens/sec	First Token	Notes
MacBook Pro M2	~45 tok/s	~200ms	Metal GPU acceleration
iPhone 15 Pro	~30 tok/s	~350ms	Metal GPU
Pixel 8	~15 tok/s	~500ms	CPU only

The streaming overhead (FRB callback per token) adds <1ms per token — negligible compared to generation time.

Common Pitfalls

1. Don’t Rebuild the Entire List

// Bad: rebuilds every message on each token
setState(() => _messages = [..._messages]);

// Good: mutate the last message, use keys for targeted rebuilds
setState(() => _messages.last.content += token.text);

2. Handle Reasoning Tags

Some models (like Qwen 3.5) emit <think>...</think> blocks for chain-of-thought reasoning. Xybrid strips these automatically in streaming mode, so you don’t need to filter them in Dart.

3. Warmup Before First Chat

The first token has extra latency (model loading, shader compilation). Call warmup() during your loading screen:

final model = await Xybrid.model(modelId: 'qwen3.5-0.8b').load();
await model.warmup(); // Pre-load weights into memory

---

Summary

The streaming pipeline:

1. Rust generates tokens via llama.cpp or Candle
2. StreamingCallback fires per token, pushes to FRB StreamSink
3. FRB marshals the value across the FFI boundary
4. Dart Stream delivers it to Flutter’s reactive UI
5. setState triggers a rebuild, token appears on screen

Total overhead per token: <1ms. Users see smooth, real-time text generation from a model running entirely on their device.

---

Public SDK source: github.com/xybrid-ai/xybrid — check the Flutter binding docs for package usage.

---

Building something with local LLMs in Flutter? Share it in the comments!