Skip to content

perf: streaming throughput audit — measure and optimize token-by-token render path #313

@Hmbown

Description

@Hmbown

Problem

The agent loop streams LLM tokens one at a time, each triggering a transcript re-render through ratatui's full layout pass. On long transcripts with active streaming, the render rate can lag the network arrival rate — users see token bursts followed by pauses rather than smooth output.

The persistence-actor fix (#) eliminates one source of stalls (sync I/O); render-pass cost is the next axis. Worth profiling separately before guessing.

Proposed solution

  1. Profile first. cargo flamegraph --bin deepseek-tui while running a long-output session (--prompt "write a 5000-line markdown summary of every README in popular open-source rust projects"). Identify what's actually slow:
    • ratatui layout pass cost
    • text shaping / unicode-width recalc
    • syntax highlighting (if any in transcript)
    • serde serialization (if persistence still on hot path)
    • history Vec re-allocation
  2. Coalesce streamed chunks. Currently each token-event triggers an event-loop tick. Buffer for ~16ms (one frame at 60fps) then emit one render. Drops token rate jitter.
  3. Damage tracking. ratatui re-renders the entire screen every frame. Track which lines changed since last frame; only re-shape those. Faster on long transcripts.
  4. Lazy line shaping. Only run unicode-width / wrapping on visible lines. Off-screen lines are kept as raw strings until scrolled into view.

Acceptance criteria

  • Flamegraph captured and posted in this issue.
  • One concrete optimization landed with a before/after measurement (ms/frame on a 5000-token transcript).
  • No regression in render correctness (pinned by buffer-render tests).

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions