Summary

Closes #146 (assistant phase 2a). The chat UI now streams replies token-by-token instead of blocking on the full LLM response. Replaces the 3-10s wait + sudden full-message paint with a Claude/ChatGPT-style live typing animation backed by real content.

Anthropic ships native SSE; OpenAI / Google / Ollama fall back to single-chunk emission of the existing sync response so the API contract is uniform regardless of which provider is in front. Backward-compatible: legacy JSON callers (no Accept: text/event-stream) still work unchanged.

What landed

LlmClient.generateStream

```ts
async *generateStream(prompt, options): AsyncIterable
```

Yields chunk events as text arrives, then exactly one done event with the assembled full content + provider/model/latency.

Provider-chain semantics differ slightly from sync generate() and the difference is load-bearing:

• Pre-first-chunk failures fall through to the next provider (same as sync). No partial output has been emitted yet, so silent retry is safe.
• Mid-stream failures do NOT fall through. Once a provider has committed by yielding even one chunk, the user has already seen text — silently retrying a different provider would produce duplicate output.

Native Anthropic streaming

packages/llm-client/src/providers/anthropic.ts:streamGenerate consumes /v1/messages with stream: true. Buffers events across reader.read() boundaries (a chunk can split mid-event). Tolerates the full Anthropic event taxonomy by ignoring everything except content_block_delta events with delta.type === 'text_delta'.

The parser is exported for unit tests so we don't have to mock fetch + Response for every behavior.

Universal fallback for non-streaming providers

makeFallbackStream wraps the sync generate as a single-chunk async iterable. Lets the chain expose a uniform streaming interface even for providers we haven't implemented native streaming for yet. Adding native streaming for OpenAI / Google / Ollama is just dropping a new streamGenerate into their provider module — no changes elsewhere.

AssistantService.replyStream

Same EnrichmentRequest semantics as reply(): when supplied AND a ContextBuilder is wired, the rendered twin/memory context block is prepended to the system prompt for this request.

Three terminal events:

• done (success) — fullContent + metadata
• error (mid-stream failure with at least one chunk landed) — partialContent + message
• Pre-first-chunk failures escape via throw — caller turns into HTTP 502 (same as sync reply())

POST /api/assistant/messages SSE branch

Branches on the Accept header. text/event-stream triggers streaming; everything else falls through to the existing sync JSON response.

Wire format:

```
event: thread
data: {"id":"...","isNew":true}

event: user
data: {...userMessage row...}

event: chunk
data: {"content":"Hello"}

event: chunk
data: {"content":" world"}

event: done
data: {...assistantMessage row...}
```

User message persists FIRST (before the LLM call) so it survives a provider outage. Assistant message persists AFTER the stream closes, using the accumulated full content. If the persist fails, done still fires with persistFailed: true so the client can warn — the audit-trail loss is recoverable, the user-facing regression isn't.

X-Accel-Buffering: no keeps nginx from buffering end-to-end. Mid-flight disconnect detection via res.writableEnded so the loop exits cleanly when the user navigates away — saves the provider's tokens.

Web client streams progressively

sendAssistantMessageStream uses fetch + manual SSE parsing (not EventSource — that's GET-only). Per-event callbacks (onThread, onUserMessage, onChunk, onDone, onError).

pages/assistant.js:handleSend updates a single bubble's text in place via data-streaming-id direct DOM update — no full repaint per token. Preserves textarea focus and scroll position during the stream. Typing dots fire only while sending AND before the first chunk; once the streaming bubble appears it shows the live text.

Mid-stream errors render the partial content + an error caveat in separate bubbles so the user sees both what landed and what went wrong.

Safety / invariants

• Trust boundary — Gmail / sender / user strings still flow through escapeHtml before landing in innerHTML. The streaming bubble's incremental DOM updates use textContent (not innerHTML), which is XSS-safe by construction.
• Provider commit — once a provider yields a chunk, the chain refuses to silently retry a different provider. Comment-documented at LlmClient.generateStream and tested.
• Audit trail integrity — assistant message persists after stream closes; persist-failure surfaces persistFailed: true to the client rather than silently losing the row.

Test plan

• pnpm build — 20 packages green
• pnpm test — 40 packages green; 24 new tests on top of the existing assistant + llm-client suites
• pnpm lint — clean
• Manual: configure Anthropic provider, navigate to #/assistant, send a long prompt, watch tokens land progressively
• Manual: configure OpenAI-only, verify single-chunk fallback still works (whole reply lands at once)
• Manual: with both providers configured, kill network mid-stream, verify partial content + error caveat render
• Manual: with no AI provider configured, verify the streaming endpoint returns 409 cleanly (pre-stream check)

Phase 2+ still in flight

• Assistant phase 2c: action-intent routing through @skytwin/decision-engine #148 action-intent routing through the decision engine (biggest feature, biggest safety surface)
• Assistant phase 3: multi-turn LlmClient API + drop prompt-flattening workaround #149 native multi-turn LlmClient API (refactor prerequisite for Assistant phase 2c: action-intent routing through @skytwin/decision-engine #148 — drops the User: / Assistant: prompt-flattening workaround)

🤖 Generated with Claude Code