feat(providers): streaming-native chat-client stack with composable routing#1313
Merged
Aaronontheweb merged 7 commits intoJun 4, 2026
Merged
Conversation
…outing Redesign the IChatClient stack so the streaming-vs-non-streaming transport distinction no longer leaks into callers, and resilience/observability are no longer entangled with transport. Supersedes netclaw-dev#1289 and netclaw-dev#1309. Transport: Netclaw now issues only streaming requests. RetryingChatClient retries the streaming path (pre-first-chunk; a mid-stream failure propagates so emitted output is never duplicated), reusing RetryPolicy. The auxiliary callers (title generation, memory distillation, compaction observation, memory curation, memory extraction) aggregate the stream via the existing StreamingResponseReader (empty-Messages guard + diagnostics) and read response.Text, instead of the non-streaming GetResponseAsync path that the OpenAI Codex backend rejects (400 'Stream must be set to true') and that drops reasoning content on some providers. Composition: per-(provider,model) pipelines are composed with Microsoft.Extensions.AI ChatClientBuilder (Logging -> Retry -> VendorOptions -> raw) in PipelineChatClientFactory; the .Use ordering is asserted by test. Routing: a router selects which composed pipeline to invoke per call. RoutingChatClient walks an ordered candidate list, so failover is 'more than one candidate' and single-provider alerting is the terminal failure of a one-entry walk -- absorbing and deleting FailoverChatClient, AlertingChatClientDecorator, NetclawChatClientProvider and ResilientChatClientProviderDecorator. The router takes a ChatRoutingContext so future per-session/per-provider routing slots in as a new policy; today's RoleBasedFailoverRouter reproduces the prior wiring. Logging: LoggingChatClient is now stateless (the per-session-intended delta/cumulative token counters on the process singleton are gone) and tags log lines with the ambient session id via SessionLoggingScope, keyed 'SessionId' to match the WithContext('SessionId', ...) key used across actors so one Seq attribute correlates actor and chat-client logs. Verified: full solution build (0 warnings); Netclaw.Daemon.Tests 668/668; Netclaw.Actors.Tests 2186/2186; slopwatch clean; file headers present.
…ndex, streaming-failover tests Self-review of the stack redesign found three issues: 1. RetryingChatClient.BackoffAsync logged retry warnings without the SessionId scope, so the hottest failure path reached Seq uncorrelated while LoggingChatClient/RoutingChatClient logs carried it. Open the scope around the warning. 2. ChatRoutingContext.AttemptIndex was structurally unfulfillable: the per-role RoutingChatClient context is frozen at construction and the router is called once per call (not per candidate), so it could never be non-zero. Removed it; SessionId remains the genuine per-session-routing seam. 3. The streaming terminal-unreachable branch of RoutingChatClient (last candidate fails pre-first-chunk -> provider.unreachable + rethrow) and single-candidate / streaming-cancellation behavior were untested. Added three streaming tests.
…t retry Reconcile the two pre-first-chunk retry layers (review follow-up): the actor's per-turn streaming retry and the transport's RetryingChatClient overlapped on the main session loop. Make the transport the single owner. RetryPolicy.ShouldRetry now also recognizes the curated ProviderException (StatusCode 408/429/5xx, walking inner exceptions) that provider transports throw — previously only a raw HttpRequestException was retried, so the transport retry barely fired for real provider 429/5xx. That was the broader coverage the actor's LlmFailureClassifier.IsTransientStreaming provided. The transport RetryPolicy is now configured from Session:Tuning:StreamingRetryPolicy (resolves the prior TODO; preserves the knob), making RetryingChatClient the single, configurable, uniform retry layer for every caller (main loop + sidecars). Remove the actor per-turn retry: the LlmCallFailed pre-content retry block, the RetryLlmCallAfterBackoff message+handler, _streamingRetryAttempt, the streaming-retry timer, and the now-unused IsTransientStreaming. Context-overflow rollback and terminal FailCurrentTurn handling are unchanged. Re-home coverage: the actor-level Streaming_retry_recovers/_exhaustion integration tests are removed (the actor no longer retries); a ProviderException(502) recover case is added to RetryingChatClientTests at the transport layer. Verified: full build (0 warnings); Daemon 672/672; Actors 2184/2184; slopwatch + headers clean.
…aming-only The streaming-native chat-client stack converts every auxiliary caller (title generation, memory extraction, compaction) to the streaming transport, so no code path invokes the non-streaming GetResponseAsync on the Codex client anymore. The StreamingOnlyChatClient wrapper (added in 0e5ba60 to serve those calls via streaming under the hood) is now dead; the Codex 'Stream must be set to true' 400 is structurally unreachable. A comment on the OAuth path records why no special handling is needed.
Aaronontheweb
commented
Jun 4, 2026
| /// Netclaw issues) and aggregates with the same empty-response fallback, so the | ||
| /// caller can read <c>result.Response.Text</c> without an empty-Messages guard. | ||
| /// </summary> | ||
| public static Task<StreamReadResult> ReadAsync( |
| /// Timer-fired message that triggers an LLM call retry after exponential backoff. | ||
| /// Carries the attempt number for observability logging. | ||
| /// </summary> | ||
| internal sealed record RetryLlmCallAfterBackoff(int Attempt) : INoSerializationVerificationNeeded; |
Collaborator
Author
There was a problem hiding this comment.
this has been moved to the IChatClient layer
Collaborator
Author
There was a problem hiding this comment.
all LLM retry logic for connectivity issues lives in the IChatClient layer now.
Collaborator
Author
There was a problem hiding this comment.
simplified this to remove all mutable state
This was referenced Jun 4, 2026
Aaronontheweb
added a commit
that referenced
this pull request
Jun 6, 2026
The streaming-native chat-client stack (#1313) introduced the routing seam — IChatClientRouter.Route(ChatRoutingContext) — as groundwork for concurrent multi-model / multi-provider routing, but the seam comments described that future work without linking the tracking issue. Cite #648 at the two authoritative seam locations (the ChatRoutingContext type and the IChatClientRouter interface) so whoever picks up #648 lands on the seam they extend. Comment-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Redesigns the
IChatClientstack so the streaming-vs-non-streaming transport distinction stops leaking into callers, and resilience/observability stop being entangled with transport. The OpenAI Codex400 {"detail":"Stream must be set to true"}on memory formation was a symptom of this structural debt (the streaming-only conclusion had already been reached twice independently — main loop andSubAgentActor).Supersedes #1289 and #1309.
Why streaming-everywhere
All 8 providers stream cleanly; non-streaming is the unreliable path — Codex rejects it, and some SDK paths drop reasoning content in it. Standardizing on streaming as the only transport Netclaw issues collapses the provider-capability question entirely (we only wanted capabilities to know who supports non-streaming).
The redesign
Transport — Netclaw issues only streaming requests.
RetryingChatClientnow retries the streaming path, pre-first-chunk (reusingRetryPolicy+ the failover iterator pattern). Once a chunk is emitted a failure propagates, so streamed output is never duplicated by a restart. Retry is now a property of the stack, not of the transport a caller happened to pick.StreamingResponseReader(empty-Messages guard + diagnostics) and readresponse.Text, replacing the non-streamingGetResponseAsyncpath. This also closes theresponse.TextvsMessages[^1].Textinconsistency and the empty-stream throw.Composition —
ChatClientBuilder. Per-(provider, model)pipelines are composed viaMicrosoft.Extensions.AI.ChatClientBuilder(Logging → Retry → VendorOptions → raw) inPipelineChatClientFactory. The.Useordering is asserted by a test (it flips silently on a bad bump).Routing — failover is the first policy, not a special case. A router selects which composed pipeline to invoke per call.
RoutingChatClientwalks an ordered candidate list, so failover = a list of length > 1 and single-provider alerting = the terminal failure of a length-1 walk — this absorbs and deletesFailoverChatClient,AlertingChatClientDecorator,NetclawChatClientProvider, andResilientChatClientProviderDecorator. The router takes aChatRoutingContext, so future per-session / per-provider routing slots in as a new policy without a rewrite (built the seam, not the feature). Today'sRoleBasedFailoverRouterreproduces the prior wiring.Logging — stateless + session-correlated.
LoggingChatClientis now stateless (the per-session-intendeddelta/cumulativetoken counters that sat on a process singleton are gone — real per-session usage already lives inLlmSessionActor). It tags log lines with the ambient session id viaSessionLoggingScope, keyedSessionIdto match theWithContext("SessionId", …)key actors already use, so one Seq attribute correlates actor and chat-client logs.Testing
Netclaw.Daemon.Tests: 668/668Netclaw.Actors.Tests: 2186/2186dotnet slopwatch analyze: clean; file headers presentPipelineChatClientFactoryorder guard,RoutingChatClientfailover/unreachable walk,RoleBasedFailoverRoutercandidate selection,LoggingChatClientsession-scope + stateless usage.Notes / not in scope
ChatRoutingContext) is in; the policy is a future drop-in.Microsoft.Extensions.AIcore package (daemon-only) forChatClientBuilder. Note: it ships its ownLoggingChatClient, which collides with Netclaw's in test files importing both namespaces — resolved with ausingalias (production uses same-namespace precedence).