fix(agent): per-provider streaming toggle for fallback providers#24621
Open
juniperbevensee wants to merge 4 commits into
Open
fix(agent): per-provider streaming toggle for fallback providers#24621juniperbevensee wants to merge 4 commits into
juniperbevensee wants to merge 4 commits into
Conversation
…chain Red-phase TDD tests for issue NousResearch#21522. Verifies that: - stream: false in fallback config sets _disable_streaming - stream: true (or omitted) keeps streaming enabled - Streaming state restores correctly across fallback chain - Runtime-detected _disable_streaming cleared by streaming fallback Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Read the `stream` field from fallback provider config and set/clear `_disable_streaming` accordingly when activating each fallback. Providers that don't support SSE streaming (custom proxies, Vertex AI REST) can now set `stream: false` to force non-streaming API calls. When advancing to a provider that supports streaming (stream: true or omitted), the flag is cleared — streaming state no longer sticks across the fallback chain. Fixes NousResearch#21522 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a new section to cli-config.yaml.example showing the fallback_providers list format and the stream: false option for custom endpoints that don't support SSE streaming. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a
streamconfig flag to fallback provider entries so custom endpoints that don't support SSE streaming can opt into non-streaming API calls.Problem: Custom fallback providers that return plain JSON instead of SSE get zero chunks from the OpenAI SDK's streaming parser. This causes valid responses to be silently discarded as "empty," triggering spurious retry loops and fallback cascades. See #21522 for details.
Solution: Read a per-provider
streamfield from fallback config. Whenstream: false, set_disable_streamingfor that provider. When advancing to a provider that supports streaming, clear the flag — streaming state no longer sticks across the fallback chain.Changes
run_agent.pystreamfrom fallback config in_try_activate_fallback(), set/clear_disable_streamingper providercli-config.yaml.exampleFallback Providerssection documenting thestreamflagtests/run_agent/test_fallback_streaming.pydocs/plan.mdExample usage
Design decisions
streamdefaults totruewhen omitted — zero behavioral change for existing configs"false","0","no","off"all treated as false (YAML parsers vary)_disable_streamingbased on its own config, so astream: falseprovider doesn't poison subsequent streaming-capable providersCloses #21522
Test plan
python -m pytest tests/run_agent/test_fallback_streaming.py -v— 7 tests, all passingpython -m pytest tests/run_agent/test_provider_fallback.py tests/run_agent/test_fallback_model.py -v— 53 existing tests, all passing (no regressions)