Skip to content

fix(agent): per-provider streaming toggle for fallback providers#24621

Open
juniperbevensee wants to merge 4 commits into
NousResearch:mainfrom
juniperbevensee:fix/fallback-provider-streaming-config
Open

fix(agent): per-provider streaming toggle for fallback providers#24621
juniperbevensee wants to merge 4 commits into
NousResearch:mainfrom
juniperbevensee:fix/fallback-provider-streaming-config

Conversation

@juniperbevensee

Copy link
Copy Markdown

Summary

Add a stream config flag to fallback provider entries so custom endpoints that don't support SSE streaming can opt into non-streaming API calls.

Problem: Custom fallback providers that return plain JSON instead of SSE get zero chunks from the OpenAI SDK's streaming parser. This causes valid responses to be silently discarded as "empty," triggering spurious retry loops and fallback cascades. See #21522 for details.

Solution: Read a per-provider stream field from fallback config. When stream: false, set _disable_streaming for that provider. When advancing to a provider that supports streaming, clear the flag — streaming state no longer sticks across the fallback chain.

Changes

File Change
run_agent.py Read stream from fallback config in _try_activate_fallback(), set/clear _disable_streaming per provider
cli-config.yaml.example New Fallback Providers section documenting the stream flag
tests/run_agent/test_fallback_streaming.py 7 new tests covering config parsing, state restoration, and edge cases
docs/plan.md Implementation plan

Example usage

fallback_providers:
  - provider: openai
    model: gpt-4o
  - provider: custom
    model: my-local-model
    base_url: http://my-proxy:8080/v1
    key_env: MY_PROXY_KEY
    stream: false    # This provider returns JSON, not SSE

Design decisions

  • Minimal change: 12 lines of production code — reads a single config field and sets/clears an existing flag
  • Backward compatible: stream defaults to true when omitted — zero behavioral change for existing configs
  • Handles YAML string coercion: "false", "0", "no", "off" all treated as false (YAML parsers vary)
  • State restoration: Each fallback activation explicitly sets or clears _disable_streaming based on its own config, so a stream: false provider doesn't poison subsequent streaming-capable providers

Closes #21522

Test plan

  • python -m pytest tests/run_agent/test_fallback_streaming.py -v — 7 tests, all passing
  • python -m pytest tests/run_agent/test_provider_fallback.py tests/run_agent/test_fallback_model.py -v — 53 existing tests, all passing (no regressions)
  • Verified defaults match previous behavior (stream: true when omitted)
  • Manual test with a non-SSE custom provider endpoint

juniperbevensee and others added 4 commits May 13, 2026 11:37
…chain

Red-phase TDD tests for issue NousResearch#21522. Verifies that:
- stream: false in fallback config sets _disable_streaming
- stream: true (or omitted) keeps streaming enabled
- Streaming state restores correctly across fallback chain
- Runtime-detected _disable_streaming cleared by streaming fallback

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Read the `stream` field from fallback provider config and set/clear
`_disable_streaming` accordingly when activating each fallback.
Providers that don't support SSE streaming (custom proxies, Vertex AI
REST) can now set `stream: false` to force non-streaming API calls.

When advancing to a provider that supports streaming (stream: true or
omitted), the flag is cleared — streaming state no longer sticks
across the fallback chain.

Fixes NousResearch#21522

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a new section to cli-config.yaml.example showing the
fallback_providers list format and the stream: false option
for custom endpoints that don't support SSE streaming.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@alt-glitch alt-glitch added type/feature New feature or request P2 Medium — degraded but workaround exists comp/agent Core agent loop, run_agent.py, prompt builder labels May 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent loop, run_agent.py, prompt builder P2 Medium — degraded but workaround exists type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Custom fallback providers fail silently when they don't support SSE streaming

2 participants