Skip to content

fix: aggregate streaming responses in auxiliary client#31598

Open
mrmign wants to merge 1 commit into
NousResearch:mainfrom
mrmign:fix/auxiliary-stream-aggregation
Open

fix: aggregate streaming responses in auxiliary client#31598
mrmign wants to merge 1 commit into
NousResearch:mainfrom
mrmign:fix/auxiliary-stream-aggregation

Conversation

@mrmign

@mrmign mrmign commented May 24, 2026

Copy link
Copy Markdown

Summary

Some custom providers return SSE streams even when stream=False is not honored. This caused title_generation and other auxiliary tasks to fail with errors like:

Auxiliary title_generation: LLM returned invalid response (type=str): 
'data: {"id": "chatcmpl-1b5552757d54", "object": "chat.completion.chunk"...'
Expected object with .choices[0].message

Changes

  • Add _aggregate_stream_response() — consumes SSE chunks into a standard .choices[0].message completion object
  • Detect streaming responses in _validate_llm_response():
    • Raw SSE strings containing data: prefix
    • Iterable objects without .choices attribute
  • Support OpenAI SDK ChatCompletionChunk objects and raw SSE text lines
  • Preserve reasoning_content from thinking models (e.g. Qwen, DeepSeek)
  • Add comprehensive test suite (13 tests)

How it works

When a custom provider ignores stream=False and returns SSE data:

  1. _validate_llm_response detects the response is a stream (string with data: or iterable without .choices)
  2. Transparently calls _aggregate_stream_response to consume all chunks
  3. Returns a SimpleNamespace matching the expected .choices[0].message shape
  4. Downstream code works unchanged

Test Plan

  • Unit tests pass (13/13)
  • Verified with custom provider that returns SSE by default
  • title_generation now succeeds instead of failing

Fixes #7264

Some custom providers return SSE streams even when stream=False is not
honored. This caused title_generation and other auxiliary tasks to fail
with 'LLM returned invalid response (type=str)' errors.

- Add _aggregate_stream_response() to consume SSE chunks into a standard
  .choices[0].message completion object
- Detect streaming responses in _validate_llm_response():
  * Raw SSE strings containing 'data:' prefix
  * Iterable objects without .choices attribute
- Support OpenAI SDK ChatCompletionChunk objects and raw SSE text lines
- Preserve reasoning_content from thinking models
- Add comprehensive test suite (13 tests)

Fixes NousResearch#7264
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent loop, run_agent.py, prompt builder labels May 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent loop, run_agent.py, prompt builder P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: auxiliary async model path can return invalid payloads and crash session_search with misleading AttributeError

3 participants