Skip to content

[Parser] Migrate response api streaming to unified parser#38755

Merged
DarkLight1337 merged 3 commits into
vllm-project:mainfrom
sfeng33:parser
Apr 8, 2026
Merged

[Parser] Migrate response api streaming to unified parser#38755
DarkLight1337 merged 3 commits into
vllm-project:mainfrom
sfeng33:parser

Conversation

@sfeng33

@sfeng33 sfeng33 commented Apr 1, 2026

Copy link
Copy Markdown
Collaborator

Co-authored with @qandrew

Purpose

Move the reasoning/tool-call streaming orchestration logic out of OpenAIServingResponses and into a new parse_delta() method in the unified parser. No behaviour change.

  • Add StreamState dataclass to hold per-stream mutable state (reasoning_ended, previous_text, previous_token_ids, etc.)
  • Add parse_delta() to DelegatingParser that orchestrates reasoning extraction, reasoning-end detection, and tool-call extraction using internal StreamState
  • Simplify _process_simple_streaming_events in serving.py — remove ~80 lines of inline parser branching, replace with single parse_delta() call

Test Plan

vllm serve Qwen/Qwen3-8B \
    --max-model-len 8192 \
    --enforce-eager \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes

 curl -s http://localhost:8000/v1/responses \
    -H "Content-Type: application/json" \
    -d '{
    "model": "Qwen/Qwen3-8B",
    "stream": false/true,
    "input": "What is the weather in San Francisco?",
    "tools": [
      {
        "type": "function",
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {"type": "string", "description": "City name"}
          },
          "required": ["location"]
        }
      }
    ]
  }'

@sfeng33 sfeng33 marked this pull request as ready for review April 1, 2026 21:20
@mergify mergify Bot added the frontend label Apr 1, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Apr 1, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the streaming event processing logic by introducing a parse_delta method in the Parser class to orchestrate reasoning and tool extraction using a new StreamState dataclass. Feedback points out that the parser initialization in serving.py lacks the required tools argument, which is necessary for tool call extraction. Additionally, the parse_delta implementation contains a bug where reasoning deltas are overwritten by tool deltas when both are present in a single chunk, leading to potential data loss.

Comment thread vllm/entrypoints/openai/responses/serving.py Outdated
Comment thread vllm/parser/abstract_parser.py
@sfeng33

sfeng33 commented Apr 1, 2026

Copy link
Copy Markdown
Collaborator Author

@qandrew @chaunceyjiang @bbrowning

@chaunceyjiang chaunceyjiang added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 2, 2026
@chaunceyjiang chaunceyjiang self-assigned this Apr 2, 2026
aarnphm
aarnphm approved these changes Apr 6, 2026
View reviewed changes

@aarnphm aarnphm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. will run ready tag for this.

@sfeng33

sfeng33 commented Apr 6, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the review!

@sfeng33
Move to unified parser
9453045 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: Andrew Xia <axia@meta.com>
sfeng33 added 2 commits April 7, 2026 18:05
@sfeng33
Comply with PR 38860
3e5fa1d 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
@sfeng33
Merge branch 'main' into parser
6df2539
@DarkLight1337 DarkLight1337 merged commit 927975e into vllm-project:main Apr 8, 2026
50 checks passed
@sfeng33 sfeng33 deleted the parser branch April 8, 2026 02:11
@bfroemel

bfroemel commented Apr 9, 2026

Copy link
Copy Markdown

Hi @sfeng33 and @chaunceyjiang did you see/consider my bug fixing effort here: #38227 ? (I was hoping that vllm moves forward with these fixes...)

mtparet pushed a commit to blackfuel-ai/vllm that referenced this pull request Apr 9, 2026
@sfeng33 @mtparet
[Parser] Migrate response api streaming to unified parser (vllm-proje…
773bf98 
…ct#38755)

Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: Andrew Xia <axia@meta.com>
@bfroemel bfroemel mentioned this pull request Apr 9, 2026
Closed
5 tasks
mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request May 10, 2026
@sfeng33
[Parser] Migrate response api streaming to unified parser (vllm-proje…
2343535 
…ct#38755)

Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: Andrew Xia <axia@meta.com>
my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request May 15, 2026
@sfeng33
[Parser] Migrate response api streaming to unified parser (vllm-proje…
3ca647b 
…ct#38755)

Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: Andrew Xia <axia@meta.com>
my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request May 15, 2026
@sfeng33
[Parser] Migrate response api streaming to unified parser (vllm-proje…
b039adf 
…ct#38755)

Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: Andrew Xia <axia@meta.com>
jhu960213 pushed a commit to jhu960213/vllm that referenced this pull request May 20, 2026
@sfeng33
[Parser] Migrate response api streaming to unified parser (vllm-proje…
a002860 
…ct#38755)

Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: Andrew Xia <axia@meta.com>
mvanhorn pushed a commit to mvanhorn/vllm that referenced this pull request Jun 4, 2026
@sfeng33 @mvanhorn
[Parser] Migrate response api streaming to unified parser (vllm-proje…
f804a53 
…ct#38755)

Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: Andrew Xia <axia@meta.com>
Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aarnphm aarnphm aarnphm approved these changes

@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337 DarkLight1337 is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@NickLucche NickLucche Awaiting requested review from NickLucche NickLucche is a code owner

@chaunceyjiang chaunceyjiang Awaiting requested review from chaunceyjiang chaunceyjiang is a code owner

@russellb russellb Awaiting requested review from russellb russellb is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@chaunceyjiang chaunceyjiang

Labels

frontend ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sfeng33 @bfroemel @aarnphm @chaunceyjiang @DarkLight1337