Skip to content

Tiny extract ModelRunnerOutput#15400

Merged
fzyzcjy merged 4 commits intosgl-project:mainfrom
fzyzcjy:feat/refactor_forward_ret
Dec 18, 2025
Merged

Tiny extract ModelRunnerOutput#15400
fzyzcjy merged 4 commits intosgl-project:mainfrom
fzyzcjy:feat/refactor_forward_ret

Conversation

@fzyzcjy
Copy link
Copy Markdown
Collaborator

@fzyzcjy fzyzcjy commented Dec 18, 2025

Motivation

for easily adding fields in the future (indeed that is the direct reason why I add it), and for consistency and readability

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

@gemini-code-assist
Copy link
Copy Markdown
Contributor

gemini-code-assist Bot commented Dec 18, 2025

Summary of Changes

Hello @fzyzcjy, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the model_runner.forward method by introducing a new ModelRunnerOutput dataclass. This change aims to improve code consistency and readability by encapsulating the method's return values into a single object, which also facilitates easier addition of new fields in the future without altering the method's signature. The update involves modifying the model_runner.forward definition and adjusting all its call sites to use the new dataclass structure.

Highlights

  • Introduce ModelRunnerOutput Dataclass: A new dataclass named ModelRunnerOutput has been introduced in model_runner.py. This dataclass encapsulates the logits_output (which can be either LogitsProcessorOutput or PPProxyTensors) and a can_run_graph boolean, providing a structured way to return multiple values from the model_runner.forward method.
  • Refactor ModelRunner.forward Return Type: The ModelRunner.forward method and its internal _forward_raw helper now return an instance of the ModelRunnerOutput dataclass instead of a tuple. This change improves consistency and readability, and simplifies future additions of new return fields.
  • Update Call Sites Across Files: All existing call sites of the model_runner.forward method across various files (e.g., bench_one_batch.py, low_confidence.py, scheduler_pp_mixin.py, tp_worker.py, eagle_worker.py, eagle_worker_v2.py) have been updated to correctly access the logits_output and can_run_graph attributes from the newly returned ModelRunnerOutput object.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Dec 18, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the return value of ModelRunner.forward to use a ModelRunnerOutput dataclass instead of a tuple. This is a good change that improves code clarity and future extensibility. The changes have been applied consistently across all call sites.

I've added a couple of comments pointing out opportunities to reduce code duplication in bench_one_batch.py and low_confidence.py, which would further improve maintainability.

Comment thread python/sglang/bench_one_batch.py
Comment thread python/sglang/srt/dllm/algorithm/low_confidence.py
@fzyzcjy fzyzcjy mentioned this pull request Dec 18, 2025
Merged
6 tasks
@fzyzcjy
Copy link
Copy Markdown
Collaborator Author

fzyzcjy commented Dec 18, 2025 •
edited
Loading

reviewed by @hnyls2002 offline, ci briefly run in #15401,
merge first and fix if main ci fails according to ci policy

@fzyzcjy fzyzcjy merged commit ad9616f into sgl-project:main Dec 18, 2025
54 of 61 checks passed
fzyzcjy added a commit to fzyzcjy/sglang that referenced this pull request Dec 19, 2025
@fzyzcjy
Tiny extract ModelRunnerOutput (sgl-project#15400)
856d0d1 
(cherry picked from commit ad9616f)

# Conflicts:
#	python/sglang/srt/managers/scheduler_pp_mixin.py
#	python/sglang/srt/managers/tp_worker.py
Prozac614 pushed a commit to Prozac614/sglang that referenced this pull request Dec 23, 2025
@fzyzcjy @Prozac614
Tiny extract ModelRunnerOutput (sgl-project#15400)
11893da
jiaming1130 pushed a commit to zhuyijie88/sglang that referenced this pull request Dec 25, 2025
@fzyzcjy @jiaming1130
Tiny extract ModelRunnerOutput (sgl-project#15400)
48fd5fb
YChange01 pushed a commit to YChange01/sglang that referenced this pull request Jan 13, 2026
@fzyzcjy @YChange01
Tiny extract ModelRunnerOutput (sgl-project#15400)
63acea4
This was referenced Feb 2, 2026
Merged
Merged
popsiclexu pushed a commit to popsiclexu/sglang that referenced this pull request Mar 25, 2026
@fzyzcjy
Tiny extract ModelRunnerOutput (sgl-project#15400)
4d019b2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner

@Fridge003 Fridge003 Awaiting requested review from Fridge003 Fridge003 is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

@xiezhq-hermann xiezhq-hermann Awaiting requested review from xiezhq-hermann xiezhq-hermann is a code owner

1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@fzyzcjy