fix: pause_generation should not populate running_batch on prefill nodes by lawrence-harmonic · Pull Request #20273 · sgl-project/sglang

lawrence-harmonic · 2026-03-10T10:25:54Z

Fix #20272

Current logic leaks prefill request into running_batch which are never cleaned up, eventually stalling the scheduler when --max-running-requests is reached

gemini-code-assist · 2026-03-10T11:09:54Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a stability issue in the generation pausing mechanism. Previously, attempting to pause generation could cause a prefill worker to crash under specific conditions related to batch state. The changes introduce a more robust way to handle batch merging, ensuring that the system operates reliably even when internal batch structures are empty, thereby improving the overall resilience of the SGLang runtime.

Highlights

Bug Fix: Resolved a critical issue where the /pause_generation command could lead to a crash in the prefill worker, specifically when self.running_batch was empty during a merge operation.
Batch Handling Logic: Modified the batch merging logic within the pause_generation method to safely handle cases where the running_batch is empty, preventing merge_batch from being called on an uninitialized or empty batch.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

python/sglang/srt/managers/scheduler.py
- Modified the pause_generation method to add a conditional check before merging self.last_batch into self.running_batch.

Activity

No human activity has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request aims to fix a crash in pause_generation when self.running_batch is empty. The change correctly checks if self.running_batch is empty before merging self.last_batch. However, the fix is incomplete as it doesn't handle the case where self.last_batch itself becomes empty after filtering, which can lead to a similar crash or inconsistent state. I've suggested an addition to make the fix more robust by checking if self.last_batch is empty before the merge operation.

_{Note: Security Review did not run due to the size of the PR.}

gemini-code-assist · 2026-03-24T01:39:22Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Current logic: * Crashes with `AttributeError: 'NoneType' object has no attribute 'merge_batch'` at `self.sampling_info.merge_batch` * Leaks prefill request into running_batch which are never cleaned up, eventually stalling the scheduler when `--max-running-requests` is reached

- Remove redundant is_prefill_only guard (disaggregation_mode check suffices) - Add running_batch.is_empty() guard for retract path to prevent crash (sgl-project#20272) - Update tests to cover retract mode instead of in_place (already fixed by sgl-project#21705) - Add idle-node retract test for empty running_batch regression

hnyls2002 · 2026-04-03T22:25:42Z

/rerun-test registered/disaggregation/test_disaggregation_basic.py registered/rl/test_update_weights_from_distributed.py registered/rl/test_update_weights_from_tensor.py

github-actions · 2026-04-03T22:26:26Z

✅ 2-gpu-h100: View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_basic.py

✅ 2-gpu-h100: View workflow run

cd test/ && python3 registered/rl/test_update_weights_from_distributed.py

✅ 1-gpu-5090: View workflow run

cd test/ && python3 registered/rl/test_update_weights_from_tensor.py

hnyls2002 · 2026-04-03T22:33:58Z

/tag-and-rerun-ci

hnyls2002 · 2026-04-03T22:40:18Z

/rerun-test registered/disaggregation/test_disaggregation_basic.py

github-actions · 2026-04-03T22:40:47Z

✅ 2-gpu-h100: View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_basic.py

…des (sgl-project#20273)

…des (#20273)

…des (sgl-project#20273)

…ng_batch on prefill nodes (sgl-project#20273) Upstream SHA: 9593d43 Cherry-picked from sgl-project/sglang Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…des (sgl-project#20273)

lawrence-harmonic requested review from Ying1123, hnyls2002, merrymercy and xiezhq-hermann as code owners March 10, 2026 10:25

lawrence-harmonic mentioned this pull request Mar 10, 2026

PipelineRL -- keep cache on weight update THUDM/slime#1694

Open

gemini-code-assist Bot reviewed Mar 10, 2026

View reviewed changes

Comment thread python/sglang/srt/managers/scheduler.py Outdated

lawrence-harmonic marked this pull request as draft March 11, 2026 16:43

lawrence-harmonic force-pushed the fix/in_place branch 2 times, most recently from af18678 to 83811d0 Compare March 24, 2026 01:36

lawrence-harmonic changed the title ~~fix: /pause_generation in_place crashes prefill worker~~ fix: correctly manage PD state in scheduler pause_generation Mar 24, 2026

lawrence-harmonic marked this pull request as ready for review March 24, 2026 01:39

lawrence-harmonic changed the title ~~fix: correctly manage PD state in scheduler pause_generation~~ fix: pause_generation should not populate running_batch on prefill nodes Mar 24, 2026

lawrence-harmonic force-pushed the fix/in_place branch 2 times, most recently from 3baa872 to 53a3554 Compare March 24, 2026 07:08

lawrence-harmonic force-pushed the fix/in_place branch from 53a3554 to a7cb9f1 Compare March 24, 2026 07:18

lawrence-harmonic and others added 5 commits March 30, 2026 22:01

Merge remote-tracking branch 'origin/main' into fix/in_place

ad969b0

Merge branch 'main' into fix/in_place

807a2b1

add missing --disaggregation-bootstrap-port to test

2220e53

assert num_running_reqs == 0 after pause/resume to verify fix

9654cc3

github-actions Bot added the run-ci label Apr 3, 2026

enable metrics on prefill node for /metrics endpoint in test

074ebbb

hnyls2002 merged commit 9593d43 into sgl-project:main Apr 3, 2026
39 of 90 checks passed

maocheng23 mentioned this pull request Apr 6, 2026

Fix pause_generation leaking prefill requests into running_batch #22212

Closed

2 tasks

JustinTong0323 pushed a commit to JustinTong0323/sglang that referenced this pull request Apr 7, 2026

fix: pause_generation should not populate running_batch on prefill no…

99d4dcd

…des (sgl-project#20273)

Fridge003 pushed a commit that referenced this pull request Apr 7, 2026

fix: pause_generation should not populate running_batch on prefill no…

b0007ef

…des (#20273)

xiezhq-hermann pushed a commit to antgroup/sglang that referenced this pull request Apr 7, 2026

fix: pause_generation should not populate running_batch on prefill no…

81c51ec

…des (sgl-project#20273)

yhyang201 pushed a commit to yhyang201/sglang that referenced this pull request Apr 22, 2026

fix: pause_generation should not populate running_batch on prefill no…

2b36f10

…des (sgl-project#20273)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: pause_generation should not populate running_batch on prefill nodes#20273

fix: pause_generation should not populate running_batch on prefill nodes#20273
hnyls2002 merged 7 commits intosgl-project:mainfrom
lawrence-harmonic:fix/in_place

lawrence-harmonic commented Mar 10, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Mar 10, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

gemini-code-assist Bot commented Mar 24, 2026

Uh oh!

hnyls2002 commented Apr 3, 2026

Uh oh!

github-actions Bot commented Apr 3, 2026

Uh oh!

hnyls2002 commented Apr 3, 2026

Uh oh!

hnyls2002 commented Apr 3, 2026

Uh oh!

github-actions Bot commented Apr 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lawrence-harmonic commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Mar 10, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist Bot commented Mar 24, 2026

Uh oh!

hnyls2002 commented Apr 3, 2026

Uh oh!

github-actions Bot commented Apr 3, 2026

Uh oh!

hnyls2002 commented Apr 3, 2026

Uh oh!

hnyls2002 commented Apr 3, 2026

Uh oh!

github-actions Bot commented Apr 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lawrence-harmonic commented Mar 10, 2026 •

edited

Loading