perf: avoid unnecessary gpu-cpu sync in eagle_info by ehuaa · Pull Request #20266 · sgl-project/sglang

ehuaa · 2026-03-10T09:24:55Z

Motivation

When stress testing Qwen3.5-397B with MTP enabled, the scheduler occasionally encounters the following scheduler watchdog error. Preliminary analysis suggests a potential deadlock arising from the interaction between GPU-CPU synchronization and collective communications (e.g., AllReduce).
According to the log below, the GPU-CPU sync happens in eagle_info.py.
This PR not only mitigates potential deadlock issues but also eliminates synchronization overhead, thereby improving overall inference performance.

[2026-03-10 02:09:02 TP2] Pyspy dump for PID 501830:
Process 501830: sglang::scheduler_TP2
Python v3.12.3 (/usr/bin/python3.12)

Thread 501830 (active): "MainThread"
    0x7f9484041d1d (libcuda.so.575.57.08)
    0x7f9483df9493 (libcuda.so.575.57.08)
    0x7f9483e3a2eb (libcuda.so.575.57.08)
    0x7f9484acc998 (libcuda.so.575.57.08)
    0x7f9484accdc8 (libcuda.so.575.57.08)
    0x7f9483e0542c (libcuda.so.575.57.08)
    0x7f9483ec4d3a (libcuda.so.575.57.08)
    0x7f9484a864e9 (libcuda.so.575.57.08)
    0x7f9483f76e37 (libcuda.so.575.57.08)
    cuStreamSynchronize (libcuda.so.575.57.08)
    0x7f95b1e128b3 (libcudart.so.12)
    cudaStreamSynchronize (libcudart.so.12)
    at::native::_local_scalar_dense_cuda(at::Tensor const&)::{lambda()#1}::operator() const (libtorch_cuda.so)
    at::native::_local_scalar_dense_cuda (libtorch_cuda.so)
    at::(anonymous namespace)::(anonymous namespace)::wrapper_CUDA___local_scalar_dense (libtorch_cuda.so)
    c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<c10::Scalar(at::Tensor const&), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CUDA___local_scalar_
dense(at::Tensor const&)>, c10::Scalar, c10::guts::typelist::typelist<at::Tensor const&> >, c10::Scalar(at::Tensor const&)>::call (libtorch_cuda.so)
    at::_ops::_local_scalar_dense::redispatch (libtorch_cpu.so)
    torch::autograd::VariableType::(anonymous namespace)::_local_scalar_dense (libtorch_cpu.so)
    c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<c10::Scalar(c10::DispatchKeySet, at::Tensor const&), &torch::autograd::VariableType::(anonymous namespace)::_lo
cal_scalar_dense(c10::DispatchKeySet, at::Tensor const&)>, c10::Scalar, c10::guts::typelist::typelist<c10::DispatchKeySet, at::Tensor const&> >, c10::Scalar(c10::DispatchKeySet, at::Tensor const&)>::call (libtorch_cpu.so)
    at::_ops::_local_scalar_dense::call (libtorch_cpu.so)
    at::native::item (libtorch_cpu.so)
    c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<c10::Scalar(at::Tensor const&), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CompositeImplicitAut
ograd__item(at::Tensor const&)>, c10::Scalar, c10::guts::typelist::typelist<at::Tensor const&> >, c10::Scalar(at::Tensor const&)>::call (libtorch_cpu.so)
    at::_ops::item::call (libtorch_cpu.so)
    at::native::is_nonzero (libtorch_cpu.so)
    at::_ops::is_nonzero::call (libtorch_cpu.so)
    torch::autograd::THPVariable_is_nonzero (libtorch_python.so)
    verify (eagle_info.py:340)
    verify (eagle_worker.py:748)
    forward_batch_generation (eagle_worker.py:316)
    run_batch (scheduler.py:2368)
    event_loop_normal (scheduler.py:1123)
   ...

Modifications

modify if-statement in eagle_info.py to use need_top_p_sampling instead of computing top_p sampling mask again.

Accuracy Tests

This pull requests does not affect model outputs.

Benchmarking and Profiling

Before

After

The comparison shows that we’ve shaved off serveral cuda kernel operations, for example, ten D2H operations (equivalent to the number of steps in the profile). And one cudaStreamSynchronize call has been eliminated, deferring the synchronization until the subsequent line:

accept_index_cpu = accept_index.tolist().

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review Process

Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
After green CI and required approvals, ask Merge Oncalls to merge.

gemini-code-assist · 2026-03-10T10:00:03Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical issue where GPU-CPU synchronization in the eagle_info.py module could lead to deadlocks, particularly under heavy load. By refactoring a conditional check, the change removes the explicit synchronization, thereby preventing potential deadlocks and simultaneously enhancing inference performance by reducing overhead.

Highlights

Deadlock Prevention: Modified an if-statement in eagle_info.py to avoid implicit GPU-CPU synchronization, which was identified as a potential cause of deadlocks during stress testing with MTP enabled.
Performance Improvement: Eliminated synchronization overhead by preventing cudaStreamSynchronize calls, leading to improved overall inference performance.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

python/sglang/srt/speculative/eagle_info.py
- Updated an if condition to use sampling_info.need_top_p_sampling instead of torch.all(sampling_info.top_ps == 1.0) to avoid implicit GPU-CPU synchronization.

Activity

No human activity has occurred on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request refactors the condition for applying top-p renormalization in the verify function within eagle_info.py. Instead of checking if all sampling_info.top_ps are equal to 1.0, the code now directly uses a dedicated boolean flag, sampling_info.need_top_p_sampling, to determine if top-p sampling is required. This change likely improves clarity and potentially efficiency or correctness by using an explicit flag.

ehuaa · 2026-03-12T08:18:13Z

Hi @hnyls2002 , can you help review this pr? Thanks.

binbabou · 2026-03-12T12:23:56Z

@ehuaa Hi, I am experiencing the same issue with Qwen3-235B deployment using sglang v0.5.4.post1. Despite simulating multiple stress testing cases, I have been unable to replicate the error. 😂
Could you provide more details on your stress testing methodology?"

eagle_info.py:308: if not torch.all(sampling_info.top_ps == 1.0):

Thread 1446 (active): "MainThread"
    verify (eagle_info.py:308)
    verify (eagle_worker.py:674)
    forward_batch_generation (eagle_worker.py:268)
    run_batch (scheduler.py:1936)
    event_loop_normal (scheduler.py:965)
    decorate_context (torch/utils/_contextlib.py:120)
    run_scheduler_process (scheduler.py:2777)
    run (multiprocessing/process.py:108)
    _bootstrap (multiprocessing/process.py:314)
    _main (multiprocessing/spawn.py:135)
    spawn_main (multiprocessing/spawn.py:122)
    <module> (<string>:1)

ehuaa · 2026-03-18T09:57:05Z

@ehuaa Hi, I am experiencing the same issue with Qwen3-235B deployment using sglang v0.5.4.post1. Despite simulating multiple stress testing cases, I have been unable to replicate the error. 😂 Could you provide more details on your stress testing methodology?"

eagle_info.py:308: if not torch.all(sampling_info.top_ps == 1.0):

Thread 1446 (active): "MainThread"
    verify (eagle_info.py:308)
    verify (eagle_worker.py:674)
    forward_batch_generation (eagle_worker.py:268)
    run_batch (scheduler.py:1936)
    event_loop_normal (scheduler.py:965)
    decorate_context (torch/utils/_contextlib.py:120)
    run_scheduler_process (scheduler.py:2777)
    run (multiprocessing/process.py:108)
    _bootstrap (multiprocessing/process.py:314)
    _main (multiprocessing/spawn.py:135)
    spawn_main (multiprocessing/spawn.py:122)
    <module> (<string>:1)

Hi @binbabou , my sglang version is v0.5.9 and my model version is Qwen3.5-397B, I simply use bench_serving.py to simulate stress testing, with 512 sequences（input 1k+8k） send to server simultaneously

Qiaolin-Yu · 2026-03-20T02:29:42Z

/rerun-ut test_eagle_infer_beta.py

github-actions · 2026-03-20T02:30:03Z

❌ /rerun-ut is not available for fork PRs (security restriction).

Please ask a maintainer to add the run-ci label and use the normal CI flow, or use /rerun-failed-ci to rerun workflows that have already passed the gate.

github-actions · 2026-03-20T02:30:12Z

❌ /rerun-ut is not available for fork PRs (security restriction).

Please ask a maintainer to add the run-ci label and use the normal CI flow, or use /rerun-failed-ci to rerun workflows that have already passed the gate.

Qiaolin-Yu

tested it locally, lgtm

Co-authored-by: root <qianhao@zhejianglab.org>

fix: avoid gpu-cpu sync in eagle_info to prevent deadlock

ed1649a

ehuaa requested review from Ying1123, hnyls2002 and merrymercy as code owners March 10, 2026 09:24

Merge branch 'main' into fix-eagle-sync-deadlock

b955b66

gemini-code-assist Bot reviewed Mar 10, 2026

View reviewed changes

ehuaa changed the title ~~fix: avoid gpu-cpu sync in eagle_info to prevent deadlock~~ fix: avoid unnecessary gpu-cpu sync in eagle_info Mar 12, 2026

Merge branch 'main' into fix-eagle-sync-deadlock

271b870

ehuaa changed the title ~~fix: avoid unnecessary gpu-cpu sync in eagle_info~~ perf: avoid unnecessary gpu-cpu sync in eagle_info Mar 18, 2026

Qiaolin-Yu approved these changes Mar 20, 2026

View reviewed changes

Qiaolin-Yu merged commit fa0d8f6 into sgl-project:main Mar 20, 2026
59 of 67 checks passed

Wangzheee pushed a commit to Wangzheee/sglang that referenced this pull request Mar 21, 2026

perf: avoid unnecessary gpu-cpu sync in eagle_info (sgl-project#20266)

479918a

Co-authored-by: root <qianhao@zhejianglab.org>

yudian0504 mentioned this pull request Mar 24, 2026

[Bug] tree_speculative_sampling_target_only produces non-deterministic results with deterministic=True in tensor parallel mode #21256

Closed

0-693 pushed a commit to 0-693/sglang that referenced this pull request Mar 25, 2026

perf: avoid unnecessary gpu-cpu sync in eagle_info (sgl-project#20266)

f0d24bf

Co-authored-by: root <qianhao@zhejianglab.org>

dutsc pushed a commit to dutsc/sglang that referenced this pull request Mar 30, 2026

perf: avoid unnecessary gpu-cpu sync in eagle_info (sgl-project#20266)

08f096d

Co-authored-by: root <qianhao@zhejianglab.org>

JustinTong0323 pushed a commit to JustinTong0323/sglang that referenced this pull request Apr 7, 2026

perf: avoid unnecessary gpu-cpu sync in eagle_info (sgl-project#20266)

c64d1b0

Co-authored-by: root <qianhao@zhejianglab.org>

yhyang201 pushed a commit to yhyang201/sglang that referenced this pull request Apr 22, 2026

perf: avoid unnecessary gpu-cpu sync in eagle_info (sgl-project#20266)

a0303f1

Co-authored-by: root <qianhao@zhejianglab.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: avoid unnecessary gpu-cpu sync in eagle_info#20266

perf: avoid unnecessary gpu-cpu sync in eagle_info#20266
Qiaolin-Yu merged 3 commits intosgl-project:mainfrom
ehuaa:fix-eagle-sync-deadlock

ehuaa commented Mar 10, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Mar 10, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

ehuaa commented Mar 12, 2026

Uh oh!

binbabou commented Mar 12, 2026

Uh oh!

ehuaa commented Mar 18, 2026

Uh oh!

Qiaolin-Yu commented Mar 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

Qiaolin-Yu left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ehuaa commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Review Process

Uh oh!

gemini-code-assist Bot commented Mar 10, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

ehuaa commented Mar 12, 2026

Uh oh!

binbabou commented Mar 12, 2026

Uh oh!

ehuaa commented Mar 18, 2026

Uh oh!

Qiaolin-Yu commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

Qiaolin-Yu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ehuaa commented Mar 10, 2026 •

edited

Loading

Qiaolin-Yu commented Mar 20, 2026 •

edited

Loading