[Fix]Pin mooncake version to 0.3.7.post2 in grace blackwell by Fridge003 · Pull Request #16502 · sgl-project/sglang

Fridge003 · 2026-01-05T16:43:13Z

Motivation

0.3.8 will cause some transfer error on GB200

[2026-01-04 09:39:43 DP0 TP0 EP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2905, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 360, in __init__
    self.init_overlap()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 996, in init_overlap
    self.future_map = FutureMap(
                      ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/overlap_utils.py", line 63, in __init__
    self.token_ids_buf = torch.empty(
                         ^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: invalid argument
Search for `cudaErrorInvalidValue' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

After reverting to 0.3.7.post2, this issue goes away

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review Process

Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments (/tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci) or contact authorized users to do so.
After green CI and required approvals, ask Merge Oncalls to merge.

gemini-code-assist · 2026-01-05T16:43:28Z

Summary of Changes

Hello @Fridge003, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical compatibility issue where mooncake-transfer-engine version 0.3.8 causes a CUDA error: invalid argument on GB200 systems. The solution involves modifying the Docker build process to conditionally install mooncake-transfer-engine version 0.3.7.post2 when the GRACE_BLACKWELL environment variable is set, ensuring stability for GB200 users while allowing other environments to continue using the newer 0.3.8 version.

Highlights

Conditional Mooncake Version Pinning: Introduced conditional logic in the Dockerfile to install mooncake-transfer-engine version 0.3.7.post2 specifically for Grace Blackwell (GB200) environments.
GB200 Compatibility Fix: Addressed a CUDA error: invalid argument encountered on GB200 systems when using mooncake-transfer-engine==0.3.8, by reverting to version 0.3.7.post2 for these specific environments.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request pins the mooncake-transfer-engine version to 0.3.7.post2 for Grace Blackwell builds to resolve a transfer error. The change is implemented by adding conditional logic to the Dockerfile. However, the added shell script is not wrapped in a RUN instruction, which would cause the Docker build to fail. I've provided a critical comment with a suggestion to fix this.

gemini-code-assist · 2026-01-05T16:51:32Z

+if [ "$GRACE_BLACKWELL" = "1" ]; then \
+    python3 -m pip install mooncake-transfer-engine==0.3.7.post2; \
+else \
+    python3 -m pip install mooncake-transfer-engine==0.3.8; \
+fi


This if statement is a shell command and must be executed within a RUN instruction. Without it, the Docker build will fail with a syntax error. The if block has been wrapped in a RUN instruction to fix this. It is also recommended to add --mount=type=cache,target=/root/.cache/pip to this RUN instruction to leverage build caching for pip, similar to other package installations in this Dockerfile.

RUN if [ "$GRACE_BLACKWELL" = "1" ]; then \ python3 -m pip install mooncake-transfer-engine==0.3.7.post2; \ else \ python3 -m pip install mooncake-transfer-engine==0.3.8; \ fi

stmatengss · 2026-01-05T17:03:02Z

Thanks for reporting this bug. I'm not certain this issue stems from different Mooncake versions, as the error originates from torch.empty. Is this CI test an EP test using mooncake backend? To the best of my knowledge, it may be caused by interference between Mooncake EP and torch. CC. @UNIDY2002

Fridge003 · 2026-01-06T02:40:42Z

Thanks for reporting this bug. I'm not certain this issue stems from different Mooncake versions, as the error originates from torch.empty. Is this CI test an EP test using mooncake backend? To the best of my knowledge, it may be caused by interference between Mooncake EP and torch. CC. @UNIDY2002

This test was launching some small scale PD disagg setting with Dynamo
I used the script here, but with the sglang version changed to latest 0.5.7
https://github.com/ishandhanani/srt-slurm/blob/main/recipies/gb200-fp4/1k1k/low-latency.yaml

Fridge003 · 2026-01-06T03:47:13Z

Images are built correctly. AMD arch with 0.3.8 and ARM arch with 0.3.7.post2
https://github.com/sgl-project/sglang/actions/runs/20737302530/job/59537071385

ShangmingCai

Thx. Will figure out what is wrong and fix this in the post version.

…ect#16502)

[Fix]Pin mooncake version to 0.3.7.post2 in grace blackwell

e1de550

Fridge003 requested review from HaiShaw, ishandhanani and ispobock as code owners January 5, 2026 16:43

gemini-code-assist Bot reviewed Jan 5, 2026

View reviewed changes

Fridge003 added 2 commits January 6, 2026 11:45

fix

5328ce5

Merge remote-tracking branch 'origin/main' into patch-1

31bae5e

ShangmingCai approved these changes Jan 6, 2026

View reviewed changes

alogfans mentioned this pull request Jan 6, 2026

[TE] Fix fabric detection in NVLink transport kvcache-ai/Mooncake#1332

Closed

15 tasks

Fridge003 merged commit 6ffe1fc into main Jan 6, 2026
53 checks passed

Fridge003 deleted the baizhou/patch-1 branch January 6, 2026 06:11

jamesjxliu pushed a commit to jamesjxliu/sglang that referenced this pull request Jan 6, 2026

[Fix]Pin mooncake version to 0.3.7.post2 in grace blackwell (sgl-proj…

bab14e7

…ect#16502)

ShangmingCai mentioned this pull request Jan 9, 2026

chore: bump mooncake version to 0.3.8.post1 #16792

Merged

5 tasks

ShangmingCai added a commit that referenced this pull request Jan 9, 2026

revert #16502

28792a9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix]Pin mooncake version to 0.3.7.post2 in grace blackwell#16502

[Fix]Pin mooncake version to 0.3.7.post2 in grace blackwell#16502
Fridge003 merged 3 commits intomainfrom
baizhou/patch-1

Fridge003 commented Jan 5, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Jan 5, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jan 5, 2026

Uh oh!

stmatengss commented Jan 5, 2026

Uh oh!

Fridge003 commented Jan 6, 2026 •

edited

Loading

Uh oh!

Fridge003 commented Jan 6, 2026 •

edited

Loading

Uh oh!

ShangmingCai left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Fridge003 commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Review Process

Uh oh!

gemini-code-assist Bot commented Jan 5, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

stmatengss commented Jan 5, 2026

Uh oh!

Fridge003 commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fridge003 commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShangmingCai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fridge003 commented Jan 5, 2026 •

edited

Loading

Fridge003 commented Jan 6, 2026 •

edited

Loading

Fridge003 commented Jan 6, 2026 •

edited

Loading