Add bootstrap_room validation to detect metadata corruption in PD disaggregation by Simon-Li · Pull Request #17430 · sgl-project/sglang

Simon-Li · 2026-01-20T15:49:26Z

Summary

Adds proactive detection for context corruption in Prefill-Decode disaggregation mode by validating bootstrap_room during metadata transfer.

Problem

Currently, metadata corruption (where decode server receives wrong first token from a different request) can only be detected through customer reports. This:

Severely impacts service quality
Makes it difficult to validate whether fixes work
Results in silent data corruption reaching users

Motivation

100% detection rate for metadata corruption
Fails fast instead of returning wrong results to users
Essential for validating effectiveness of Add an env var to allow transferring small metadata via TCP for PD #16951 and future fixes
Zero computation overhead (simple integer comparison)
Minimal memory cost (64 bytes per metadata buffer slot)
Production safety best practice for critical data paths

Solution

Store and validate bootstrap_room in metadata buffer:

Prefill side:

Store bootstrap_room in metadata buffer during set_buf()
Padded to 64 bytes for RDMA efficiency as suggested

Decode side:

Validate bootstrap_room matches before using metadata
Abort request immediately if mismatch detected
Log detailed error for debugging

Accuracy Tests

N/A

Benchmarking and Profiling

N/A

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review Process

Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
After green CI and required approvals, ask Merge Oncalls to merge.

Store and validate bootstrap_room to detect metadata corruption. Prevents decode server from using wrong metadata by validating bootstrap_room match before committing transfer.

gemini-code-assist · 2026-01-20T15:49:46Z

Summary of Changes

Hello @Simon-Li, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the robustness of the Prefill-Decode disaggregation system by introducing a critical validation mechanism. It prevents metadata corruption, where a decode server might process data from an incorrect request, by verifying the bootstrap_room identifier during metadata transfer. This ensures data integrity, allows for immediate detection of issues, and provides better debugging information, ultimately improving service quality and reliability.

Highlights

Proactive Context Corruption Detection: Introduces a mechanism to proactively detect context corruption in Prefill-Decode disaggregation mode, preventing silent data issues.
Bootstrap Room Validation: Implements validation of the bootstrap_room identifier during metadata transfer between prefill and decode servers to ensure data integrity.
Prefill Side Storage: On the prefill side, the bootstrap_room is now stored in the metadata buffer, padded to 64 bytes for optimized RDMA efficiency.
Decode Side Validation and Abort: On the decode side, if a mismatch is detected between the expected and received bootstrap_room, the request is immediately aborted, and a detailed error is logged.
Improved Reliability and Debugging: Aims for a 100% detection rate for metadata corruption, allowing the system to fail fast and provide better debugging information, which is crucial for validating fixes like Add an env var to allow transferring small metadata via TCP for PD #16951.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request introduces a robust mechanism to detect metadata corruption in Prefill-Decode disaggregation mode by validating bootstrap_room during metadata transfer. This is a critical improvement for service quality and data integrity. The changes involve adding a bootstrap_rooms tensor to MetadataBuffers in utils.py and implementing validation logic in decode.py. Overall, the solution addresses the problem effectively with minimal overhead.

Simon-Li · 2026-01-21T02:29:14Z

/tag-and-rerun-ci

ShangmingCai · 2026-01-21T02:58:05Z

Use pre-commit run --all-files to fix lint.

When actual_room == 0, metadata has not arrived yet due to timing. The request should stay in the transfer queue for the next poll instead of being aborted immediately.

Simon-Li · 2026-01-21T10:14:14Z

Use pre-commit run --all-files to fix lint.

Fixed lint issues with black and isort. Ready for review.

ShangmingCai

Looks good to me. Let's check the CI.

ShangmingCai · 2026-01-21T10:42:16Z

/tag-and-rerun-ci

Simon-Li · 2026-01-26T15:32:47Z

Looks good to me. Let's check the CI.

CI mostly passed. The failing tests look unrelated to the changes. Let me know if you'd like me to investigate any specific failures.

ShangmingCai · 2026-01-27T14:29:04Z

/rerun-failed-ci

ShangmingCai · 2026-01-27T14:30:22Z

Looks good to me. Let's check the CI.

CI mostly passed. The failing tests look unrelated to the changes. Let me know if you'd like me to investigate any specific failures.

Need to wait for the results of h20, h200, b200 tests.

ShangmingCai · 2026-01-29T06:36:12Z

https://github.com/sgl-project/sglang/actions/runs/21203050166/job/61827774360?pr=17430 The disaggregation basic test has failed.

…for better readability

ShangmingCai · 2026-01-30T03:25:00Z

I have fixed the bug. Let's try another round of CI.

Simon-Li · 2026-01-30T09:37:25Z

I have fixed the bug. Let's try another round of CI.

@ShangmingCai Thanks for the fix! The warmup/fake mode bypass makes sense - I hadn't considered that scenario where no actual metadata transfer occurs.

ShangmingCai · 2026-02-02T02:43:13Z

CI has passed. The failed test is due to a broken dependency in the H200 CI machine; it should be irrelevant.

…aggregation (sgl-project#17430) Co-authored-by: 继优 <jiyou.ljy@alibaba-inc.com> Co-authored-by: Shangming Cai <csmthu@gmail.com>

Add metadata validation for PD disaggregation

466f5fc

Store and validate bootstrap_room to detect metadata corruption. Prevents decode server from using wrong metadata by validating bootstrap_room match before committing transfer.

Simon-Li requested review from ByronHsu, ShangmingCai and hnyls2002 as code owners January 20, 2026 15:49

gemini-code-assist Bot reviewed Jan 20, 2026

View reviewed changes

Comment thread python/sglang/srt/disaggregation/decode.py Outdated

Comment thread python/sglang/srt/disaggregation/utils.py Outdated

Comment thread python/sglang/srt/disaggregation/utils.py Outdated

Simon-Li closed this Jan 20, 2026

Simon-Li reopened this Jan 21, 2026

Merge branch 'main' into feat/metadata-validation

4664ba3

ShangmingCai reviewed Jan 21, 2026

View reviewed changes

Comment thread python/sglang/srt/disaggregation/decode.py Outdated

ShangmingCai reviewed Jan 21, 2026

View reviewed changes

Comment thread python/sglang/srt/disaggregation/utils.py Outdated

ShangmingCai reviewed Jan 21, 2026

View reviewed changes

Comment thread python/sglang/srt/disaggregation/utils.py Outdated

继优 added 2 commits January 21, 2026 16:40

Fix metadata validation to handle timing issues in decode transfer queue

026d6dc

When actual_room == 0, metadata has not arrived yet due to timing. The request should stay in the transfer queue for the next poll instead of being aborted immediately.

Apply black formatting

2e5cd48

ShangmingCai reviewed Jan 21, 2026

View reviewed changes

github-actions Bot added the run-ci label Jan 21, 2026

ShangmingCai mentioned this pull request Jan 29, 2026

[PD] feat: support mooncake intra-node nvlink kv transfer #17866

Merged

5 tasks

ShangmingCai added 3 commits January 30, 2026 11:14

fix warmup/fake mode bug

67c8fd0

fix name, metabuffer is for per request, so we shall not use 'rooms' …

1756aee

…for better readability

revert pointless change

8b9d46f

fix lint

04763df

Merge branch 'main' into feat/metadata-validation

57ce1d5

Merge branch 'main' into feat/metadata-validation

8ed2e09

ShangmingCai merged commit 8ed35df into sgl-project:main Feb 2, 2026
520 of 564 checks passed

Oseltamivir mentioned this pull request Feb 24, 2026

Fix bootstrap_room uint64 overflow in MetadataBuffers.set_buf #19259

Closed

Conversation

Simon-Li commented Jan 20, 2026

Summary

Problem

Motivation

Solution

Accuracy Tests

Benchmarking and Profiling

Checklist

Review Process

Uh oh!

gemini-code-assist Bot commented Jan 20, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Simon-Li commented Jan 21, 2026

Uh oh!

ShangmingCai commented Jan 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Simon-Li commented Jan 21, 2026

Uh oh!

ShangmingCai left a comment

Choose a reason for hiding this comment

Uh oh!

ShangmingCai commented Jan 21, 2026

Uh oh!

Simon-Li commented Jan 26, 2026

Uh oh!

ShangmingCai commented Jan 27, 2026

Uh oh!

ShangmingCai commented Jan 27, 2026

Uh oh!

ShangmingCai commented Jan 29, 2026

Uh oh!

ShangmingCai commented Jan 30, 2026

Uh oh!

Simon-Li commented Jan 30, 2026

Uh oh!

ShangmingCai commented Feb 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants