[Doc] refine spec decode docs for SpecV2/STANDALONE/NGRAM by alphabetc1 · Pull Request #18321 · sgl-project/sglang

alphabetc1 · 2026-02-05T19:41:24Z

Motivation

Fix #18268: add docs for SGLANG_ENABLE_SPEC_V2/STANDALONE/NGRAM

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review Process

Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
After green CI and required approvals, ask Merge Oncalls to merge.

gemini-code-assist · 2026-02-05T19:41:41Z

Summary of Changes

Hello @alphabetc1, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the documentation for SGLang's speculative decoding capabilities. It introduces detailed guides and runnable examples for three distinct speculative decoding algorithms: Standalone, Speculative Decoding V2 with an overlap scheduler, and Ngram-based decoding. These additions aim to provide users with clearer instructions and practical demonstrations for leveraging these advanced inference optimization techniques.

Highlights

EAGLE Decoding Parameter Clarification: The description for speculative_draft_model_path in EAGLE decoding has been refined to clarify its typical requirement for EAGLE/EAGLE3 and STANDALONE, and to note that it can be omitted for some MTP-enabled models where SGLang may auto-handle it.
New Standalone Speculative Decoding Documentation: A new section has been added to document 'Standalone Speculative Decoding (Small Draft Model)'. This includes instructions on how to enable it, a list of relevant parameters, and a practical code example demonstrating its usage.
New Speculative Decoding V2 Documentation: Documentation for 'Speculative Decoding V2 (Overlap Scheduler)' has been introduced. This covers its experimental nature, how to enable it via the SGLANG_ENABLE_SPEC_V2 environment variable, important notes regarding speculative-eagle-topk constraints, and a runnable code example.
New Ngram Speculative Decoding Documentation: A dedicated section for 'Ngram Speculative Decoding' has been added. It explains this method which uses an ngram cache instead of a separate draft model, lists its common parameters, highlights its limitations (e.g., CUDA-only, no dp-attention support), and provides a code example.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

docs/advanced_features/speculative_decoding.ipynb
- Clarified the description of the speculative_draft_model_path parameter for EAGLE decoding.
- Added a new markdown section and corresponding code cells demonstrating 'Standalone Speculative Decoding (Small Draft Model)'.
- Added a new markdown section and corresponding code cells demonstrating 'Speculative Decoding V2 (Overlap Scheduler)'.
- Added a new markdown section and corresponding code cells demonstrating 'Ngram Speculative Decoding'.

Activity

No review comments or discussions have been posted yet.
The pull request was opened by alphabetc1.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

alphabetc1 · 2026-02-05T19:41:57Z

@b8zhong PTAL

gemini-code-assist

Code Review

This pull request significantly improves the documentation for speculative decoding by adding detailed explanations and examples for STANDALONE, SpecV2, and NGRAM modes. The new content is clear and will be very helpful for users looking to leverage these advanced features. I have one minor suggestion to enhance the clarity of the notes for the experimental SpecV2 feature to prevent potential user errors.

gemini-code-assist · 2026-02-05T19:47:24Z

+    "- SpecV2 currently only supports `--speculative-eagle-topk 1`. When SpecV2 is enabled, **set `--speculative-eagle-topk 1` explicitly**.\n",
+    "- If you explicitly set `--speculative-eagle-topk > 1`, the server will error. If you omit `--speculative-eagle-topk`, auto-tuning may pick `topk > 1` for some models (e.g. Llama), which is not supported by SpecV2.\n",


For better clarity, consider rephrasing these two points. The current explanation of why --speculative-eagle-topk 1 is required is a bit dense. Explaining the reasoning more directly could help users avoid errors with this experimental feature.

Suggested change

"- SpecV2 currently only supports `--speculative-eagle-topk 1`. When SpecV2 is enabled, **set `--speculative-eagle-topk 1` explicitly**.\n",

"- If you explicitly set `--speculative-eagle-topk > 1`, the server will error. If you omit `--speculative-eagle-topk`, auto-tuning may pick `topk > 1` for some models (e.g. Llama), which is not supported by SpecV2.\n",

"- SpecV2 currently only supports `--speculative-eagle-topk 1`. When SpecV2 is enabled, **you must set `--speculative-eagle-topk 1` explicitly** to avoid errors.\n",

"- **Why is this necessary?** If you omit this parameter, auto-tuning might select a `topk > 1` for some models (e.g., Llama), which is unsupported. Explicitly setting a value greater than 1 will also cause an error.\n",

b8zhong · 2026-02-05T19:56:09Z

/tag-and-rerun-ci

b8zhong

Great job. Could you add a summary to beginning, so it could give an overview of all the methods?

zhaochenyang20 · 2026-02-07T03:21:16Z

I do not know why this execute notebook is passed, thus our documentation release is broken since this morning:

https://github.com/sgl-project/sglang/actions/runs/21769074765/job/62812303842

zhaochenyang20 · 2026-02-07T03:23:07Z

This is the error:

FileNotFoundError: [Errno 2] No such file or directory: 'SGLANG_ENABLE_SPEC_V2=True'

zhaochenyang20 · 2026-02-07T03:28:33Z

I do not quite understand why this PR is directly changing the document, but still skipped the execute document CI?

https://github.com/sgl-project/sglang/actions/runs/21725846483

zhaochenyang20 · 2026-02-07T04:31:34Z

#18390

just fix this. Our CI seems to skip the docs CI for long. Thanks for your PR here:

#18321

alphabetc1 · 2026-02-07T05:16:43Z

#18390

just fix this. Our CI seems to skip the docs CI for long. Thanks for your PR here:

#18321

Sorry, my bad! Thanks for the fix!

…t#18321)

docs: refine spec decode docs for SpecV2/ngram

40be585

github-actions Bot added documentation Improvements or additions to documentation speculative-decoding labels Feb 5, 2026

gemini-code-assist Bot reviewed Feb 5, 2026

View reviewed changes

github-actions Bot added the run-ci label Feb 5, 2026

b8zhong approved these changes Feb 5, 2026

View reviewed changes

b8zhong merged commit 8b21dd4 into sgl-project:main Feb 5, 2026
48 checks passed

alphabetc1 deleted the doc/spec branch February 5, 2026 20:38

alphabetc1 mentioned this pull request Feb 5, 2026

[Doc] add a summary section for spec decode document #18323

Merged

5 tasks

charlesHsuGG pushed a commit to charlesHsuGG/sglang that referenced this pull request Feb 9, 2026

[Doc] refine spec decode docs for SpecV2/STANDALONE/NGRAM (sgl-projec…

cde35fe

…t#18321)

Johnsonms pushed a commit to Johnsonms/sglang that referenced this pull request Feb 14, 2026

[Doc] refine spec decode docs for SpecV2/STANDALONE/NGRAM (sgl-projec…

5f6802f

…t#18321)

magicYang1573 pushed a commit to magicYang1573/sglang that referenced this pull request Mar 9, 2026

[Doc] refine spec decode docs for SpecV2/STANDALONE/NGRAM (sgl-projec…

1ce63a6

…t#18321)

Wangzheee pushed a commit to Wangzheee/sglang that referenced this pull request Mar 21, 2026

[Doc] refine spec decode docs for SpecV2/STANDALONE/NGRAM (sgl-projec…

74b6ad3

…t#18321)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Doc] refine spec decode docs for SpecV2/STANDALONE/NGRAM#18321

[Doc] refine spec decode docs for SpecV2/STANDALONE/NGRAM#18321
b8zhong merged 1 commit intosgl-project:mainfrom
alphabetc1:doc/spec

alphabetc1 commented Feb 5, 2026 •

edited by b8zhong

Loading

Uh oh!

gemini-code-assist Bot commented Feb 5, 2026

Uh oh!

alphabetc1 commented Feb 5, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Feb 5, 2026

Uh oh!

b8zhong commented Feb 5, 2026

Uh oh!

b8zhong left a comment •

edited

Loading

Uh oh!

Uh oh!

zhaochenyang20 commented Feb 7, 2026

Uh oh!

zhaochenyang20 commented Feb 7, 2026

Uh oh!

zhaochenyang20 commented Feb 7, 2026

Uh oh!

zhaochenyang20 commented Feb 7, 2026

Uh oh!

alphabetc1 commented Feb 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		"- SpecV2 currently only supports `--speculative-eagle-topk 1`. When SpecV2 is enabled, set `--speculative-eagle-topk 1` explicitly.\n",
		"- If you explicitly set `--speculative-eagle-topk > 1`, the server will error. If you omit `--speculative-eagle-topk`, auto-tuning may pick `topk > 1` for some models (e.g. Llama), which is not supported by SpecV2.\n",

Conversation

alphabetc1 commented Feb 5, 2026 • edited by b8zhong Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Review Process

Uh oh!

gemini-code-assist Bot commented Feb 5, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

alphabetc1 commented Feb 5, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

b8zhong commented Feb 5, 2026

Uh oh!

b8zhong left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhaochenyang20 commented Feb 7, 2026

Uh oh!

zhaochenyang20 commented Feb 7, 2026

Uh oh!

zhaochenyang20 commented Feb 7, 2026

Uh oh!

zhaochenyang20 commented Feb 7, 2026

Uh oh!

alphabetc1 commented Feb 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alphabetc1 commented Feb 5, 2026 •

edited by b8zhong

Loading

b8zhong left a comment •

edited

Loading