Skip to content

[Refactor] Algebraic data type for nextn config + some basic refactors#17347

Merged
Fridge003 merged 4 commits intosgl-project:mainfrom
xyjixyjixyji:refactor/deepseek_weightload_nextn
Jan 23, 2026
Merged

[Refactor] Algebraic data type for nextn config + some basic refactors#17347
Fridge003 merged 4 commits intosgl-project:mainfrom
xyjixyjixyji:refactor/deepseek_weightload_nextn

Conversation

@xyjixyjixyji
Copy link
Copy Markdown
Contributor

@xyjixyjixyji xyjixyjixyji commented Jan 19, 2026 •
edited
Loading

Motivation

The nextn config's data fields and usage is scattering everywhere in the weightloader code, which is relatively hard to reason about the data flow.

This PR introduced an algebraic data type for nextn config and uses typed branching based on that to make the control flow more clear.

And some minor readability improvements.

Link: #16291

Modifications

As above.

Accuracy Tests

Server:

python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-R1  --tp 8 --speculative-algo EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --trust-remote --moe-runner-backend triton

Client:

root@11c14e22e279:/sgl-workspace/sglang# python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319 --data-path /tmp/test.jsonl
Downloading from https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/test.jsonl to /tmp/test.jsonl
/tmp/test.jsonl: 732kB [00:00, 45.9MB/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [07:51<00:00,  2.80it/s]
Accuracy: 0.956
Invalid: 0.000
Latency: 473.622 s
Output throughput: 280.956 token/s

Benchmarking and Profiling

On B200

master branch

python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-R1  --tp 8 --speculative-algo EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --trust-remote --moe-runner-backend triton

+-------------+--------+------------+-----------------+
| Latency (s) | Tokens | Acc Length | Speed (token/s) |
+-------------+--------+------------+-----------------+
|    4.017    |  512   |   2.393    |     127.45      |
+-------------+--------+------------+-----------------+

refactor branch

python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-R1  --tp 8 --speculative-algo EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --trust-remote --moe-runner-backend triton

+-------------+--------+------------+-----------------+
| Latency (s) | Tokens | Acc Length | Speed (token/s) |
+-------------+--------+------------+-----------------+
|    3.756    |  512   |   2.393    |     136.30      |
+-------------+--------+------------+-----------------+

Checklist

Review Process

  1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
  4. After green CI and required approvals, ask Merge Oncalls to merge.

@gemini-code-assist
Copy link
Copy Markdown
Contributor

gemini-code-assist Bot commented Jan 19, 2026

Summary of Changes

Hello @xyjixyjixyji, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the handling of nextn speculative decoding configuration within the DeepSeekV2 weight loader. By introducing an algebraic data type (NextNConfig) and leveraging Python's pattern matching, the changes aim to clarify the data flow and control logic related to nextn settings, making the codebase more robust and easier to reason about.

Highlights

  • Introduced Algebraic Data Type for NextN Configuration: Defined NextNEnabledConfig and NextNDisabledConfig dataclasses, and a NextNConfig union type to encapsulate nextn related parameters, replacing a simple boolean flag.
  • Refactored Conditional Logic with Pattern Matching: Replaced if/else statements dependent on the is_nextn boolean with Python's match statement and the new NextNConfig type, improving code clarity and maintainability.
  • Centralized NextN Configuration Initialization: Extracted the logic for determining nextn configuration parameters into a new private helper method, _initialize_nextn_conf, reducing scattering of nextn related data flow.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@xyjixyjixyji xyjixyjixyji marked this pull request as draft January 19, 2026 09:57
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jan 19, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an algebraic data type for the nextn configuration, significantly improving the clarity and maintainability of the DeepseekV2WeightLoaderMixin class. The refactoring effectively centralizes nextn related logic into NextNEnabledConfig and NextNDisabledConfig dataclasses, and leverages structural pattern matching (match case) for cleaner conditional handling. This change enhances type safety and makes the data flow easier to reason about, aligning well with the motivation stated in the PR description.

@xyjixyjixyji xyjixyjixyji marked this pull request as ready for review January 20, 2026 07:52
@gemini-code-assist
Copy link
Copy Markdown
Contributor

gemini-code-assist Bot commented Jan 20, 2026

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@Qiaolin-Yu
Copy link
Copy Markdown
Collaborator

Qiaolin-Yu commented Jan 21, 2026

/tag-and-rerun-ci

@xyjixyjixyji
Copy link
Copy Markdown
Contributor Author

xyjixyjixyji commented Jan 22, 2026

/rerun-failed-ci

@xyjixyjixyji
Copy link
Copy Markdown
Contributor Author

xyjixyjixyji commented Jan 22, 2026 •
edited
Loading

Checked CI, seems not related / timeout.

@xyjixyjixyji
Copy link
Copy Markdown
Contributor Author

xyjixyjixyji commented Jan 23, 2026

/rerun-failed-ci

@xyjixyjixyji
Copy link
Copy Markdown
Contributor Author

xyjixyjixyji commented Jan 23, 2026

/rerun-failed-ci

Fridge003
Fridge003 approved these changes Jan 23, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@Fridge003 Fridge003 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really Cool

@Fridge003 Fridge003 merged commit 010c17a into sgl-project:main Jan 23, 2026
223 of 233 checks passed
@Fridge003 Fridge003 mentioned this pull request Jan 23, 2026
Merged
Johnsonms pushed a commit to Johnsonms/sglang that referenced this pull request Feb 14, 2026
@xyjixyjixyji @Johnsonms
[Refactor] Algebraic data type for nextn config + some basic refactors (
3666f43 
sgl-project#17347)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Fridge003 Fridge003 Fridge003 approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@Fridge003 Fridge003

Labels

deepseek run-ci

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@xyjixyjixyji @Qiaolin-Yu @Fridge003