Skip to content

Support piecewise cuda graph for fused marlin moe#15100

Merged
ispobock merged 3 commits intosgl-project:mainfrom
ispobock:pcg-marlin-moe
Dec 16, 2025
Merged

Support piecewise cuda graph for fused marlin moe#15100
ispobock merged 3 commits intosgl-project:mainfrom
ispobock:pcg-marlin-moe

Conversation

@ispobock
Copy link
Copy Markdown
Collaborator

@ispobock ispobock commented Dec 14, 2025 •
edited
Loading

Motivation

Support piecewise cuda graph for models using fused_marlin_moe like moe models with gptq/awq quantization, kimi-k2-thinking model.

Accuracy Tests

python3 -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B-GPTQ-Int4 --trust-remote-code --enable-piecewise-cuda-graph

python3 benchmark/gsm8k/bench_sglang.py --num-questions 1400 --parallel 1400

Accuracy: 0.903
Invalid: 0.000
Latency: 17.246 s
Output throughput: 9550.159 token/s

@gemini-code-assist
Copy link
Copy Markdown
Contributor

gemini-code-assist Bot commented Dec 14, 2025

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@ispobock
Copy link
Copy Markdown
Collaborator Author

ispobock commented Dec 14, 2025

/tag-and-rerun-ci

@ispobock
Copy link
Copy Markdown
Collaborator Author

ispobock commented Dec 14, 2025

https://github.com/sgl-project/sglang/actions/runs/20204229586/job/58004459976?pr=15100 ci passed

@ispobock ispobock merged commit b399e3a into sgl-project:main Dec 16, 2025
366 of 399 checks passed
This was referenced Dec 16, 2025
Closed
Closed
tonyluj pushed a commit to openanolis/sglang that referenced this pull request Dec 17, 2025
@ispobock @tonyluj
Support piecewise cuda graph for fused marlin moe (sgl-project#15100)
c1b9add
jiaming1130 pushed a commit to zhuyijie88/sglang that referenced this pull request Dec 25, 2025
@ispobock @jiaming1130
Support piecewise cuda graph for fused marlin moe (sgl-project#15100)
15faa0a
YChange01 pushed a commit to YChange01/sglang that referenced this pull request Jan 13, 2026
@ispobock @YChange01
Support piecewise cuda graph for fused marlin moe (sgl-project#15100)
2fd73bf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BBuf BBuf BBuf approved these changes

@ch-wan ch-wan Awaiting requested review from ch-wan ch-wan is a code owner

@Edwardf0t1 Edwardf0t1 Awaiting requested review from Edwardf0t1 Edwardf0t1 is a code owner

@FlamingoPg FlamingoPg Awaiting requested review from FlamingoPg FlamingoPg is a code owner

@AniZpZ AniZpZ Awaiting requested review from AniZpZ AniZpZ is a code owner

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@Fridge003 Fridge003 Awaiting requested review from Fridge003 Fridge003 is a code owner

@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner

Assignees

No one assigned

Labels

run-ci

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ispobock @BBuf