Skip to content

Upgrade DeepGEMM to unify hopper with blackwell#9167

Closed
fzyzcjy wants to merge 65 commits intosgl-project:mainfrom
fzyzcjy:feat/deepgemm_upgrade
Closed

Upgrade DeepGEMM to unify hopper with blackwell#9167
fzyzcjy wants to merge 65 commits intosgl-project:mainfrom
fzyzcjy:feat/deepgemm_upgrade

Conversation

@fzyzcjy
Copy link
Copy Markdown
Collaborator

@fzyzcjy fzyzcjy commented Aug 14, 2025 •
edited
Loading

Motivation

baseline PR
Hopper + DeepEP Normal 61.3-61.9k 58.2-58.9k
Hopper + DeepEP LL 111.7-112.8s 107.7-108.6s
Blackwell + DeepEP Normal 72.8-73.6k 74.2-74.4k
Blackwell + DeepEP LL 73.6-84.0s 71.7-76.3s

Remarks

  • I use bench_one_batch_server, thus the prefill speed can be directly read, while decode speed cannot (and thus I put the overall time for the latter)
  • 3/4 cases have speedup or unchanged speed, while the Hopper + DeepEP normal slow down. I checked the profile and e.g. the deepgemm for attn changes from 1.31 to 1.47ms. Thus I guess it is because DeepGEMM has not optimize on that shape.

gsm8k: for normal it is 93.6 and 93.9 (same as baseline), for ll it is 90.0 and 89.4 (same as 89.4 and 89.8 baseline - iirc months ago I have seen something like this but this case is not used by real users)

Commands

num_gpu=8
PYTHONUNBUFFERED=1 SGLANG_TORCH_PROFILER_DIR=/host_home/temp_sglang_server2local \
  python3 -m sglang.launch_server \
  --model-path /dev/shm/DeepSeek-V3-0324 --trust-remote-code \
  --tp-size ${num_gpu} --dp-size ${num_gpu} --enable-dp-attention \
  --mem-fraction-static 0.8 \
  --chunked-prefill-size $((8192*${num_gpu})) \
  --max-running-requests $((${num_gpu}*256)) \
  --context-length 4096 \
  --disable-radix-cache \
  --enable-deepep-moe --deepep-mode normal \
  --disable-cuda-graph \
  --decode-log-interval 1

num_gpu=8
PYTHONUNBUFFERED=1 SGLANG_TORCH_PROFILER_DIR=/host_home/temp_sglang_server2local \
  python3 -m sglang.launch_server \
  --model-path /dev/shm/DeepSeek-V3-0324 --trust-remote-code \
  --tp-size ${num_gpu} --dp-size ${num_gpu} --enable-dp-attention \
  --mem-fraction-static 0.8 \
  --chunked-prefill-size $((128*${num_gpu})) \
  --max-running-requests $((${num_gpu}*128)) \
  --context-length 4096 \
  --disable-radix-cache \
  --enable-deepep-moe --deepep-mode low_latency \
  --cuda-graph-max-bs 128 \
  --moe-dense-tp-size 1 --enable-dp-lm-head \
  --decode-log-interval 1

while true; do (cd /host_home/primary_synced/sglang && python3 benchmark/gsm8k/bench_sglang.py --num-questions 1400 --parallel 10000 --port 30000); done
while true; do python3 -m sglang.bench_one_batch_server --base-url http://localhost:30000 --model-path /dev/shm/DeepSeek-V3-0324 --batch-size 2048 --input-len 2048 --output-len 1 --skip-warmup; done
while true; do python3 -m sglang.bench_one_batch_server --base-url http://localhost:30000 --model-path /dev/shm/DeepSeek-V3-0324 --batch-size 2048 --input-len 1 --output-len 1024 --skip-warmup; done

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Aug 14, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @fzyzcjy, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request upgrades the DeepGEMM integration to unify its behavior across different NVIDIA GPU architectures, specifically Hopper and Blackwell. It achieves this by removing the explicit recipe parameter from DeepGEMM function calls, implying that the underlying DeepGEMM library now handles the necessary optimizations internally, simplifying the API and ensuring consistent performance across target hardware.

Highlights

  • DeepGEMM API Simplification: The recipe parameter, previously used to specify DeepGEMM configurations for Blackwell architectures, has been removed from forward_deepgemm and forward_deepgemm_masked functions.
  • Unified DeepGEMM Operation: The recipe parameter has also been removed from the grouped_gemm_nt_f8f8bf16_masked function, indicating a unified approach to DeepGEMM operations that no longer requires explicit architecture-specific tuning via this parameter.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Aug 14, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the DeepGEMM integration to unify the code paths for Hopper and Blackwell architectures. The changes primarily involve removing the recipe parameter from calls to grouped_gemm_nt_f8f8bf16_masked, which was specific to the Blackwell implementation. This simplifies the code in ep_moe/layer.py and deep_gemm_wrapper/entrypoint.py by removing conditional logic and parameters, leading to a cleaner and more maintainable implementation. The changes are consistent and appear correct, reflecting an update in the underlying deep_gemm library API. Overall, this is a good simplification and unification of the codebase.

@fzyzcjy fzyzcjy mentioned this pull request Aug 14, 2025
Open
@fzyzcjy fzyzcjy mentioned this pull request Aug 15, 2025
Closed
@hnyls2002 hnyls2002 mentioned this pull request Aug 17, 2025
Merged
@FlamingoPg FlamingoPg mentioned this pull request Aug 19, 2025
Merged
4 tasks
@fzyzcjy
Copy link
Copy Markdown
Collaborator Author

fzyzcjy commented Aug 24, 2025

This is currently blocked by: deepseek-ai/DeepGEMM#160 (comment)

FlamingoPg added a commit to FlamingoPg/sglang that referenced this pull request Aug 25, 2025
@FlamingoPg
chore: cherry-pick PR sgl-project#9167 (squash)
bf710d0
zhyncs
zhyncs reviewed Aug 27, 2025
View reviewed changes
Comment thread .github/workflows/pr-test-pd-router.yml Outdated
zhyncs
zhyncs reviewed Aug 27, 2025
View reviewed changes
Comment thread .github/workflows/vllm-dependency-test.yml Outdated
zhyncs
zhyncs reviewed Aug 27, 2025
View reviewed changes
Comment thread python/pyproject.toml Outdated
zhyncs
zhyncs reviewed Aug 27, 2025
View reviewed changes
Comment thread python/sglang/srt/entrypoints/engine.py Outdated
@Sulfur6 Sulfur6 mentioned this pull request Aug 27, 2025
Merged
@zhyncs
Copy link
Copy Markdown
Collaborator

zhyncs commented Aug 27, 2025

co author in #9340

@zhyncs zhyncs closed this Aug 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner

@ch-wan ch-wan Awaiting requested review from ch-wan ch-wan is a code owner

@BBuf BBuf Awaiting requested review from BBuf BBuf is a code owner

@kushanam kushanam Awaiting requested review from kushanam

@Edwardf0t1 Edwardf0t1 Awaiting requested review from Edwardf0t1 Edwardf0t1 is a code owner

@CatherineSue CatherineSue Awaiting requested review from CatherineSue

@slin1237 slin1237 Awaiting requested review from slin1237

2 more reviewers

@zhyncs zhyncs zhyncs left review comments

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@zhyncs zhyncs

Labels

high priority

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@fzyzcjy @zhyncs @FlamingoPg