[Feature] Support DeepSeek MTP on NPU#11897
Merged
hnyls2002 merged 18 commits intosgl-project:mainfrom Oct 30, 2025
Merged
Conversation
43b19de to
009918a
Compare
009918a to
4537d03
Compare
Collaborator
|
Can we add an PR Test for NPU MTP？ |
|
Hi, which DeepSeek model is this, and is it running on one machine or two? |
Collaborator
Author
Hi, this supports both V3 and V3.2, and it could run on one machine if hbm capacity allows |
Collaborator
Author
for sure, we have test_ascend_deepseek_mtp.py for pr-test now |
ping1jing2
approved these changes
Oct 26, 2025
Alcanderian
reviewed
Oct 28, 2025
hnyls2002
approved these changes
Oct 30, 2025
sglang-bot
reviewed
Nov 2, 2025
| export PATH="/usr/local/Ascend/8.3.RC1/compiler/bishengir/bin:${PATH}" | ||
| cd test/srt | ||
| python3 run_suite.py --suite per-commit-16-ascend-a3 --timeout-per-file 3600 | ||
| python3 run_suite.py --suite per-commit-16-ascend-a3 --timeout-per-file 3600 --auto-partition-id ${{ matrix.part }} --auto-partition-size 2 |
Member
There was a problem hiding this comment.
| if not _is_npu: | ||
| device: str = "cuda" | ||
| else: | ||
| device: str = "npu" |
Member
There was a problem hiding this comment.
- you should set this value to
npuwhen you create it instead of adding if/else here - or do this
device: str = "cuda" if not is_npu else "npu"
| self.lm_head.weight = head | ||
| torch.cuda.empty_cache() | ||
| torch.cuda.synchronize() | ||
| if not _is_npu: |
| if not _is_npu: | ||
| device = "cuda" | ||
| else: | ||
| device = "npu" |
Member
There was a problem hiding this comment.
read from the global variable?
| ) | ||
|
|
||
| if is_all_greedy or not TREE_SPEC_KERNEL_AVAILABLE: | ||
| if is_all_greedy or not TREE_SPEC_KERNEL_AVAILABLE or _is_npu: |
Member
There was a problem hiding this comment.
Style: use more general filed to replace is_npu
|
|
||
| # Sample tokens | ||
| if sampling_info.is_all_greedy: | ||
| if sampling_info.is_all_greedy or _is_npu: |
| bs, | ||
| ) | ||
| else: | ||
| sgl_build_tree_kernel_efficient( |
Member
There was a problem hiding this comment.
the GPU code should be in the first branch of if/else
| if _is_cuda or _is_hip: | ||
| from sgl_kernel import verify_tree_greedy | ||
|
|
||
| verify_tree_greedy( |
Member
There was a problem hiding this comment.
you can try to add more arguments to sgl kernel and remove these
Contributor
|
Hello, is deepseek the only model supporting speculative decoding on NPU? Will qwen3 etc. be supported? |
28 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This pr primarily aims to support deepseek's mtp on ascend npus.
Modifications
Accuracy Tests
Benchmarking and Profiling
Checklist