First time I tested this commit (on 8*MI300X node) I got two Decode Batch logs and then sglang hung:

Tested on 8*MI308X GPUs.

# server
python3 -m sglang.launch_server --host 0.0.0.0 --trust-remote-code --tp 8 --model-path deepseek-ai/DeepSeek-V3  --speculative-algo NEXTN --speculative-draft SGLang/DeepSeek-V3-NextN --speculative-num-steps 2 --speculative-eagle-topk 4 --speculative-num-draft-tokens 4  --mem-fraction-static 0.5

# bench
python3 -m sglang.bench_one_batch_server --model None --base-url http://127.0.0.1:30000 --batch-size 1 --input-len 256 --output-len 256

First time I tested this commit (on 8*MI300X node) I got two Decode Batch logs and then sglang hung:

Can you share more details? For example your launch_server and bench command

First time I tested this commit (on 8*MI300X node) I got two Decode Batch logs and then sglang hung:

Can you share more details? For example your launch_server and bench command

yea sorry.

python3 -m sglang.launch_server --model-path /mnt/models/DeepSeek-R1/DeepSeek-R1/ --port 5000 --host 0.0.0.0 --served-model-name deepseek-r1 --trust-remote-code --tp-size 8 --enable-metrics --mem-fraction-static 0.7 --speculative-algo NEXTN --speculative-draft /mnt/models/DeepSeek-R1/DeepSeek-R1//MTP-NextN-Weights/ --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2
I also tried --speculative-eagle-topk 2 --speculative-num-draft-tokens 4, it booted and logged a Prefill batch then hung again.

I generated the nextn weights using scripts/export_deepseek_nextn.py.

Haven't had time to dive as deep as using something like py-spy to attach to the python procs and inspect current stack.

First time I tested this commit (on 8*MI300X node) I got two Decode Batch logs and then sglang hung:

Can you share more details? For example your launch_server and bench command

yea sorry.

python3 -m sglang.launch_server --model-path /mnt/models/DeepSeek-R1/DeepSeek-R1/ --port 5000 --host 0.0.0.0 --served-model-name deepseek-r1 --trust-remote-code --tp-size 8 --enable-metrics --mem-fraction-static 0.7 --speculative-algo NEXTN --speculative-draft /mnt/models/DeepSeek-R1/DeepSeek-R1//MTP-NextN-Weights/ --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 I also tried --speculative-eagle-topk 2 --speculative-num-draft-tokens 4, it booted and logged a Prefill batch then hung again.

I generated the nextn weights using scripts/export_deepseek_nextn.py.

Haven't had time to dive as deep as using something like py-spy to attach to the python procs and inspect current stack.

I cannot reproduce and I don't have MI300X access. Can you try different machines or environment? Since you boot successfully, the kernels this commit added should be working.

@HaiShaw hai, could you take a look? Thanks!

First time I tested this commit (on 8*MI300X node) I got two Decode Batch logs and then sglang hung:

Can you share more details? For example your launch_server and bench command

i can run it on mi300x. however, it got lower perf than without it.

I used 0.4.3post2-rocm630.

Serve cmdline -
python3 -m sglang.launch_server --host 0.0.0.0 --trust-remote-code --tp 8 --model-path /models/DeepSeek-R1 --speculative-algo NEXTN --speculative-draft /models/DeepSeek-R1-NextN --speculative-num-steps 2 --speculative-eagle-topk 4 --speculative-num-draft-tokens 4 --mem-fraction-static 0.5

Client -
python3 -m sglang.bench_one_batch_server --model None --base-url http://127.0.0.1:30000 --batch-size 1 --input-len 3200 --output-len 800

Perf:
batch size: 16
latency: 8.79 s
output throughput: 29.14 token/s
(input + output) throughput: 1894.00 token/s
batch size: 1
latency: 50.04 s
output throughput: 15.99 token/s
(input + output) throughput: 79.93 token/s

Perf without it:
batch size: 16
latency: 6.94 s
output throughput: 36.87 token/s
(input + output) throughput: 2396.78 token/s
batch size: 1
latency: 26.61 s
output throughput: 30.06 token/s
(input + output) throughput: 150.30 token/s

Do you get better perf with it on AMD GPU?

Do you get better perf with it on AMD GPU?

@andyluo7 Yes. With --batch-size 1 --input-len 256 --output-len 256 on MI308X

@yiakwy-xpu-ml-framework-team @HaiShaw Could you take a look at this? Thanks!

@yiakwy-xpu-ml-framework-team @HaiShaw Could you take a look at this? Thanks!

Thank you @zhaochenyang20 for including me in. Yes I am watching it , really great job @alexsun07.

I am having a look at the algorithm design if it is helpful to facilitat merging.

vLLM is also working on this import feature in a different approach.

@yiakwy-xpu-ml-framework-team, what shall we do next?

@zhaochenyang20 I am looking into it.

Hit out-of-bounds error at running gsm8k tests.

python3 -m sglang.launch_server --host 0.0.0.0 --trust-remote-code --tp 8 --model-path /data/deepseek-ai/DeepSeek-V3 --speculative-algo NEXTN --speculative-draft SGLang/DeepSeek-V3-NextN --speculative-num-steps 2 --speculative-eagle-topk 4 --speculative-num-draft-tokens 4 --mem-fraction-static 0.7 --disable-radix

python3 benchmark/gsm8k/bench_sglang.py --num-questions 2000 --parallel 2000

[2025-02-23 09:27:42 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/dockerx/0222/sglang/python/sglang/srt/managers/scheduler.py", line 1827, in run_scheduler_process
    scheduler.event_loop_normal()
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/dockerx/0222/sglang/python/sglang/srt/managers/scheduler.py", line 479, in event_loop_normal
    self.process_batch_result(batch, result)
  File "/dockerx/0222/sglang/python/sglang/srt/managers/scheduler.py", line 1120, in process_batch_result
    self.process_batch_result_prefill(batch, result)
  File "/dockerx/0222/sglang/python/sglang/srt/managers/scheduler.py", line 1181, in process_batch_result_prefill
    self.tree_cache.cache_unfinished_req(req)
  File "/dockerx/0222/sglang/python/sglang/srt/mem_cache/chunk_cache.py", line 62, in cache_unfinished_req
    kv_indices = self.req_to_token_pool.req_to_token[
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: index 2280 is out of bounds for dimension 0 with size 2280

sure, thansk1!

Hit out-of-bounds error at running gsm8k tests.

This issue has been fixed with merging upstream.
The root cause is that for MTP case, req_to_token_pool fail to reset its req_pool_idx and the req_pool_idx will go beyond the bound and hit the IndexError you met. Now the community have completely changed the original logic and this issue is fixed when I merge upstream into my repo.

@saienduri can we have test/srt/test_mla_deepseek_v3.py covered?

Please enable this for AMD

Added in test. @zhyncs Would you please review again?

Didn't noticed that I shouldn't use main branch. Close this one, track with new PR: #4631