Skip to content

[PD] Support PD disaggregation with Prefill PP#8846

Merged
zhyncs merged 35 commits intomainfrom
fixing_pd_pp
Aug 17, 2025
Merged

[PD] Support PD disaggregation with Prefill PP#8846
zhyncs merged 35 commits intomainfrom
fixing_pd_pp

Conversation

@ShangmingCai
Copy link
Copy Markdown
Collaborator

@ShangmingCai ShangmingCai commented Aug 6, 2025 •
edited
Loading

Motivation

To reduce TTFT, we want to support Prefill PP with PD disaggregation through this PR.

Decode PP will be supported in the next PR. @ssssnow

Accuracy Test

Qwen3-8B(Prefill TP 2 PP2, Decode TP 1):
python sglang/benchmark/gsm8k/bench_sglang.py --port 8000 --parallel 300 --num-questions 300
100%|█| 300/300 [02:29<00:00,  2.0
Accuracy: 0.943
Invalid: 0.000
Latency: 149.493 s
Output throughput: 249.315 token/s

Qwen3-8B(Prefill TP 2 PP2, Decode TP 2):
python sglang/benchmark/gsm8k/bench_sglang.py --port 8000 --parallel 300 --num-questions 300
100%|█████| 300/300 [00:24<00:00, 12.01it/s]
Accuracy: 0.947
Invalid: 0.000
Latency: 25.186 s
Output throughput: 1469.696 token/s

Qwen3-8B(Prefill TP 1 PP2, Decode TP 2):
python sglang/benchmark/gsm8k/bench_sglang.py --port 8000 --parallel 300 --num-questions 300
100%|█| 300/300 [00:26<00:00, 11.15i
Accuracy: 0.950
Invalid: 0.000
Latency: 27.047 s
Output throughput: 1378.700 token/s

Qwen3-8B(Prefill DP 2 TP 1 PP2, Decode TP 4):
python sglang/benchmark/gsm8k/bench_sglang.py --port 8000 --parallel 300 --num-questions 300
100%|█| 300/300 [00:20<00:00, 14.44i
Accuracy: 0.950
Invalid: 0.000
Latency: 21.158 s
Output throughput: 1755.561 token/s

DeepSeek(Prefill TP 8 PP2, Decode TP 16): verified by @ssssnow 
python benchmark/gsm8k/bench_sglang.py --port 9091 --parallel 300 --num-questions 300
Accuracy: 0.947
Invalid: 0.000
Latency: 32.184 s
Output throughput: 927.104 token/s

Benchmark & Profiling

Checklist

ShangmingCai and others added 17 commits July 30, 2025 19:32
@ShangmingCai
[PD] Refactor parallel sizes and add pp support for mooncake
4414782 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
revert false remove
4774f60 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
fix isort
d8dfc02 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
fix log error
2e639c0 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
fix comment
940048b 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
fix comment
8d7bd97 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
fix comment
b0c5bbf 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
fix decode
4a51b04 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
fix pd pp
223b8af 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
fix prefill pp schedule issues
e8e4df9
@ShangmingCai
update mooncake meta
d13e0f3 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@Ying1123 @ShangmingCai
update
d241c9b
@ShangmingCai
port from 2aed40c
5b9d63d 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
Merge remote-tracking branch 'up/main' into fixing_pd_pp
d610d68
@ShangmingCai
fix missing import
6d55920 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
fix layer calculate
ac563d5 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
@ShangmingCai
update
b591734 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
@gemini-code-assist
Copy link
Copy Markdown
Contributor

gemini-code-assist Bot commented Aug 6, 2025

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

ShangmingCai and others added 3 commits August 6, 2025 13:07
@ShangmingCai
fix lint
bfe2dac 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
@ShangmingCai
fix codespell
1d2e0ff 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
@ssssnow
Fixing pd pp fork (#8858)
9b95d4f
@ssssnow
Copy link
Copy Markdown
Contributor

ssssnow commented Aug 6, 2025

Notice: pp related modification in deepseek_v2.py is borrowed from #6434, related author should be added as co-authors

@ShangmingCai ShangmingCai changed the title [WIP][PD] Support PD disaggregation with Prefill PP [PD] Support PD disaggregation with Prefill PP Aug 7, 2025
@ShangmingCai ShangmingCai mentioned this pull request Aug 3, 2025
Closed
22 tasks
ShangmingCai and others added 14 commits August 11, 2025 10:17
@zhyncs zhyncs merged commit 384f8ab into main Aug 17, 2025
126 of 130 checks passed
@zhyncs zhyncs deleted the fixing_pd_pp branch August 17, 2025 01:31
@hebiao064 hebiao064 mentioned this pull request Aug 17, 2025
Closed
4 tasks
narutolhy pushed a commit to narutolhy/sglang that referenced this pull request Aug 17, 2025
@ShangmingCai @Ying1123
@ssssnow @zhjc1124 @narutolhy
[PD] Support PD disaggregation with Prefill PP (sgl-project#8846)
ad9c55d 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: root <huzhiyuan@xiaohongshu.com>
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: Francis <38564764+ssssnow@users.noreply.github.com>
Co-authored-by: zitto <zhjc1124@gmail.com>
@ShangmingCai ShangmingCai mentioned this pull request Aug 29, 2025
Open
MahmoudAshraf97 pushed a commit to MahmoudAshraf97/sglang that referenced this pull request Sep 8, 2025
@ShangmingCai @Ying1123
@ssssnow @zhjc1124 @MahmoudAshraf97
[PD] Support PD disaggregation with Prefill PP (sgl-project#8846)
82827e5 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: root <huzhiyuan@xiaohongshu.com>
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: Francis <38564764+ssssnow@users.noreply.github.com>
Co-authored-by: zitto <zhjc1124@gmail.com>
@zhjc1124 zhjc1124 mentioned this pull request Oct 16, 2025
Closed
6 tasks
@ShangmingCai ShangmingCai mentioned this pull request Oct 20, 2025
Open
25 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner

@xiezhq-hermann xiezhq-hermann Awaiting requested review from xiezhq-hermann xiezhq-hermann is a code owner

@ByronHsu ByronHsu Awaiting requested review from ByronHsu ByronHsu is a code owner

@zhyncs zhyncs Awaiting requested review from zhyncs

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ShangmingCai @ssssnow @zhyncs @Ying1123 @zhjc1124