Skip to content

[PD] Support P/D role conversion of running sglang server#9325

Open
LLLL114 wants to merge 62 commits intosgl-project:mainfrom
LLLL114:main-support-pdconversion
Open

[PD] Support P/D role conversion of running sglang server#9325
LLLL114 wants to merge 62 commits intosgl-project:mainfrom
LLLL114:main-support-pdconversion

Conversation

@LLLL114
Copy link
Copy Markdown
Contributor

@LLLL114 LLLL114 commented Aug 19, 2025 •
edited
Loading

Motivation

parent issue: #8210

Currently, user can only adjust the P/D node ratio by killing and restarting the server.
This PR provide a http interface, which can reuse the loaded model weights and convert P/D role without killing server.

Modifications

  1. Add a http interface in http_server.py and mini_lb.py to support pass convert req in running server.
  2. Use the same DecodeReqToTokenPool in prefill and decode node. This allows to reuse cuda graph runner and attn_backend when convert between P and D.
  3. Add several threading event to break event loop in scheduler, MooncakeKVManager and hicache.
  4. release the resources of P/D when break event loop in scheduler, and complete the conversion process.

Accuracy Tests

launch 2P2D of Qwen3-1.7B with --enable-pd-convert:
default:

python sglang/benchmark/gsm8k/bench_sglang.py --port 8188 --parallel 30 --num-questions 300
100%|██| 300/300 [00:36<00:00,  8.30it/s]
Accuracy: 0.953
Invalid: 0.000
Latency: 36.419 s
Output throughput: 1024.578 token/s

use the req to convert P/D:

curl -X POST http://0.0.0.0:8188/convert_pd_role \
  -H "Content-Type: application/json" \
  -d '{"server_url": "http://0.0.0.0:30000","disable_cuda_graph": false}'
curl -X POST http://0.0.0.0:8188/convert_pd_role \
  -H "Content-Type: application/json" \
  -d '{"server_url": "http://0.0.0.0:31000","disable_cuda_graph": true,"disable_radix_cache": false}'

python sglang/benchmark/gsm8k/bench_sglang.py --port 8188 --parallel 30 --num-questions 300
100%|██| 300/300 [00:35<00:00,  8.55it/s]
Accuracy: 0.950
Invalid: 0.000
Latency: 35.368 s
Output throughput: 1040.173 token/s

Benchmarking and Profiling

  1. Compare covert time with different size of model:
截屏2025-08-19 10 23 57
  1. use genai_bench to compare
genai-bench benchmark --api-backend sglang \
            --api-base "http://localhost:8188" \
            --api-key "sglang" \
            --api-model-name "/root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B" \
            --model-tokenizer "/root/.cache/modelscope/hub/models/Qwen/Qwen3-0.6B" \
            --task text-to-text \
            --max-time-per-run 10 \
            --traffic-scenario "D(16,128)" \
            --max-requests-per-run 1000 \
            --server-engine "SGLang" \
            --num-concurrency 10
截屏2025-08-19 10 26 55

Checklist

LLLL114 added 29 commits July 7, 2025 17:07
@LLLL114
support convert prefill to decode without restart the server
e0224c1 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
support convert decode to prefill
d18d172 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main_for_sync' into main-support-pdconversion
1fa610d
@LLLL114
Fix reqtotokenpool memory leak
005316a 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
support reinit cuda graph
b4cac2e 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
support load PD parameters from request
1044f69 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
fix memory leak of tree cache
6820341 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
support set environ parameter
f495c0d 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
fix bug
d1e4bb7 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main_for_sync' into main-support-pdconversion
6988045
@LLLL114
fix radix cache bugs & support set args of cuda graph
bf42fe2 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
fix del TransferEngine bug & cuda_graph_max_bs bug
734ecc7 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
clean code
b3b99d8 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
fix bug
38e1841 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
minor
d125061 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main_for_sync' into main-support-pdconversion
d1039e2
@LLLL114
fix conflict
d999882 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main_for_sync' into main-support-pdconversion
dc85c96
@LLLL114
release source of hicache storage completely
8bf91ce 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
code style
2534ace 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
fix
1d10a3b 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main_for_sync' into main-support-pdconversion
f3d4b64
@LLLL114
fix
6a6611c 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
lint
d785bfc 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
reuse cuda graph when convert & no cuda memory leak
fdfb2c8 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
lint
a57e5a3 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
code
a57e3b9 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main_for_sync' into main-support-pdconversion
32c780f
@LLLL114
fix
9b8e98b 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114 LLLL114 requested a review from merrymercy as a code owner August 19, 2025 02:28
LLLL114 and others added 5 commits August 22, 2025 17:01
@LLLL114
statu code
cf0c592 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
better idle check
4fdd3ca 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
name fix
11affd9 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main' into main-support-pdconversion
fb6b849
@LLLL114
minor
22040e6 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@hzh0425 hzh0425 self-assigned this Aug 28, 2025
@zhyncs
Copy link
Copy Markdown
Collaborator

zhyncs commented Sep 1, 2025

@ShangmingCai

@ShangmingCai
Copy link
Copy Markdown
Collaborator

ShangmingCai commented Sep 1, 2025

@ShangmingCai

@zhyncs Sure, will start the reviewing process these days, just busy with too many things. And @hzh0425 will help me review this PR as well. When I think it is ready, I will ask Byron for another round of review.

LLLL114 and others added 15 commits September 1, 2025 17:16
@LLLL114
fix server args check logic
3d3cf47 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
minimize scheduler code change
8236641 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main' into main-support-pdconversion
1815045
@LLLL114
Merge branch 'main' into main-support-pdconversion
cb79d65
@LLLL114
support multi tokenizer mode and new mini_lb
0e10b6f 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main' into main-support-pdconversion
ef510a5
@LLLL114
Merge branch 'main_for_sync' into main-support-pdconversion
83b949e
@LLLL114
fix conflict as tokenizer manager refactored
b7f8f29 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main' into main-support-pdconversion
9d1cc9b
@LLLL114
update
4bfff7f 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main' into main-support-pdconversion
5f25553
@LLLL114
update
cd482ac 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
remove unused import
18b226e 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
update bootstrap_port setting when convert
c22481e 
Signed-off-by: huanglong <huanglong@linux.alibaba.com>
@LLLL114
Merge branch 'main' into main-support-pdconversion
a32cf79
@kartikx
Copy link
Copy Markdown
Contributor

kartikx commented Nov 19, 2025

I am interested in this feature. Is there an expected time by when it will be merged in? Thanks

@ShangmingCai
Copy link
Copy Markdown
Collaborator

ShangmingCai commented Nov 20, 2025

I am interested in this feature. Is there an expected time by when it will be merged in? Thanks

@kartikx This is a good feature, but currently it can not support PD with different TP/DP, so it can only serve limited scenes like small models with the same TP. We might consider integrating checkpoint engine to see in the future to test if we can fast redistribute the weight, but it might be no difference compared to shutting down and restarting another one with checkpoint engine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner

@zhyncs zhyncs Awaiting requested review from zhyncs

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

@xiezhq-hermann xiezhq-hermann Awaiting requested review from xiezhq-hermann xiezhq-hermann is a code owner

@CatherineSue CatherineSue Awaiting requested review from CatherineSue CatherineSue is a code owner

@slin1237 slin1237 Awaiting requested review from slin1237 slin1237 is a code owner

@ShangmingCai ShangmingCai Awaiting requested review from ShangmingCai ShangmingCai is a code owner

@ByronHsu ByronHsu Awaiting requested review from ByronHsu ByronHsu is a code owner

@Fridge003 Fridge003 Awaiting requested review from Fridge003 Fridge003 is a code owner

@JustinTong0323 JustinTong0323 Awaiting requested review from JustinTong0323 JustinTong0323 is a code owner

1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

@slin1237 slin1237

@hzh0425 hzh0425

@ShangmingCai ShangmingCai

Labels

high priority

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@LLLL114 @yiakwy-xpu-ml-framework-team @zhyncs @ShangmingCai @kartikx @slin1237 @hzh0425