Skip to content

[PD] Fix the infinite loop in deocde resolve_pending_reqs#20371

Merged
hnyls2002 merged 5 commits intomainfrom
fix_decode_loop
Mar 11, 2026
Merged

[PD] Fix the infinite loop in deocde resolve_pending_reqs#20371
hnyls2002 merged 5 commits intomainfrom
fix_decode_loop

Conversation

@ShangmingCai
Copy link
Copy Markdown
Collaborator

@ShangmingCai ShangmingCai commented Mar 11, 2026 •
edited
Loading

Motivation

Fix #20252
Can only reproduce the bug when using sglang-router (mini-lb will reject the req if prefill is dead immediately, so it won't go into the pending_reqs)

Fix plan:

  • Fix the infinite loop in decode resolve_pending_reqs
  • Reduce max_retries, 20 seconds is too long. Also, the sglang-router also has a retry mechanism (it can handle the case if this request really needs to be done), so we don't need to retry 20 times to block the scheduling. (Also consider running _resolve_pending_reqs asynchronously in the next PR)

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Review Process

  1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
  4. After green CI and required approvals, ask Merge Oncalls to merge.

@ShangmingCai
[PD] Fix the infinite loop in deocde resolve_pending_reqs
a77c2fe 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
@ShangmingCai
Copy link
Copy Markdown
Collaborator Author

ShangmingCai commented Mar 11, 2026

/rerun-stage stage-c-test-8-gpu-h20

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 11, 2026

✅ Triggered stage-c-test-8-gpu-h20 to run independently (skipping dependencies).

@ShangmingCai ShangmingCai mentioned this pull request Mar 11, 2026
Closed
5 tasks
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 11, 2026

🔗 View workflow run

@gemini-code-assist
Copy link
Copy Markdown
Contributor

gemini-code-assist Bot commented Mar 11, 2026

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@ShangmingCai
update metric
508b565 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
@ShangmingCai
reduce max_retries
5f89e72 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
@ShangmingCai
revert
a3f3296 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
@ShangmingCai
Copy link
Copy Markdown
Collaborator Author

ShangmingCai commented Mar 11, 2026

/rerun-stage stage-c-test-8-gpu-h20

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 11, 2026

✅ Triggered stage-c-test-8-gpu-h20 to run independently (skipping dependencies).

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 11, 2026

🔗 View workflow run

@ShangmingCai
reduce max_retries
b2fad09 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
@ShangmingCai
Copy link
Copy Markdown
Collaborator Author

ShangmingCai commented Mar 11, 2026

/rerun-stage stage-c-test-8-gpu-h20

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 11, 2026

✅ Triggered stage-c-test-8-gpu-h20 to run independently (skipping dependencies).

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 11, 2026

🔗 View workflow run

@hnyls2002
Copy link
Copy Markdown
Collaborator

hnyls2002 commented Mar 11, 2026

Actually, it is a bug introduced by me; I only considered one prefill instance. Your fix is right, we should group all the requests by the bootstrap address.

@hnyls2002 hnyls2002 merged commit af4c289 into main Mar 11, 2026
87 of 95 checks passed
@hnyls2002 hnyls2002 deleted the fix_decode_loop branch March 11, 2026 21:11
@hnyls2002
Copy link
Copy Markdown
Collaborator

hnyls2002 commented Mar 11, 2026

BTW, can we remove the ensure_parallel_info in the receiver? I thought that if we could gather the parallel info retrieval in the same place (before adding it to the queue), then the code logic would be much cleaner.

@ShangmingCai

@ShangmingCai ShangmingCai mentioned this pull request Mar 13, 2026
Merged
5 tasks
liubiyongge pushed a commit to liubiyongge/sglang that referenced this pull request Mar 13, 2026
@ShangmingCai @liubiyongge
[PD] Fix the infinite loop in deocde resolve_pending_reqs (sgl-projec…
a03cf1e 
…t#20371)

Signed-off-by: Shangming Cai <csmthu@gmail.com>
whybeyoung pushed a commit to whybeyoung/sglang that referenced this pull request Mar 14, 2026
@ShangmingCai @whybeyoung
[PD] Fix the infinite loop in deocde resolve_pending_reqs (sgl-projec…
2975def 
…t#20371)

Signed-off-by: Shangming Cai <csmthu@gmail.com>
Wangzheee pushed a commit to Wangzheee/sglang that referenced this pull request Mar 21, 2026
@ShangmingCai @Wangzheee
[PD] Fix the infinite loop in deocde resolve_pending_reqs (sgl-projec…
724d487 
…t#20371)

Signed-off-by: Shangming Cai <csmthu@gmail.com>
HanHan009527 pushed a commit to bytedance-iaas/sglang that referenced this pull request Apr 3, 2026
@ShangmingCai @HanHan009527
[PD] Fix the infinite loop in deocde resolve_pending_reqs (sgl-projec…
9c19a83 
…t#20371)

Signed-off-by: Shangming Cai <csmthu@gmail.com>
JustinTong0323 pushed a commit to JustinTong0323/sglang that referenced this pull request Apr 7, 2026
@ShangmingCai
[PD] Fix the infinite loop in deocde resolve_pending_reqs (sgl-projec…
62fc6f3 
…t#20371)

Signed-off-by: Shangming Cai <csmthu@gmail.com>
yhyang201 pushed a commit to yhyang201/sglang that referenced this pull request Apr 22, 2026
@ShangmingCai
[PD] Fix the infinite loop in deocde resolve_pending_reqs (sgl-projec…
379c30f 
…t#20371)

Signed-off-by: Shangming Cai <csmthu@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hnyls2002 hnyls2002 hnyls2002 approved these changes

@ByronHsu ByronHsu Awaiting requested review from ByronHsu ByronHsu is a code owner

Assignees

@hnyls2002 hnyls2002

@ShangmingCai ShangmingCai

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] Large scale PD Disagression bug : cascading failure in Decode/prefill servers when corresponding Prefill/decode servers go offline

2 participants

@ShangmingCai @hnyls2002