[Spec] Split accept_length into num_accepted_drafts and num_accepted_tokens#23962
Merged
[Spec] Split accept_length into num_accepted_drafts and num_accepted_tokens#23962
accept_length into num_accepted_drafts and num_accepted_tokens#23962Conversation
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
…pt-length # Conflicts: # python/sglang/srt/managers/io_struct.py # python/sglang/srt/managers/scheduler_output_processor_mixin.py # python/sglang/srt/managers/tokenizer_manager.py # python/sglang/srt/managers/utils.py # python/sglang/srt/speculative/dflash_info.py # python/sglang/srt/speculative/dflash_worker.py # python/sglang/srt/speculative/eagle_worker.py # python/sglang/srt/speculative/multi_layer_eagle_worker.py # python/sglang/srt/speculative/ngram_info.py # python/sglang/srt/speculative/ngram_worker.py
spec_info.accept_length always drafts-only; rename to num_accepted_draftsaccept_length into num_accepted_drafts and num_accepted_tokens
Collaborator
Author
|
/rerun-stage stage-c-test-8-gpu-h20 |
Collaborator
Author
|
/rerun-stage stage-c-test-4-gpu-h100 |
Contributor
|
✅ Triggered |
Contributor
|
✅ Triggered |
vguduruTT
pushed a commit
to vguduruTT/sglang
that referenced
this pull request
May 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up #23530.
Summary
accept_lengthinto two explicit fields onEagleDraftInput/NgramVerifyInput:num_accepted_drafts(strict drafts-only) andnum_accepted_tokens(includes the bonus token; equals drafts + 1 per req)accept_length.add_(1)in-place mutation that flipped the variable's semantics mid-functionaccept/draftnaming convention from [Spec] Fixspec_accept_rateand unifyaccept/draftnaming #23530: name containsdraft→ drafts-only; containsacceptwithoutdraft→ includes bonusWhy dual-tensor
+ 1patterns scattered across attention backends and CUDA graph runnersbs-sized int32 tensor per spec_info (~few KB), negligibleChanges
Spec info classes
EagleDraftInput.num_accepted_tokens: torch.Tensorandnum_accepted_tokens_cpu: List[int]added alongside the drafts-only fieldsNgramVerifyInput.num_accepted_tokensadded alongsidenum_accepted_draftsLifecycle decoupling
eagle_info_v2.py:sample()returnsnum_accepted_drafts + 1out-of-place instead of.add_(1)mutationeagle_info.py:prepare_extend_after_decode()no longer mutatesself.num_accepted_drafts; uses localextend_lensfor the kernel calleagle_worker_v2.pyV2 path sets both fields frombatch_result.accept_lens(includes bonus) andaccept_lens - 1(drafts-only)CUDA graph runners
EagleDraftExtendInputBuffersandMultiLayerEagleDraftExtendInputBuffersadd a parallelnum_accepted_tokensbuffer; copy and alias both fields during replayAttention backends
aiter,flashattention,trtllm_mha,nsa,nsa_backend_mtp_precompute,wave,triton: readspec_info.num_accepted_tokens(or_cpu) directly, removing the explicit+ 1Mechanical rename
accept_length→num_accepted_draftsacross spec workers, info classes, attention backends, cuda graph runners, testsaccept_lens/accept_len(the rule applies to the value, not just the previous name)SpeculativeMetrics.accept_lengthretained (vLLM-compatible metric, includes bonus)Breaking
meta_infokeys, and CLI args unchanged.Test plan
test_eagle_infer_a.pytest_eagle_infer_b.pytest_eagle_infer_beta.pytest_ngram_speculative_decoding.pytest_dflash.pytest_standalone_speculative_decoding.pytest_eagle_dp_attention.py(multi_layer_eagle_worker path)nsa_backend_mtp_precomputeextend path)