[Chore][CI]: K3 MP output token quantity tolerance#3030
Merged
sammshen merged 1 commit intoLMCache:devfrom Apr 14, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a 1% token tolerance in the vLLM benchmark verification scripts to handle minor drifts in token counts and refactors the verification logic into a helper function. A critical issue was identified in pyproject.toml where the torch version was updated to 2.11.0, a version that does not exist on PyPI, which will lead to build failures.
| "setuptools>=77.0.3,<81.0.0", | ||
| "setuptools_scm>=8", | ||
| "torch==2.10.0", | ||
| "torch==2.11.0", |
Contributor
There was a problem hiding this comment.
The version 2.11.0 for torch does not exist on PyPI (the current stable versions are in the 2.x range, e.g., 2.5.1, 2.6.0). This appears to be a typo, likely intended to be 2.1.1 (given the previous value was 2.10.0, which was likely a typo for 2.1.0). Using a non-existent version will cause build failures when resolving dependencies for the build system.
d8d6c5a to
d421250
Compare
vLLM's RandomDataset decodes and re-encodes generated token sequences (vllm/benchmarks/datasets.py) to avoid string-level drift, but the roundtrip is not guaranteed to preserve exact token counts — the benchmark itself only warns when token_mismatch != 0. The strict -eq assertion against NUM_PROMPTS * RANDOM_INPUT_LEN was failing with a 0.08% overage (500400 vs 500000) on Qwen3-14B after a vLLM upgrade. Switch to a ±1% tolerance check, which matches the benchmark's own semantics while still catching real workload-size regressions. Signed-off-by: Samuel Shen <slshen@uchciago.edu>
d421250 to
a9f407e
Compare
royyhuang
approved these changes
Apr 14, 2026
ekaynar
pushed a commit
to ekaynar/LMCache
that referenced
this pull request
Apr 15, 2026
[CI] Allow 1% tolerance on vllm_bench total_input_tokens check vLLM's RandomDataset decodes and re-encodes generated token sequences (vllm/benchmarks/datasets.py) to avoid string-level drift, but the roundtrip is not guaranteed to preserve exact token counts — the benchmark itself only warns when token_mismatch != 0. The strict -eq assertion against NUM_PROMPTS * RANDOM_INPUT_LEN was failing with a 0.08% overage (500400 vs 500000) on Qwen3-14B after a vLLM upgrade. Switch to a ±1% tolerance check, which matches the benchmark's own semantics while still catching real workload-size regressions. Signed-off-by: Samuel Shen <slshen@uchciago.edu> Co-authored-by: Samuel Shen <slshen@uchciago.edu>
ftian1
pushed a commit
to ftian1/LMCache
that referenced
this pull request
Apr 20, 2026
[CI] Allow 1% tolerance on vllm_bench total_input_tokens check vLLM's RandomDataset decodes and re-encodes generated token sequences (vllm/benchmarks/datasets.py) to avoid string-level drift, but the roundtrip is not guaranteed to preserve exact token counts — the benchmark itself only warns when token_mismatch != 0. The strict -eq assertion against NUM_PROMPTS * RANDOM_INPUT_LEN was failing with a 0.08% overage (500400 vs 500000) on Qwen3-14B after a vLLM upgrade. Switch to a ±1% tolerance check, which matches the benchmark's own semantics while still catching real workload-size regressions. Signed-off-by: Samuel Shen <slshen@uchciago.edu> Co-authored-by: Samuel Shen <slshen@uchciago.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Special notes for your reviewers:
If applicable:
Note
Low Risk
Low risk: CI-only bash script changes that relax an assertion to account for known vLLM random-dataset token length drift; no production code or security-sensitive logic touched.
Overview
Updates the Buildkite vLLM bench verification scripts to treat
total_input_tokensas approximate rather than requiring an exact match.Both
run-vllm-bench.shvariants now compute a 1% tolerance and use a sharedcheck_input_tokenshelper for LMCache and baseline runs, reducing spurious CI failures caused by vLLM’s random dataset re-tokenization drift.Reviewed by Cursor Bugbot for commit a9f407e. Bugbot is set up for automated code reviews on this repo. Configure here.