[CI] Split V1 Others into 3 separate jobs#37016
Merged
khluu merged 1 commit intovllm-project:mainfrom Mar 23, 2026
Merged
Conversation
Split the single ~55m "V1 Others" job into three smaller jobs targeting ~20m each: - V1 Spec Decode (~19m): spec_decode tests - V1 Sample + Logits (~18m): sample, logits_processors, test_oracle, test_request, test_outputs - V1 Core + KV + Metrics (~18m): core, executor, kv_offload, worker, kv_connector/unit, metrics, lm_eval integration test Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Code Review
This pull request refactors a single large CI job into three smaller, parallel jobs to improve runtime. The split of tests and dependencies appears logical. However, I've identified a missing configuration in one of the new jobs which contradicts the stated goal in the pull request description and could lead to incomplete test coverage on AMD hardware.
Comment on lines
+42
to
65
| - label: V1 Core + KV + Metrics | ||
| timeout_in_minutes: 30 | ||
| source_file_dependencies: | ||
| - vllm/ | ||
| - tests/v1/core | ||
| - tests/v1/executor | ||
| - tests/v1/kv_offload | ||
| - tests/v1/worker | ||
| - tests/v1/kv_connector/unit | ||
| - tests/v1/metrics | ||
| - tests/entrypoints/openai/correctness/test_lmeval.py | ||
| commands: | ||
| - uv pip install --system -r /vllm-workspace/requirements/kv_connectors.txt | ||
| - export VLLM_WORKER_MULTIPROC_METHOD=spawn | ||
| # split the test to avoid interference | ||
| - pytest -v -s -m 'not cpu_test' v1/core | ||
| - pytest -v -s v1/executor | ||
| - pytest -v -s v1/kv_offload | ||
| - pytest -v -s v1/sample | ||
| - pytest -v -s v1/logits_processors | ||
| - pytest -v -s v1/worker | ||
| # TODO: create another `optional` test group for slow tests | ||
| - pytest -v -s -m 'not slow_test' v1/spec_decode | ||
| - pytest -v -s -m 'not cpu_test' v1/kv_connector/unit | ||
| - pytest -v -s -m 'not cpu_test' v1/metrics | ||
| - pytest -v -s v1/test_oracle.py | ||
| - pytest -v -s v1/test_request.py | ||
| - pytest -v -s v1/test_outputs.py | ||
| # Integration test for streaming correctness (requires special branch). | ||
| - pip install -U git+https://github.com/robertgshaw2-redhat/lm-evaluation-harness.git@streaming-api | ||
| - pytest -v -s entrypoints/openai/correctness/test_lmeval.py::test_lm_eval_accuracy_v1_engine |
Contributor
There was a problem hiding this comment.
The V1 Core + KV + Metrics job is missing the mirror configuration for AMD GPUs. The PR description states that 'AMD mirrors preserved for all three jobs', but this job lacks the configuration present in the other two new jobs. This omission will prevent this set of tests from running on the specified AMD hardware.
- label: V1 Core + KV + Metrics
timeout_in_minutes: 30
source_file_dependencies:
- vllm/
- tests/v1/core
- tests/v1/executor
- tests/v1/kv_offload
- tests/v1/worker
- tests/v1/kv_connector/unit
- tests/v1/metrics
- tests/entrypoints/openai/correctness/test_lmeval.py
commands:
- uv pip install --system -r /vllm-workspace/requirements/kv_connectors.txt
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
# split the test to avoid interference
- pytest -v -s -m 'not cpu_test' v1/core
- pytest -v -s v1/executor
- pytest -v -s v1/kv_offload
- pytest -v -s v1/worker
- pytest -v -s -m 'not cpu_test' v1/kv_connector/unit
- pytest -v -s -m 'not cpu_test' v1/metrics
# Integration test for streaming correctness (requires special branch).
- pip install -U git+https://github.com/robertgshaw2-redhat/lm-evaluation-harness.git@streaming-api
- pytest -v -s entrypoints/openai/correctness/test_lmeval.py::test_lm_eval_accuracy_v1_engine
mirror:
amd:
device: mi325_1
depends_on:
- image-build-amd
Collaborator
Author
|
V1 Spec Decode : 24m |
Collaborator
|
@khluu is this planned to get merged? Only asking so that I know if I should revert test amd yaml as well. |
Collaborator
Author
I'll merge it, but for future reference I think you should only change |
jikunshang
approved these changes
Mar 23, 2026
RhizoNymph
pushed a commit
to RhizoNymph/vllm
that referenced
this pull request
Mar 26, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
HenryTangDev
pushed a commit
to HenryTangMain/vllm
that referenced
this pull request
Mar 27, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
khairulkabir1661
pushed a commit
to khairulkabir1661/vllm
that referenced
this pull request
Mar 27, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Monishver11
pushed a commit
to Monishver11/vllm
that referenced
this pull request
Mar 27, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com>
nithinvc
pushed a commit
to nithinvc/vllm
that referenced
this pull request
Mar 27, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Nithin Chalapathi <nithin.ch10@gmail.com>
JiantaoXu
pushed a commit
to JiantaoXu/vllm
that referenced
this pull request
Mar 28, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
vrdn-23
pushed a commit
to vrdn-23/vllm
that referenced
this pull request
Mar 30, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vinay Damodaran <vrdn@hey.com>
EricccYang
pushed a commit
to EricccYang/vllm
that referenced
this pull request
Apr 1, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: EricccYang <yangyang4991@gmail.com>
liuchenbing2026
pushed a commit
to liuchenbing2026/vllm
that referenced
this pull request
Apr 4, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
rishitdholakia13
pushed a commit
to rishitdholakia13/vllm
that referenced
this pull request
Apr 7, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: rishitdholakia13 <rishit+github@cohere.com>
puririshi98
pushed a commit
to puririshi98/vllm
that referenced
this pull request
Apr 7, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Rishi Puri <riship@nvidia.com>
big-yellow-duck
pushed a commit
to EmbeddedLLM/vllm
that referenced
this pull request
Apr 8, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
source_file_dependenciesinstead of the broadtests/v1Test plan
AI assistance was used (Claude). This is not duplicating any existing PR.
🤖 Generated with Claude Code