Skip to content

[llm] bump vllm to 0.9.0.1#53443

Merged
kouroshHakha merged 17 commits intoray-project:masterfrom
lk-chen:bump_vllm_090
Jun 9, 2025
Merged

[llm] bump vllm to 0.9.0.1#53443
kouroshHakha merged 17 commits intoray-project:masterfrom
lk-chen:bump_vllm_090

Conversation

@lk-chen
Copy link
Copy Markdown
Contributor

@lk-chen lk-chen commented May 30, 2025

Why are these changes needed?

Many important reasons

  1. P/D disaggregation feature is added in 0.9.0
  2. There's this dependency chain: vllm0.8.5 -> opentelemetry1.26.0 -> protobuf3, ray wants to upgrade to protobuf4
  • vllm0.9.0 allows opentelemetry 1.27, which accepts protobuf4
  1. some users want vllm 0.9.0

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@lk-chen lk-chen requested review from a team as code owners May 30, 2025 17:46
@eicherseiji
Copy link
Copy Markdown
Contributor

eicherseiji commented May 30, 2025 •
edited
Loading

@lk-chen Can we change the import path here:

# python/ray/llm/_internal/serve/deployments/llm/vllm/vllm_engine.py
        if self.llm_config.log_engine_metrics:
            from ray.llm._internal.serve.deployments.llm.vllm.vllm_loggers import (
                RayPrometheusStatLogger,
            )

            # V1 AsyncLLMEngine does not yet support add_logger
            # For now, assume folks enabling log_engine_metrics do not require LoggingStatLogger, PrometheusStatLogger
            custom_stat_loggers = [RayPrometheusStatLogger]

From

            from ray.llm._internal.serve.deployments.llm.vllm.vllm_loggers import (

to

            from vllm.v1.metrics.ray_wrappers import (

and delete python/ray/llm/_internal/serve/deployments/llm/vllm/vllm_loggers.py?

release/llm_tests/serve/test_llm_serve_integration.py would have to be changed as well. Lmk if it's easier for me to just upload a patch.

@eicherseiji eicherseiji self-assigned this May 30, 2025
@eicherseiji
Copy link
Copy Markdown
Contributor

eicherseiji commented May 30, 2025

Taking this over

@lk-chen lk-chen changed the title [llm] bump vllm to 0.9.0 [llm] bump vllm to 0.9.0.1 Jun 2, 2025
@eicherseiji eicherseiji added the go add ONLY when ready to merge, run all tests label Jun 3, 2025
lk-chen
lk-chen commented Jun 3, 2025
View reviewed changes
@eicherseiji
Copy link
Copy Markdown
Contributor

eicherseiji commented Jun 4, 2025

Workarounds in this PR:

@eicherseiji eicherseiji removed request for a team and pcmoritz June 4, 2025 22:56
kouroshHakha
kouroshHakha reviewed Jun 7, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@kouroshHakha kouroshHakha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple of qs before we merge.

release/llm_tests/batch/test_batch_vllm.py Outdated
Copy link
Copy Markdown
Contributor

@kouroshHakha kouroshHakha Jun 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that if I run the script using compiled graph twice in a row the second one will die because of this hythersis effect?

Copy link
Copy Markdown
Contributor

@eicherseiji eicherseiji Jun 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah; I think pytest isn't letting the engine/pytorch clean up correctly between parameterizations so this is the workaround.

lk-chen and others added 17 commits June 8, 2025 08:41
@lk-chen @eicherseiji
bump 0.9.0
ecbd989 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen @eicherseiji
update req.txt
ca35ec4 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen @eicherseiji
clear todo
3213e3c 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen @eicherseiji
lint
924ee37 
Signed-off-by: Linkun Chen <github@lkchen.net>
@eicherseiji
Bump to 0.9.0.1
b5371ca 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Test unsetting CUDA_VISIBLE_DEVICES if empty
5342902 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Force torch cache cleanup
a3786da 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Check existence
3bccc6d 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Delete cache on worker
e73908b 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Change test model
a907e00 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Delay upstream RayPrometheusLogger pending bugfix
b68f153 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Change VLM model to mistral-community/pixtral-12b
38e3812 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Rearrange imports
1895fbc 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Revert to Qwen, try to fix cache cleanup logic
906e82d 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Adapt P/D mock vLLM engine to 0.9.0.1 EngineClient
3e160b4 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Workaround until vllm-project/vllm/ray-project#18979 is merged
190af3f 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji
Improve comments for workarounds
2e162db 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@kouroshHakha kouroshHakha merged commit a872286 into ray-project:master Jun 9, 2025
5 checks passed
@lk-chen lk-chen mentioned this pull request Jun 16, 2025
Closed
@lk-chen lk-chen deleted the bump_vllm_090 branch August 7, 2025 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eicherseiji eicherseiji eicherseiji left review comments

@kouroshHakha kouroshHakha kouroshHakha approved these changes

@aslonnie aslonnie aslonnie approved these changes

@richardliaw richardliaw Awaiting requested review from richardliaw richardliaw is a code owner

@edoakes edoakes Awaiting requested review from edoakes edoakes is a code owner

Assignees

@eicherseiji eicherseiji

Labels

go add ONLY when ready to merge, run all tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@lk-chen @eicherseiji @kouroshHakha @aslonnie