[llm] Update vllm to 0.11.0 and Nixl to 0.6.0 by kouroshHakha · Pull Request #57201 · ray-project/ray

kouroshHakha · 2025-10-06T00:36:41Z

Updating vLLM and Nixl -- notable changes:

Remove V0 from stack
Embedding is added to V1. So it should be ok

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>

Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com>

kouroshHakha · 2025-10-07T02:00:52Z

python/ray/llm/tests/batch/gpu/processor/test_vllm_engine_proc.py

+        # runtime_env=dict(
+        #     env_vars=dict(
+        #         VLLM_USE_V1="0",
+        #     ),
+        # ),


if this passes the tests we should remove it.

nrghosh

unit tests fixed
release tests (probes/test_basic.py::test_logprobs)
logprobs problem: -1 is being interpreted as a vocab size instead of an invalid # for logprobs so all release tests are barfing b/c that probe test fails.
release tests 1p1d / 2p6d faliing due to nixl error

TypeError: nixl_agent_config.__init__() got an unexpected keyword argument 'num_threads'

num_threads was added to nixl_agent_config between release/0.4.1 and release/0.6.0 at 9f77cc4

Issue: release env is still using nixl=0.4.1 which is causing the break

New issue - nixl linker / backend error

failed to load plugin from /usr/local/nixl/lib/x86_64-linux-gnu/plugins/libplugin_UCX_MO.so: /usr/local/nixl/lib/x86_64-linux-gnu/plugins/libplugin_UCX_MO.so: undefined symbol: _ZN12nixlDescListI12nixlMetaDescEC1ERK10nixl_mem_tRKbRKi

(RayWorkerWrapper pid=128409) INFO 10-07 11:58:24 [factory.py:51] Creating v1 connector with name: NixlConnector and engine_id: 16561289-139e-44af-90e7-1c86a61925b2-10.0.111.115-40773
(RayWorkerWrapper pid=128409) INFO 10-07 11:58:24 [nixl_connector.py:465] Initializing NIXL wrapper
(RayWorkerWrapper pid=128409) INFO 10-07 11:58:24 [nixl_connector.py:466] Initializing NIXL worker 16561289-139e-44af-90e7-1c86a61925b2-10.0.111.115-40773
(RayWorkerWrapper pid=128409) E1007 11:58:24.959858  128409 nixl_plugin_manager.cpp:122] Failed to load plugin from /usr/local/nixl/lib/x86_64-linux-gnu/plugins/libplugin_UCX_MO.so: /usr/local/nixl/lib/x86_64-linux-gnu/plugins/libplugin_UCX_MO.so: undefined symbol: _ZN12nixlDescListI12nixlMetaDescEC1ERK10nixl_mem_tRKbRKi
(RayWorkerWrapper pid=128409) E1007 11:58:24.959885  128409 nixl_plugin_manager.cpp:288] Failed to load plugin 'UCX_MO' from any directory

Resolved by reinstalling nixl (local issue)

- vllm-project/vllm#23868 - PR in vLLM changed interpretation of num_logprobs = -1 - Overrides to model_config.get_vocab_size(), which triggers openai.APIError instead of openai.badRequestError - Test expects the latter, which causes failure - Instead of broadening / changing the expected failure type, we use -2 Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com>

Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com> Signed-off-by: nikhil <nikhil@anyscale.com>

python/ray/llm/_internal/batch/stages/vllm_engine_stage.py

aslonnie

can we wait till 2.50 branch cut to merge this?

kouroshHakha · 2025-10-07T21:24:19Z

@aslonnie ideally we can land this for summit?

nrghosh · 2025-10-07T21:26:03Z

can we wait till 2.50 branch cut to merge this?

I think we want to get this in ASAP, it has a lot of performance improvements

eicherseiji · 2025-10-07T21:29:20Z

can we wait till 2.50 branch cut to merge this?

If we ship the switch to OpenTelemetry without vLLM 0.11.0 we will have no vLLM metrics in Ray 2.50. cc: @can-anyscale

nrghosh

Unit tests and release tests both passing

Doc build failure seems ephemeral / unrelated, as full doc build succeeds locally with this branch.

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com> Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com> Signed-off-by: nikhil <nikhil@anyscale.com> Co-authored-by: Nikhil Ghosh <nikhil@anyscale.com> Co-authored-by: Nikhil G <nrghosh@users.noreply.github.com>

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com> Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com> Signed-off-by: nikhil <nikhil@anyscale.com> Co-authored-by: Nikhil Ghosh <nikhil@anyscale.com> Co-authored-by: Nikhil G <nrghosh@users.noreply.github.com> Signed-off-by: Josh Kodi <joshkodi@gmail.com>

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com> Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com> Signed-off-by: nikhil <nikhil@anyscale.com> Co-authored-by: Nikhil Ghosh <nikhil@anyscale.com> Co-authored-by: Nikhil G <nrghosh@users.noreply.github.com>

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com> Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com> Signed-off-by: nikhil <nikhil@anyscale.com> Co-authored-by: Nikhil Ghosh <nikhil@anyscale.com> Co-authored-by: Nikhil G <nrghosh@users.noreply.github.com> Signed-off-by: xgui <xgui@anyscale.com>

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com> Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com> Signed-off-by: nikhil <nikhil@anyscale.com> Co-authored-by: Nikhil Ghosh <nikhil@anyscale.com> Co-authored-by: Nikhil G <nrghosh@users.noreply.github.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com> Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com> Signed-off-by: nikhil <nikhil@anyscale.com> Co-authored-by: Nikhil Ghosh <nikhil@anyscale.com> Co-authored-by: Nikhil G <nrghosh@users.noreply.github.com>

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com> Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com> Signed-off-by: nikhil <nikhil@anyscale.com> Co-authored-by: Nikhil Ghosh <nikhil@anyscale.com> Co-authored-by: Nikhil G <nrghosh@users.noreply.github.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com> Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com> Signed-off-by: nikhil <nikhil@anyscale.com> Co-authored-by: Nikhil Ghosh <nikhil@anyscale.com> Co-authored-by: Nikhil G <nrghosh@users.noreply.github.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

kouroshHakha added 3 commits October 5, 2025 17:27

wip

6f62620

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>

wip

fc75df3

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>

wip

9eb0c7e

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>

kouroshHakha added the go add ONLY when ready to merge, run all tests label Oct 6, 2025

kouroshHakha and others added 3 commits October 5, 2025 17:45

wip

1462445

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>

log_engine_metrics by default

5f5f3fb

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>

wip - fix test import

2c00977

Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com>

nrghosh force-pushed the kh/update-vllm-0.11.0 branch from 2309954 to 2c00977 Compare October 7, 2025 01:52

kouroshHakha commented Oct 7, 2025

View reviewed changes

nrghosh reviewed Oct 7, 2025

View reviewed changes

nrghosh force-pushed the kh/update-vllm-0.11.0 branch from 4717751 to e62627f Compare October 7, 2025 05:23

eicherseiji mentioned this pull request Oct 7, 2025

Bump vLLM -> 0.11.0 #57216

Closed

8 tasks

update nixl in llm-requirements and re-compile deps

8e6b2ab

Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com> Signed-off-by: nikhil <nikhil@anyscale.com>

nrghosh force-pushed the kh/update-vllm-0.11.0 branch from 35427b5 to 8e6b2ab Compare October 7, 2025 21:02

nrghosh marked this pull request as ready for review October 7, 2025 21:04

nrghosh requested review from a team, aslonnie, edoakes and richardliaw as code owners October 7, 2025 21:04

Merge branch 'master' into kh/update-vllm-0.11.0

ca4215d

cursor bot reviewed Oct 7, 2025

View reviewed changes

python/ray/llm/_internal/batch/stages/vllm_engine_stage.py Show resolved Hide resolved

aslonnie reviewed Oct 7, 2025

View reviewed changes

nrghosh approved these changes Oct 8, 2025

View reviewed changes

kouroshHakha enabled auto-merge (squash) October 8, 2025 01:01

aslonnie self-requested a review October 8, 2025 01:07

ray-gardener bot added the serve Ray Serve Related Issue label Oct 8, 2025

ray-gardener bot added the llm label Oct 8, 2025

aslonnie approved these changes Oct 8, 2025

View reviewed changes

kouroshHakha merged commit 32e59c2 into ray-project:master Oct 8, 2025
6 of 7 checks passed

eicherseiji mentioned this pull request Oct 10, 2025

[serve][llm] Enable engine metrics by default #57615

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[llm] Update vllm to 0.11.0 and Nixl to 0.6.0#57201

[llm] Update vllm to 0.11.0 and Nixl to 0.6.0#57201
kouroshHakha merged 9 commits intoray-project:masterfrom
kouroshHakha:kh/update-vllm-0.11.0

kouroshHakha commented Oct 6, 2025

Uh oh!

kouroshHakha Oct 7, 2025

Uh oh!

nrghosh left a comment •

edited

Loading

Uh oh!

Uh oh!

aslonnie left a comment

Uh oh!

kouroshHakha commented Oct 7, 2025

Uh oh!

nrghosh commented Oct 7, 2025 •

edited

Loading

Uh oh!

eicherseiji commented Oct 7, 2025 •

edited

Loading

Uh oh!

nrghosh left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

kouroshHakha commented Oct 6, 2025

Uh oh!

kouroshHakha Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

nrghosh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aslonnie left a comment

Choose a reason for hiding this comment

Uh oh!

kouroshHakha commented Oct 7, 2025

Uh oh!

nrghosh commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eicherseiji commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nrghosh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nrghosh left a comment •

edited

Loading

nrghosh commented Oct 7, 2025 •

edited

Loading

eicherseiji commented Oct 7, 2025 •

edited

Loading