[serve][llm] Disable model downloading for RunAI streamer, introduce optimized download function by hao-aaron · Pull Request #57854 · ray-project/ray

hao-aaron · 2025-10-17T19:36:37Z

Description

When running serve llm with runai streamer, current codepath unnecessarily downloads model first. Also, current model download function is not parallelized.

Changes:

change worker_node_download_model depending on load_format in LLMConfig.engine_kwargs
add new download_model_parallel function which is used by CloudDownloader callback
add unit tests for LLMConfig to ensure download model is set correctly

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

gemini-code-assist

Code Review

This pull request introduces two main improvements: disabling model downloads for streaming loaders like runai_streamer and optimizing cloud downloads by parallelizing them. The logic to conditionally disable downloads based on load_format is well-implemented and tested. The new parallel download function download_files_parallel correctly uses pyarrow for efficient transfers. However, I've found a critical typo that would cause runtime failures and a high-severity issue in error handling that could lead to silent partial downloads. I've also suggested a minor clarification to a docstring. Once these points are addressed, the PR will be in great shape.

python/ray/llm/_internal/common/utils/cloud_utils.py

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

python/ray/llm/_internal/common/utils/cloud_utils.py

python/ray/llm/tests/serve/cpu/configs/test_models.py

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

ruisearch42

Overall LGTM

python/ray/llm/_internal/common/callbacks/cloud_downloader.py

python/ray/llm/_internal/serve/utils/node_initialization_utils.py

python/ray/llm/_internal/common/utils/cloud_utils.py

…ions Signed-off-by: ahao-anyscale <ahao@anyscale.com>

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

hao-aaron · 2025-10-21T16:54:25Z

failing lmcache bug unrelated to this PR, see LMCache/LMCache#1768

ruisearch42

Some readability & cleanness suggestions

python/ray/llm/_internal/common/callbacks/cloud_downloader.py

python/ray/llm/_internal/common/utils/cloud_utils.py

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

ruisearch42

Thanks for addressing the comments

python/ray/llm/_internal/common/callbacks/cloud_downloader.py

angelinalg

stamp

kouroshHakha · 2025-10-28T21:51:30Z

Hehe Why did no one press go label :D

…optimized download function (ray-project#57854) Signed-off-by: ahao-anyscale <ahao@anyscale.com>

…optimized download function (ray-project#57854) Signed-off-by: ahao-anyscale <ahao@anyscale.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…optimized download function (ray-project#57854) Signed-off-by: ahao-anyscale <ahao@anyscale.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

…optimized download function (ray-project#57854) Signed-off-by: ahao-anyscale <ahao@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

runai streamer do not download model, add optimized download code

c69156c

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

hao-aaron requested a review from a team as a code owner October 17, 2025 19:36

This comment was marked as outdated.

Sign in to view

gemini-code-assist bot reviewed Oct 17, 2025

View reviewed changes

python/ray/llm/_internal/common/utils/cloud_utils.py Outdated Show resolved Hide resolved

python/ray/llm/_internal/common/utils/cloud_utils.py Outdated Show resolved Hide resolved

python/ray/llm/_internal/common/utils/cloud_utils.py Outdated Show resolved Hide resolved

kouroshHakha changed the title ~~[serve.llm] Disable model downloading for RunAI streamer, introduce optimized download function~~ [serve][llm] Disable model downloading for RunAI streamer, introduce optimized download function Oct 17, 2025

fix bugs

1dfa841

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

This comment was marked as outdated.

Sign in to view

fix bugs

1405e2a

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

hao-aaron force-pushed the startup-optimizations branch from 7f73524 to 1405e2a Compare October 17, 2025 20:28

ray-gardener bot added the serve Ray Serve Related Issue label Oct 18, 2025

kouroshHakha requested a review from ruisearch42 October 18, 2025 03:06

kouroshHakha reviewed Oct 18, 2025

View reviewed changes

python/ray/llm/_internal/common/utils/cloud_utils.py Outdated Show resolved Hide resolved

python/ray/llm/_internal/common/utils/cloud_utils.py Show resolved Hide resolved

python/ray/llm/tests/serve/cpu/configs/test_models.py Outdated Show resolved Hide resolved

mock tests change

3327ddc

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

ruisearch42 reviewed Oct 19, 2025

View reviewed changes

add tests, change comments, refactor filtering function, misc suggest…

782260e

…ions Signed-off-by: ahao-anyscale <ahao@anyscale.com>

This comment was marked as outdated.

Sign in to view

Merge remote-tracking branch 'origin/master' into startup-optimizations

a234314

This comment was marked as outdated.

Sign in to view

bug fix

28208fc

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

This comment was marked as outdated.

Sign in to view

bugfix

fc8b4a3

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

ruisearch42 reviewed Oct 23, 2025

View reviewed changes

kouroshHakha mentioned this pull request Oct 23, 2025

[serve][llm] Model files still being downloaded with runai_streamer mode #58024

Closed

jiangwu300 mentioned this pull request Oct 23, 2025

[llm][serve] Fixing serve using runai_streamer #58051

Closed

cleanliness fixes

77bd80d

Signed-off-by: ahao-anyscale <ahao@anyscale.com>

ruisearch42 approved these changes Oct 28, 2025

View reviewed changes

python/ray/llm/_internal/common/callbacks/cloud_downloader.py Show resolved Hide resolved

angelinalg approved these changes Oct 28, 2025

View reviewed changes

kouroshHakha approved these changes Oct 28, 2025

View reviewed changes

kouroshHakha added the go add ONLY when ready to merge, run all tests label Oct 28, 2025

kouroshHakha enabled auto-merge (squash) October 28, 2025 21:51

kouroshHakha merged commit 29ba2ab into ray-project:master Oct 28, 2025
7 of 8 checks passed

YoussefEssDS pushed a commit to YoussefEssDS/ray that referenced this pull request Nov 8, 2025

[serve][llm] Disable model downloading for RunAI streamer, introduce …

7e48795

…optimized download function (ray-project#57854) Signed-off-by: ahao-anyscale <ahao@anyscale.com>

hao-aaron mentioned this pull request Nov 8, 2025

[serve][llm] Ray LLM Cloud Filesystem Restructuring: Provider-Specific Implementations #58469

Merged

landscapepainter pushed a commit to landscapepainter/ray that referenced this pull request Nov 17, 2025

[serve][llm] Disable model downloading for RunAI streamer, introduce …

35733d8

…optimized download function (ray-project#57854) Signed-off-by: ahao-anyscale <ahao@anyscale.com>

Conversation

hao-aaron commented Oct 17, 2025

Description

Uh oh!

This comment was marked as outdated.

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ruisearch42 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

hao-aaron commented Oct 21, 2025

Uh oh!

ruisearch42 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ruisearch42 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

angelinalg left a comment

Choose a reason for hiding this comment

Uh oh!

kouroshHakha commented Oct 28, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants