[Autoscaler] Add bundle_label_selector to request_resources sdk by ryanaoleary · Pull Request #54843 · ray-project/ray

ryanaoleary · 2025-07-22T22:30:17Z

Why are these changes needed?

This change adds a bundle_label_selector argument to the request_resources sdk command for the v2 Ray autoscaler. This command is used by several Ray libraries. The bundle_label_selector is a parallel list to the bundles of resource shapes specified by the user and are applied per-bundle. These label selectors are passed to the repeated LabelSelector label_selectors field in the ResourceRequest message that gets built by RequestClusterResourceConstraint.

This change depends on #53578.

Related issue number

Contributes to #51564

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Note

Adds per-bundle label selectors to resource requests and wires them through Python SDK, autoscaler v2, Cython GCS client, and GCS RPC, with tests.

Autoscaler SDK/API:
- python/ray/autoscaler/sdk/sdk.py: Extend request_resources with bundle_label_selectors; validate types/length and label syntax; update docs/examples.
- python/ray/autoscaler/_private/commands.py: Emit requests as {"resources": ..., "label_selector": ...}; support selectors per bundle; forward to v2 SDK.
- python/ray/autoscaler/v2/sdk.py: Accept per-bundle label_selector, normalize legacy format, aggregate by (resources, selector), and forward selectors to GCS.
GCS client & RPC plumbing:
- python/ray/includes/common.pxd, python/ray/includes/gcs_client.pxi: Add label_selectors param to request_cluster_resource_constraint.
- src/ray/gcs_rpc_client/accessor.{h,cc}: Method signature updated to take selectors; build proto LabelSelector per bundle.
Label selector utils:
- src/ray/common/scheduling/label_selector.{h,cc}: Add generic ctor from map and keep ToProto; hashing/equality unchanged.
Tests:
- python/ray/autoscaler/v2/tests/test_sdk.py: New test verifying per-bundle label selectors and operators; imports LabelSelectorOperator.
- python/ray/tests/test_multi_node_2.py: Adjust expected resource_requests format to include resources and empty label_selector.

^{Written by Cursor Bugbot for commit 4409f1d. This will update automatically on new commits. Configure here.}

gemini-code-assist · 2025-07-22T22:30:21Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

ryanaoleary · 2025-07-22T22:30:27Z

cc: @MengjinYan

github-actions · 2025-08-07T12:25:41Z

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

edoakes · 2025-08-07T21:58:05Z

Tons of merge conflicts @ryanaoleary

ryanaoleary · 2025-08-11T15:13:28Z

Rebased and re-tested, should all be fixed now cc: @edoakes @MengjinYan

edoakes · 2025-08-15T21:50:04Z

ping @MengjinYan

python/ray/autoscaler/sdk/sdk.py

python/ray/autoscaler/_private/commands.py

python/ray/autoscaler/v2/sdk.py

jjyao · 2025-09-16T23:48:28Z

You need to rebase now.

python/ray/autoscaler/sdk/sdk.py

python/ray/autoscaler/v2/sdk.py

python/ray/autoscaler/sdk/sdk.py

python/ray/autoscaler/v2/sdk.py

ryanaoleary · 2025-09-24T09:25:04Z

cc: @MengjinYan fixed merge conflicts and other comments

python/ray/autoscaler/_private/commands.py

python/ray/autoscaler/v2/sdk.py

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

python/ray/autoscaler/v2/sdk.py

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

python/ray/autoscaler/v2/sdk.py

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

python/ray/autoscaler/v2/sdk.py

jjyao

LG

python/ray/autoscaler/v2/sdk.py

src/ray/gcs_rpc_client/accessor.cc

src/ray/gcs_rpc_client/accessor.h

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

python/ray/autoscaler/v2/sdk.py

python/ray/autoscaler/_private/commands.py

python/ray/autoscaler/v2/sdk.py

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

cursor

Bug: Autoscaler v2 Missing Label Selectors

When autoscaler v2 is enabled, the request_resources function calls request_cluster_resources without passing the bundle_label_selectors. This means the v2 autoscaler won't receive label selector information, potentially causing a TypeError or incorrect resource provisioning.

python/ray/autoscaler/_private/commands.py#L234-L236

ray/python/ray/autoscaler/_private/commands.py

Lines 234 to 236 in 11ef4ac

    
           gcs_address = internal_kv_get_gcs_client().address 
        
           request_cluster_resources(gcs_address, to_request)

ryanaoleary · 2025-10-07T10:47:23Z

Bug: Autoscaler v2 Missing Label Selectors

When autoscaler v2 is enabled, the request_resources function calls request_cluster_resources without passing the bundle_label_selectors. This means the v2 autoscaler won't receive label selector information, potentially causing a TypeError or incorrect resource provisioning.

python/ray/autoscaler/_private/commands.py#L234-L236

ray/python/ray/autoscaler/_private/commands.py

Lines 234 to 236 in 11ef4ac

gcs_address = internal_kv_get_gcs_client().address

request_cluster_resources(gcs_address, to_request)

Invalid comment, the label selectors are part of to_request and the tests are with the v2 autoscaler

MengjinYan · 2025-10-15T07:46:13Z

@jjyao Ping to merge

jjyao · 2025-10-15T15:59:40Z

python/ray/autoscaler/v2/sdk.py

-            Each bundle is a dict of resource name to resource quantity, e.g:
-            [{"CPU": 1}, {"GPU": 1}].
+        to_request: A list of resource requests to request the cluster to have.
+            Each resource request is a tuple of resources and a label_selector


Each resource request is a tuple

It's a dict not tuple?

I think it was previously a dict, but we changed it to be a tuple of two dicts (resources and labels) here:

ray/python/ray/autoscaler/v2/sdk.py

Line 17 in c6b8c9f

class ResourceRequest(NamedTuple):

.

…project#54843) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com> Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>

…project#54843) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com> Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com> Signed-off-by: xgui <xgui@anyscale.com>

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com> Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…project#54843) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com> Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>

…project#54843) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com> Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…project#54843) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com> Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

ryanaoleary requested a review from a team as a code owner July 22, 2025 22:30

ray-gardener bot added community-contribution Contributed by the community docs An issue or change related to documentation core Issues that should be addressed in Ray Core labels Jul 23, 2025

github-actions bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Aug 7, 2025

ryanaoleary mentioned this pull request Aug 7, 2025

[Core] Ray Label Selector API Implementation Tracker #51564

Open

36 tasks

github-actions bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Aug 8, 2025

ryanaoleary force-pushed the request-resources-labels branch 3 times, most recently from 83d8bcc to a0199b0 Compare August 11, 2025 14:36

MengjinYan self-assigned this Sep 9, 2025

MengjinYan reviewed Sep 10, 2025

View reviewed changes

jjyao requested a review from MengjinYan September 16, 2025 23:48

ryanaoleary force-pushed the request-resources-labels branch from 912f181 to a88a6a2 Compare September 18, 2025 20:21

MengjinYan reviewed Sep 23, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

ryanaoleary force-pushed the request-resources-labels branch from 7d0e84f to 933c0a0 Compare September 24, 2025 08:48

ryanaoleary requested a review from MengjinYan September 24, 2025 09:24

This comment was marked as outdated.

Sign in to view

Merge branch 'master' into request-resources-labels

26dc7f9

jjyao reviewed Sep 29, 2025

View reviewed changes

python/ray/autoscaler/_private/commands.py Show resolved Hide resolved

python/ray/autoscaler/v2/sdk.py Outdated Show resolved Hide resolved

Add assert for length

4409f1d

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

ryanaoleary requested a review from jjyao September 29, 2025 22:41

Remove legacy format

7cb35ec

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

cursor bot reviewed Oct 3, 2025

View reviewed changes

python/ray/autoscaler/v2/sdk.py Outdated Show resolved Hide resolved

ryanaoleary and others added 2 commits October 3, 2025 04:29

Merge branch 'master' into request-resources-labels

7930dcf

Fix test_autoscaler

668fd8d

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

cursor bot reviewed Oct 3, 2025

View reviewed changes

python/ray/autoscaler/v2/sdk.py Outdated Show resolved Hide resolved

Fix comment

b0bdf9e

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

cursor bot reviewed Oct 4, 2025

View reviewed changes

python/ray/autoscaler/v2/sdk.py Show resolved Hide resolved

jjyao reviewed Oct 7, 2025

View reviewed changes

python/ray/autoscaler/v2/sdk.py Outdated Show resolved Hide resolved

python/ray/autoscaler/v2/sdk.py Outdated Show resolved Hide resolved

src/ray/gcs_rpc_client/accessor.cc Show resolved Hide resolved

src/ray/gcs_rpc_client/accessor.h Outdated Show resolved Hide resolved

Update python/ray/autoscaler/v2/sdk.py

93f4409

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

cursor bot reviewed Oct 7, 2025

View reviewed changes

python/ray/autoscaler/v2/sdk.py Show resolved Hide resolved

python/ray/autoscaler/_private/commands.py Outdated Show resolved Hide resolved

python/ray/autoscaler/v2/sdk.py Outdated Show resolved Hide resolved

Fix comments

11ef4ac

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

cursor bot reviewed Oct 7, 2025

View reviewed changes

ryanaoleary requested a review from jjyao October 7, 2025 10:47

jjyao and others added 2 commits October 14, 2025 16:33

Merge branch 'master' into request-resources-labels

4aae6f5

Merge branch 'master' into request-resources-labels

06a88b6

jjyao reviewed Oct 15, 2025

View reviewed changes

jjyao merged commit 4036252 into ray-project:master Oct 15, 2025
6 checks passed

dragongu mentioned this pull request Jan 5, 2026

[Autoscaler] Fix v1 autoscaler TypeError when using bundle_label_selectors #59850

Merged


	gcs_address = internal_kv_get_gcs_client().address
	request_cluster_resources(gcs_address, to_request)

Conversation

ryanaoleary commented Jul 22, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

gemini-code-assist bot commented Jul 22, 2025

Uh oh!

ryanaoleary commented Jul 22, 2025

Uh oh!

github-actions bot commented Aug 7, 2025

Uh oh!

edoakes commented Aug 7, 2025

Uh oh!

ryanaoleary commented Aug 11, 2025

Uh oh!

edoakes commented Aug 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jjyao commented Sep 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

ryanaoleary commented Sep 24, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jjyao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Bug: Autoscaler v2 Missing Label Selectors

Uh oh!

ryanaoleary commented Oct 7, 2025

Bug: Autoscaler v2 Missing Label Selectors

Uh oh!

MengjinYan commented Oct 15, 2025

Uh oh!

jjyao Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

ryanaoleary Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

ryanaoleary commented Jul 22, 2025 •

edited by cursor bot

Loading