Skip to content

[Data] Fix autoscaler to request previous allocation instead of empty resources when not scaling up#60321

Merged
bveeramani merged 9 commits intoray-project:masterfrom
machichima:autoscaler-request-previous-allocation
Jan 23, 2026
Merged

[Data] Fix autoscaler to request previous allocation instead of empty resources when not scaling up#60321
bveeramani merged 9 commits intoray-project:masterfrom
machichima:autoscaler-request-previous-allocation

Conversation

@machichima
Copy link
Copy Markdown
Contributor

@machichima machichima commented Jan 20, 2026 •
edited
Loading

Description

When the DefaultAutoscalerV2 decides not to scale up (e.g., utilization is low), it currently sends an empty resource request ([]) to the autoscaling coordinator. This should be changed to request the previous allocation instead, preserving the dataset's current resource footprint.

Related issues

Closes #60191

Additional information

  • Send current resources rather than [] when not scale up
  • Add test

@machichima
feat: send curr resources rather then []
bbd54f2 
Signed-off-by: machichima <nary12321@gmail.com>
@machichima
test: test getting prev resource for low utilization
589ba0c 
Signed-off-by: machichima <nary12321@gmail.com>
@bveeramani bveeramani marked this pull request as ready for review January 20, 2026 23:03
@bveeramani bveeramani requested a review from a team as a code owner January 20, 2026 23:03
@bveeramani bveeramani enabled auto-merge (squash) January 20, 2026 23:03
@github-actions github-actions bot added the go add ONLY when ready to merge, run all tests label Jan 20, 2026
bveeramani
bveeramani approved these changes Jan 20, 2026
View reviewed changes
python/ray/data/tests/test_default_cluster_autoscaler_v2.py Outdated
)

# Initialization should send []
_send_resource_request.assert_called_with([])
Copy link
Copy Markdown
Member

@bveeramani bveeramani Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a general practice, I think we should avoid testing against implementation details, but I think it's okay in this case if there's not a clear alternative

Copy link
Copy Markdown
Contributor Author

@machichima machichima Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I refactored the test to prevent mocking the _send_resource_request

bb4d311

@github-actions github-actions bot disabled auto-merge January 21, 2026 00:30
@ray-gardener ray-gardener bot added data Ray Data-related issues community-contribution Contributed by the community labels Jan 21, 2026
cursor[bot]
cursor bot reviewed Jan 21, 2026
View reviewed changes
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

bveeramani and others added 5 commits January 20, 2026 23:48
@bveeramani
Merge branch 'master' into autoscaler-request-previous-allocation
ceeab6c
@machichima
test: avoid testing against implementation details
bb4d311 
Signed-off-by: machichima <nary12321@gmail.com>
@machichima
fix: add missing resource_limits
d97d57e 
Signed-off-by: machichima <nary12321@gmail.com>
@bveeramani
Refactor test
7ac3ec5 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Simplify test
bc98633 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani bveeramani enabled auto-merge (squash) January 23, 2026 21:26
@bveeramani bveeramani merged commit 8e120db into ray-project:master Jan 23, 2026
7 checks passed
xinyuangui2 pushed a commit to xinyuangui2/ray that referenced this pull request Jan 26, 2026
@machichima @bveeramani @xinyuangui2
[Data] Fix autoscaler to request previous allocation instead of empty…
8ae18f7 
… resources when not scaling up (ray-project#60321)

## Description

When the DefaultAutoscalerV2 decides not to scale up (e.g., utilization
is low), it currently sends an empty resource request ([]) to the
autoscaling coordinator. This should be changed to request the previous
allocation instead, preserving the dataset's current resource footprint.

## Related issues
Closes ray-project#60191

## Additional information
- Send current resources rather than [] when not scale up
- Add test

---------

Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu>
jinbum-kim pushed a commit to jinbum-kim/ray that referenced this pull request Jan 29, 2026
@machichima @bveeramani @jinbum-kim
[Data] Fix autoscaler to request previous allocation instead of empty…
a470296 
… resources when not scaling up (ray-project#60321)

## Description

When the DefaultAutoscalerV2 decides not to scale up (e.g., utilization
is low), it currently sends an empty resource request ([]) to the
autoscaling coordinator. This should be changed to request the previous
allocation instead, preserving the dataset's current resource footprint.

## Related issues
Closes ray-project#60191

## Additional information
- Send current resources rather than [] when not scale up
- Add test

---------

Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: jinbum-kim <jinbum9958@gmail.com>
zzchun pushed a commit to antgroup/ant-ray that referenced this pull request Jan 29, 2026
@machichima @bveeramani @zzchun
[Data] Fix autoscaler to request previous allocation instead of empty…
1885a84 
… resources when not scaling up (ray-project#60321)

## Description

When the DefaultAutoscalerV2 decides not to scale up (e.g., utilization
is low), it currently sends an empty resource request ([]) to the
autoscaling coordinator. This should be changed to request the previous
allocation instead, preserving the dataset's current resource footprint.

## Related issues
Closes ray-project#60191

## Additional information
- Send current resources rather than [] when not scale up
- Add test

---------

Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu>
400Ping pushed a commit to 400Ping/ray that referenced this pull request Feb 1, 2026
@machichima @bveeramani @400Ping
[Data] Fix autoscaler to request previous allocation instead of empty…
4732eff 
… resources when not scaling up (ray-project#60321)

## Description

When the DefaultAutoscalerV2 decides not to scale up (e.g., utilization
is low), it currently sends an empty resource request ([]) to the
autoscaling coordinator. This should be changed to request the previous
allocation instead, preserving the dataset's current resource footprint.

## Related issues
Closes ray-project#60191

## Additional information
- Send current resources rather than [] when not scale up
- Add test

---------

Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: 400Ping <jiekaichang@apache.org>
zzchun pushed a commit to antgroup/ant-ray that referenced this pull request Feb 5, 2026
@machichima @bveeramani @zzchun
[Data] Fix autoscaler to request previous allocation instead of empty…
6825b94 
… resources when not scaling up (ray-project#60321)

## Description

When the DefaultAutoscalerV2 decides not to scale up (e.g., utilization
is low), it currently sends an empty resource request ([]) to the
autoscaling coordinator. This should be changed to request the previous
allocation instead, preserving the dataset's current resource footprint.

## Related issues
Closes ray-project#60191

## Additional information
- Send current resources rather than [] when not scale up
- Add test

---------

Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu>
peterxcli pushed a commit to peterxcli/ray that referenced this pull request Feb 25, 2026
@machichima @bveeramani @peterxcli
[Data] Fix autoscaler to request previous allocation instead of empty…
1f79a37 
… resources when not scaling up (ray-project#60321)

## Description

When the DefaultAutoscalerV2 decides not to scale up (e.g., utilization
is low), it currently sends an empty resource request ([]) to the
autoscaling coordinator. This should be changed to request the previous
allocation instead, preserving the dataset's current resource footprint.

## Related issues
Closes ray-project#60191

## Additional information
- Send current resources rather than [] when not scale up
- Add test

---------

Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: peterxcli <peterxcli@gmail.com>
peterxcli pushed a commit to peterxcli/ray that referenced this pull request Feb 25, 2026
@machichima @bveeramani @peterxcli
[Data] Fix autoscaler to request previous allocation instead of empty…
7ec7e26 
… resources when not scaling up (ray-project#60321)

## Description

When the DefaultAutoscalerV2 decides not to scale up (e.g., utilization
is low), it currently sends an empty resource request ([]) to the
autoscaling coordinator. This should be changed to request the previous
allocation instead, preserving the dataset's current resource footprint.

## Related issues
Closes ray-project#60191

## Additional information
- Send current resources rather than [] when not scale up
- Add test

---------

Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: peterxcli <peterxcli@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cursor cursor[bot] cursor[bot] left review comments

@bveeramani bveeramani bveeramani approved these changes

Assignees

No one assigned

Labels

community-contribution Contributed by the community data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Data] Fix autoscaler to request previous allocation instead of empty resources when not scaling up

2 participants

@machichima @bveeramani