Skip to content

[Data] Revising resource allocator task scheduling decision to factor in pending task outputs#60639

Merged
alexeykudinkin merged 8 commits intomasterfrom
ak/res-allc-fup-1
Feb 3, 2026
Merged

[Data] Revising resource allocator task scheduling decision to factor in pending task outputs#60639
alexeykudinkin merged 8 commits intomasterfrom
ak/res-allc-fup-1

Conversation

@alexeykudinkin
Copy link
Copy Markdown
Contributor

@alexeykudinkin alexeykudinkin commented Jan 31, 2026 •
edited
Loading

Description

This change reverts back to behavior where

  1. can_submit_new_task still evaluates whether to schedule task based on estimated task outputs pending in the buffer.
  2. Estimate of the pending task outputs to rely on target_max_block_size until actual estimate becomes available

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

@alexeykudinkin alexeykudinkin requested a review from a team as a code owner January 31, 2026 19:31
gemini-code-assist[bot]
gemini-code-assist bot reviewed Jan 31, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request revises the resource allocator's task scheduling decision to be more robust, especially concerning object store memory. It introduces a fallback mechanism to estimate the size of pending task outputs when no historical data is available, using target_max_block_size. This estimate is then used to check if there is sufficient object store memory budget before submitting a new task.

The overall logic is sound and improves scheduling decisions. However, I found a critical issue in resource_manager.py where self._metrics is used instead of op.metrics, which will lead to an AttributeError.

@alexeykudinkin alexeykudinkin added the go add ONLY when ready to merge, run all tests label Jan 31, 2026
cursor[bot]
cursor bot reviewed Jan 31, 2026
View reviewed changes
cursor[bot]
cursor bot reviewed Jan 31, 2026
View reviewed changes
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

@ray-gardener ray-gardener bot added the data Ray Data-related issues label Feb 1, 2026
@alexeykudinkin alexeykudinkin changed the title [WIP][Data] Revising resource allocator task scheduling decision [Data] Revising resource allocator task scheduling decision to factor in pending task outputs Feb 2, 2026
@alexeykudinkin alexeykudinkin enabled auto-merge (squash) February 3, 2026 00:56
@alexeykudinkin
Reverting obj_store_mem_max_pending_output_per_task to use `target_…
5af63ea 
…max_block_size` until estimate becomes available

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Updated can_submit_task to check whether there's enough object stor…
a5c270b 
…e budget to hold pending task outputs

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Fixing invalid ref
2714458 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Fixed tests
2e0e16b 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
lint
4b89bcf 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@github-actions github-actions bot disabled auto-merge February 3, 2026 01:57
@alexeykudinkin
Updated test
0de4df8 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
lint
38b2393 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin alexeykudinkin enabled auto-merge (squash) February 3, 2026 07:55
@alexeykudinkin
Fixed conditional
dd26b34 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@github-actions github-actions bot disabled auto-merge February 3, 2026 07:59
@alexeykudinkin alexeykudinkin merged commit 1315d95 into master Feb 3, 2026
6 checks passed
@alexeykudinkin alexeykudinkin deleted the ak/res-allc-fup-1 branch February 3, 2026 17:24
rayhhome pushed a commit to rayhhome/ray that referenced this pull request Feb 4, 2026
@alexeykudinkin @rayhhome
[Data] Revising resource allocator task scheduling decision to factor…
47681c8 
… in pending task outputs (ray-project#60639)

## Description

This change reverts back to behavior where

1. `can_submit_new_task` still evaluates whether to schedule task based
on estimated task outputs pending in the buffer.
2. Estimate of the pending task outputs to rely on
`target_max_block_size` until actual estimate becomes available

## Related issues
> Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to
ray-project#1234".

## Additional information
> Optional: Add implementation details, API changes, usage examples,
screenshots, etc.

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: Sirui Huang <ray.huang@anyscale.com>
Sparks0219 pushed a commit to Sparks0219/ray that referenced this pull request Feb 9, 2026
@alexeykudinkin
[Data] Revising resource allocator task scheduling decision to factor…
2852512 
… in pending task outputs (ray-project#60639)

## Description

This change reverts back to behavior where 

1. `can_submit_new_task` still evaluates whether to schedule task based
on estimated task outputs pending in the buffer.
2. Estimate of the pending task outputs to rely on
`target_max_block_size` until actual estimate becomes available

## Related issues
> Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to
ray-project#1234".

## Additional information
> Optional: Add implementation details, API changes, usage examples,
screenshots, etc.

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
elliot-barn pushed a commit that referenced this pull request Feb 9, 2026
@alexeykudinkin @elliot-barn
[Data] Revising resource allocator task scheduling decision to factor…
4699d03 
… in pending task outputs (#60639)

## Description

This change reverts back to behavior where 

1. `can_submit_new_task` still evaluates whether to schedule task based
on estimated task outputs pending in the buffer.
2. Estimate of the pending task outputs to rely on
`target_max_block_size` until actual estimate becomes available

## Related issues
> Link related issues: "Fixes #1234", "Closes #1234", or "Related to
#1234".

## Additional information
> Optional: Add implementation details, API changes, usage examples,
screenshots, etc.

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
elliot-barn pushed a commit that referenced this pull request Feb 9, 2026
@alexeykudinkin @elliot-barn
[Data] Revising resource allocator task scheduling decision to factor…
2bdcd19 
… in pending task outputs (#60639)

## Description

This change reverts back to behavior where 

1. `can_submit_new_task` still evaluates whether to schedule task based
on estimated task outputs pending in the buffer.
2. Estimate of the pending task outputs to rely on
`target_max_block_size` until actual estimate becomes available

## Related issues
> Link related issues: "Fixes #1234", "Closes #1234", or "Related to
#1234".

## Additional information
> Optional: Add implementation details, API changes, usage examples,
screenshots, etc.

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
ans9868 pushed a commit to ans9868/ray that referenced this pull request Feb 18, 2026
@alexeykudinkin @ans9868
[Data] Revising resource allocator task scheduling decision to factor…
d4cb367 
… in pending task outputs (ray-project#60639)

## Description

This change reverts back to behavior where

1. `can_submit_new_task` still evaluates whether to schedule task based
on estimated task outputs pending in the buffer.
2. Estimate of the pending task outputs to rely on
`target_max_block_size` until actual estimate becomes available

## Related issues
> Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to
ray-project#1234".

## Additional information
> Optional: Add implementation details, API changes, usage examples,
screenshots, etc.

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: Adel Nour <ans9868@nyu.edu>
peterxcli pushed a commit to peterxcli/ray that referenced this pull request Feb 25, 2026
@alexeykudinkin @peterxcli
[Data] Revising resource allocator task scheduling decision to factor…
0d0d8e0 
… in pending task outputs (ray-project#60639)

## Description

This change reverts back to behavior where

1. `can_submit_new_task` still evaluates whether to schedule task based
on estimated task outputs pending in the buffer.
2. Estimate of the pending task outputs to rely on
`target_max_block_size` until actual estimate becomes available

## Related issues
> Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to
ray-project#1234".

## Additional information
> Optional: Add implementation details, API changes, usage examples,
screenshots, etc.

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: peterxcli <peterxcli@gmail.com>
peterxcli pushed a commit to peterxcli/ray that referenced this pull request Feb 25, 2026
@alexeykudinkin @peterxcli
[Data] Revising resource allocator task scheduling decision to factor…
08ee2db 
… in pending task outputs (ray-project#60639)

## Description

This change reverts back to behavior where

1. `can_submit_new_task` still evaluates whether to schedule task based
on estimated task outputs pending in the buffer.
2. Estimate of the pending task outputs to rely on
`target_max_block_size` until actual estimate becomes available

## Related issues
> Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to
ray-project#1234".

## Additional information
> Optional: Add implementation details, API changes, usage examples,
screenshots, etc.

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: peterxcli <peterxcli@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cursor cursor[bot] cursor[bot] left review comments

@bveeramani bveeramani bveeramani approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Ray fails to serialize self-reference objects

2 participants

@alexeykudinkin @bveeramani