Skip to content

[Data] AutoscalingCoordinator double-allocates resources if there are multiple datasets #59685

Closed
#59740
Closed
[Data] AutoscalingCoordinator double-allocates resources if there are multiple datasets#59685
#59740
Assignees
machichima
Labels
P2Important issue, but not time-criticaldataRay Data-related issues
@bveeramani

Description

@bveeramani
bveeramani
opened on Dec 26, 2025

ray/python/ray/data/_internal/cluster_autoscaler/default_autoscaling_coordinator.py

Lines 401 to 411 in 6dc66d4

# Allocate remaining resources.
# NOTE, to handle the case where multiple datasets are running concurrently,
# now we double-allocate remaining resources to all requesters with
# `request_remaining=True`.
# This achieves parity with the behavior before Ray Data was integrated with
# AutoscalingCoordinator, where each dataset assumes it has the whole cluster.
# TODO(hchen): handle multiple request_remaining requests better.
for ongoing_req in ongoing_reqs:
if ongoing_req.request_remaining:
ongoing_req.allocated_resources.extend(cluster_node_resources)

TODO: Scope this fix

Metadata

Metadata

Assignees

Labels

P2Important issue, but not time-criticaldataRay Data-related issues

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions