Also we think there are two bugs in the scheduler resource reporting logic:

1. The autoscaler expects that the resource_demands is unique per resource shape. however the way scheduler resource reporting implemented didn't dedupe the resource shapes between tasks_to_schedule_ and tasks_to_dispatch_.

2. We report client side pending requests from worker to local raylet (https://github.com/ray-project/ray/blob/master/src/ray/raylet/scheduling/cluster_task_manager.cc#L843); however in the case that the lease request is sent to a remote raylet (spilling, or locality aware scheduling), this might under count the resource requirement.

cc @wuisawesome @jjyao

Just want to make sure I understand it correctly, after this one,
src/ray/raylet/scheduling/scheduler_resource_reporter.h basically will be something like the Reporter concept we talked about before, right?

We report client side pending requests from worker to local raylet (https://github.com/ray-project/ray/blob/master/src/ray/raylet/scheduling/cluster_task_manager.cc#L843); however in the case that the lease request is sent to a remote raylet (spilling, or locality aware scheduling), this might under count the resource requirement.

We only report tasks without pending lease requests to local raylet. For tasks with pending lease requests, they will be reported by the raylets that receive the lease requests. Combined together, gcs should see an almost accurate resource requirement.

The autoscaler expects that the resource_demands is unique per resource shape. however the way scheduler resource reporting implemented didn't dedupe the resource shapes between tasks_to_schedule_ and tasks_to_dispatch_.

Let's remove that comment. The code doesn't seem to make that assumption. https://sourcegraph.com/github.com/ray-project/ray/-/blob/python/ray/autoscaler/_private/monitor.py?L85

We report client side pending requests from worker to local raylet (https://github.com/ray-project/ray/blob/master/src/ray/raylet/scheduling/cluster_task_manager.cc#L843); however in the case that the lease request is sent to a remote raylet (spilling, or locality aware scheduling), this might under count the resource requirement.

I think that line reference changed? But if you're referring to the worker backlog reporting, I think the leawse request will get counted, but at the other raylet? (there is a short race condition when it's in transit and not counted i guess).