Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/500/
Test PASSed.

@ls-daniel Can you verify if this fixes the autoscaler? Or would it be possible to share the autoscaler config that was failing?

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13834/
Test FAILed.

Is this PR also able to fix #4655?

This doesn't work.

If we add print("static_resources", static_resources) to that function, and start running some tasks, the logs look like this:

2019-04-18 13:55:00,805 DEBUG static_resources {b'CPU': 1.0}
2019-04-18 13:55:00,921 DEBUG static_resources {b'CPU': 1.0}
2019-04-18 13:55:01,030 DEBUG static_resources {}
2019-04-18 13:55:01,142 DEBUG static_resources {}
2019-04-18 13:55:01,253 DEBUG static_resources {}

that is, the static resources disappear, as I mentioned earlier. So you can't use the static_resources to imply a dynamic resource should be zero. The static resources should not change, a node with 1 CPU always has 1 CPU regardless of what's running on it.

Start ray with --num-cpus=1, min_workers=0. Other configuration parameters shouldn't matter.

You'll never scale the first worker, because the static resources disappear as soon as you do anything on the head node.

If I revert the other commit it works fine. I'm fairly sure it's not an issue in the autoscaler. I've had a look at the commit to try and figure out which bit of the code is adjusting static resources.

It makes sense to me that you may wish for a dynamic resource to disappear instead of being zero. A static resource should a) never be zero and b) never disappear, it's static :)

Thanks @ls-daniel. I agree, static resources should indeed never be zero or disappear.

The LoadMetrics.update method is called from monitor.py, which updates it with values from the raylet heartbeats:

This behavior is a bit unexpected since the resources_total_ in cluster_resource_map is never modified once created from the GCS entry, and thus ResourcesTotalCapacity in the heartbeat should never change. cc @robertnishihara

Is this PR also able to fix #4655?

Yes, I think this is related to #4655 and should have fixed it.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/537/
Test PASSed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13886/
Test FAILed.

This change solved #4655 for me.

I've tried with @Rnishihara's change and that doesn't work either. Same behaviour.

The autoscaler on Ray master is broken for me, further changes to the resource mechanisms have meant reverting 4555 isn't easily possible any more either, so I'm stuck about 30 commits back now

@ls-daniel thanks, @williamma12 is looking into it.

@ls-daniel I was able to reproduce the error and pushed a fix that fixed it for me, can you let me know if it fixes it for you?

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/678/
Test PASSed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14082/
Test FAILed.

@williamma12 this works perfectly cherry-picked into my branch - Thank you :)

I'm now rebasing on master to test again and will report back.

Thanks @williamma12! @ls-daniel The autoscaler is unfortunately not as well tested because it interacts with AWS or GCP. Do you have thoughts about how to improve the testing setup?