Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/422/
Test PASSed.

cc @ericl who is more familiar with this code

I was hoping to get around this by saying ray start --head --num-gpus=0.01, but that gives me Error: Invalid value for "--num-gpus": 0.01 is not a valid integer.

Another workaround could be to say ray start --head --num-gpus=1 --num-cpus=1 and have trials take > 1 cpu. This leads to the following tune resource allocation:

== Status ==
Using FIFO scheduling algorithm.
Resources requested: 2/1 CPUs, 0.5/1 GPUs
Memory usage on this node: 1.0/15.8 GB

Ray does not allocate anything though and the autoscaler thinks that there's no resource usage:

2019-04-11 01:01:05,030 INFO autoscaler.py:625 -- StandardAutoscaler: 0/0 target nodes (0 pending)

which is why the cluster doesn't scale at all.

That said, I don't actually know if this PR would fix the issue.

This won't actually trigger scaling up, unless you set min worker nodes =1. This is a limitation of the scheduler right now.

Specifying initial_workers > 1 together with ray start --num-gpus=1 --num-cpus=1 ... allows a trial to be started on a worker, but things still hang.

@ericl Do you know any workaround for running cluster with non-gpu head node and gpu workers?

Edit: Nevermind, the worker starts running fine with this setup after all. For some reason the training iteration took almost 15 minutes which got me confused.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13712/
Test FAILed.

I think we can close this for now as it seems like it's not a right fix?