Describe the problem
Autoscaler launch currently doesn't work on workers if the following command is run:
most_recent() { echo pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/$(aws s3 ls s3://ray-wheels --recursive | grep $1 | sort -r | head -n 1 | awk '{print $4}'); } && $( most_recent "cp36-cp36m-manylinux1" ) || $( most_recent "cp35-cp35m-manylinux1" )
Plausible solutions:
- Add a
latest folder to the S3 wheels folder so that aws cli isn't needed on workers
- Add more permissions to the head node so that worker nodes are endowed with permissions to read from S3