To simplify debugging, we should always run monitor.py on all Ray clusters (perhaps with some dummy autoscaling config). This will be critical if we start reporting metrics using the autoscaler as well.

cc @wuisawesome @AmeerHajAli