Will the --worker-image passed in https://github.com/dask/dask-kubernetes/blob/master/.circleci/config.yml#L73 need to be dask / distributed master as well (or built locally as part of the CI)?

OK, I think 25543e5 is testing dask-kubernetes against dask & distributed master. https://circleci.com/gh/dask/dask-kubernetes/356?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link The changes are

1. Install dask & distributed master in our test Conda environment running things
2. Build a docker image with dask & distributed master, for passing as the --worker-image.

Things are failing as expected. I'll start picking them off.

Just a note: circle-CI caching the conda env may cause some headaches in the future if we intend to test against distributed master going forward. The previous run was restoring the conda env from a couple days ago, and building the worker-image from master today.

Long term, we probably want multiple jobs. One against a minimum distributed, and another against dask-master. The one against the minimum distributed could be cached safely.

Yay, we're passing :)

So the changes were

1. Adding a KubeCluster.workers property
2. Changing when KubeCluster.__init__ creates the LocalCluster, to satisfy stricter checks in the test (ensuring that all the instances of SpecCluster are gone)
3. bokeh -> dashboard

Now, what versions of distributed do we want this repo to support? I'd say pin the minimum to the next release of distributed and sync a release of dask-kubernetes with the next version of distributed.

I'd say pin the minimum to the next release of distributed and sync a release of dask-kubernetes with the next version of distributed.

+1

OK, I think I'll merge this as is then. In prep for the distributed release, we'll want to bump our required minimum version here.

If we want to be especially responsible, we can backport a commit pinning the maximum version of distributed to distributed<2, and back a release, but that may be more effort than necessary.

@TomAugspurger are we ok to release this after dask/dask and dask/distributed are released?