Add basic JupyterHub + Dask deployment by mrocklin · Pull Request #57 · pangeo-data/pangeo

mrocklin · 2017-12-26T22:09:28Z

This is a vanilla JupyterHub deployment alongside a vanilla Dask deployment . It includes instructions and helm config files to set things up.

jhamman · 2017-12-27T03:21:51Z

+    name: daskdev/dask-notebook
+    tag: latest
+  extraEnv:
+    EXTRA_PIP_PACKAGES: gcsfs xarray


How hard would it be to put xarray-master in here instead? Also, if we can the optional xarray dependencies (scipy/netcdf4/zarr), that will allow us to start testing a few workflows.

We can anything that is pip or conda installable. Are there particular branches that you'd like us to target?

I would also be quite happy to walk you through how to manipulate the cluster if you're interested. Changing this stuff is surprisingly easy.

Conda installing scipy/netcdf4 each time is taking a while. I'm building and pushing new docker images.

I've created new docker images and included the files in this PR. We now have scipy and netCDF4 installed in the image and pip install gcsfs, xarray, and zarr from git master on each image load.

jhamman

@mrocklin - What do you want to do here? Should we merge this and then incrementally add functionality to these configs? Autoscaling would be an obvious feature to work in but that doesn't have to happen immediately.

mrocklin · 2018-01-05T16:48:41Z

@mrocklin - What do you want to do here? Should we merge this and then incrementally add functionality to these configs? Autoscaling would be an obvious feature to work in but that doesn't have to happen immediately.

I'm happy to wait before merging.

mrocklin · 2018-01-05T16:49:44Z

OK, for fuse it looks like the docker container needs to be run with elevated priveleges. Locally I do the following:

docker run -it --device /dev/fuse --cap-add SYS_ADMIN --privileged daskdev/pangeo-notebook

How do I specify flags like --device /dev/fuse --cap-add SYS_ADMIN --privileged in the config.yaml file? cc @yuvipanda

rabernat · 2018-01-05T23:11:02Z

@@ -0,0 +1,30 @@
+# Start cluster on Google cloud
+gcloud container clusters create pangeo --num-nodes=10 --machine-type=n1-standard-2 --zone=us-central1-b  --cluster-version=1.8.4-gke.1
+gcloud container clusters get-credentials pangeo --zone us-central1-b --project pangeo-181919


At some point, using preemptible VMs might save us lots of money
https://cloud.google.com/kubernetes-engine/docs/concepts/preemptible-vm

I agree. There is some delay in launching new nodes, but generally I agree that long term this is a good strategy. Short term I'm just leaving a few nodes on for a while. I'm judging that the cost in expense is worth it to reduce friction when getting started. Please let me know if you feel differently and I can adjust.

This is good for now.

rabernat · 2018-01-05T23:11:13Z

+gcloud container clusters get-credentials pangeo --zone us-central1-b --project pangeo-181919
+
+# Set up Kubernetes
+kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=mrocklinWgmail.com


typo in email address?

rabernat · 2018-01-05T23:12:07Z

+    xarray \
+    netcdf4 \
+    scipy \
+    && conda clean -tipsy


Maybe we should be using a standard environment file here?

Sure, I would be fine with that

yuvipanda · 2018-01-06T01:44:49Z

+   clientSecret: SECRET
+   callbackUrl: "http://pangeo.pydata.org/hub/oauth_callback"
+
+admin:


This needs to be under 'auth' for it to take effect.

This no longer relies on the dask-foo images. Fuse now works on the notebook and worker image Adds .daskernetes.yaml file to define workers

mrocklin · 2018-01-06T21:57:41Z

+COPY prepare.sh /usr/bin/prepare.sh
+RUN chmod +x /usr/bin/prepare.sh
+RUN mkdir /home/$NB_USER/examples && chown -R $NB_USER /home/$NB_USER/examples
+COPY examples/ /home/$NB_USER/examples


@yuvipanda will this work or will the user's home directory be overridden by jupyterhub?

Also add a --death-timeout for the workers to clean them up

yuvipanda · 2018-01-06T22:20:20Z

It will be overwritten. You can put them in a non home directory and copy them post start. Or you can turn off persistent storage completely and that will make these show up.

…

On Jan 6, 2018 1:57 PM, "Matthew Rocklin" ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In gce/notebook/Dockerfile <#57 (comment)>: > +RUN jupyter labextension install @***@***.*** + +RUN pip install click jedi kubernetes --upgrade + +RUN pip install git+https://github.com/yuvipanda/daskernetes \ + git+https://github.com/zarr-developers/zarr \ + ***@***.***/zarr_set_attrs \ + ***@***.*** \ + fusepy + +USER root +RUN mkdir /gcs && chown -R $NB_USER /gcs +COPY prepare.sh /usr/bin/prepare.sh +RUN chmod +x /usr/bin/prepare.sh +RUN mkdir /home/$NB_USER/examples && chown -R $NB_USER /home/$NB_USER/examples +COPY examples/ /home/$NB_USER/examples @yuvipanda <https://github.com/yuvipanda> will this work or will the user's home directory be overridden by jupyterhub? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#57 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB23hZ3vJoxM9BU15M-kEoV3P8W_cU6ks5tH-xXgaJpZM4RNCEz> .

mrocklin · 2018-01-06T22:24:56Z

Persistent storage seems pretty valuable. I'll copy them over in the startup script. Thanks!

…

On Sat, Jan 6, 2018 at 4:20 PM, Yuvi Panda ***@***.***> wrote: It will be overwritten. You can put them in a non home directory and copy them post start. Or you can turn off persistent storage completely and that will make these show up. On Jan 6, 2018 1:57 PM, "Matthew Rocklin" ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In gce/notebook/Dockerfile > <#57 (comment)>: > > > +RUN jupyter labextension install @jupyter-widgets/jupyterlab- ***@***.*** > + > +RUN pip install click jedi kubernetes --upgrade > + > +RUN pip install git+https://github.com/yuvipanda/daskernetes \ > + git+https://github.com/zarr-developers/zarr \ > + ***@***.***/zarr_set_attrs \ > + ***@***.*** \ > + fusepy > + > +USER root > +RUN mkdir /gcs && chown -R $NB_USER /gcs > +COPY prepare.sh /usr/bin/prepare.sh > +RUN chmod +x /usr/bin/prepare.sh > +RUN mkdir /home/$NB_USER/examples && chown -R $NB_USER /home/$NB_USER/examples > +COPY examples/ /home/$NB_USER/examples > > @yuvipanda <https://github.com/yuvipanda> will this work or will the > user's home directory be overridden by jupyterhub? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#57# pullrequestreview-87085455>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAB23hZ3vJoxM9BU15M- kEoV3P8W_cU6ks5tH-xXgaJpZM4RNCEz> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#57 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszNroFHntURGJNLi6vhUgDzVnS-sCks5tH_GlgaJpZM4RNCEz> .

yuvipanda · 2018-01-12T21:13:15Z

Things from this we have to upstream (from looking at customizations we had to perform with modify_pod_hook):

Privileged notebook access (Privileged notebook containers jupyterhub/zero-to-jupyterhub-k8s#300) (although you probably want FlexVolume or a CSI provider in the long term, see the linked issue)
Notebook capabilities add / remove
Set default URL for launching (allows us to set JupyterLab instead of notebook as default) (Allow making JupyterLab default thing to launch jupyterhub/zero-to-jupyterhub-k8s#375)
Set service account to be used (allow setting service account from config.ayml jupyterhub/zero-to-jupyterhub-k8s#404)

I'll file bugs in z2jh for these and link from here appropriately.

rabernat · 2018-01-17T13:22:23Z

Should we consider merging this?

jhamman · 2018-01-19T05:09:42Z

Unless there is any opposition, I'll merge this tomorrow afternoon.

Add basic JupyterHub + Dask deployment

b449a89

jhamman reviewed Dec 27, 2017

View reviewed changes

mrocklin added 3 commits December 26, 2017 22:21

add pip/conda packages

37e6742

Add xarray, netCDF, scipy to docker images

b37fdc6

add admin users

8034e09

mrocklin mentioned this pull request Dec 28, 2017

Add support for "grouped" adaptive scaling and adaptive behavior overrides. dask/distributed#1632

Merged

3 tasks

jhamman approved these changes Jan 2, 2018

View reviewed changes

mrocklin added 2 commits January 2, 2018 19:02

Merge branch 'master' into gce

b2e501b

Add fuse to notebook image

4e04aa7

mrocklin added 2 commits January 5, 2018 14:27

update config files

98e6109

add sudo to pangeo-notebook

95be16e

rabernat reviewed Jan 5, 2018

View reviewed changes

yuvipanda reviewed Jan 6, 2018

View reviewed changes

mrocklin added 3 commits January 5, 2018 21:41

update config

6d6091b

add jedi to notebook

2074e16

Redesign docker files

035711b

This no longer relies on the dask-foo images. Fuse now works on the notebook and worker image Adds .daskernetes.yaml file to define workers

mrocklin commented Jan 6, 2018

View reviewed changes

add a few missing libraries found when using Zarr

d91110b

Also add a --death-timeout for the workers to clean them up

mrocklin added 6 commits January 6, 2018 17:23

clean up notebook docker image

fbd2aa7

add matplotlib

6972d63

add notebook prepare script and notebook

725925c

add --sys-prefix to nbsererproxy install

bd44fbd

add notebooks

2b51b8f

cull users after 10m

a8cc25b

mrocklin mentioned this pull request Jan 10, 2018

test GCP cluster on geotiff / landast #66

Closed

remove dask helm values.yaml

a459e43

yuvipanda mentioned this pull request Jan 12, 2018

Dask integration with JupyterHub jupyterhub/zero-to-jupyterhub-k8s#326

Closed

update pangeo gce instructions

cc28531

mrocklin merged commit 7171fd3 into pangeo-data:master Jan 19, 2018

mrocklin deleted the gce branch January 19, 2018 14:06

Conversation

mrocklin commented Dec 26, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jhamman left a comment

Choose a reason for hiding this comment

Uh oh!

mrocklin commented Jan 5, 2018

Uh oh!

mrocklin commented Jan 5, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuvipanda commented Jan 6, 2018 via email

Uh oh!

mrocklin commented Jan 6, 2018 via email

Uh oh!

yuvipanda commented Jan 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rabernat commented Jan 17, 2018

Uh oh!

jhamman commented Jan 19, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yuvipanda commented Jan 12, 2018 •

edited

Loading