It seems that the dask pytest suite is exceeding the number of open file handles allowed by default on at least two major operating systems. We should consider updating the contributing guide to include this information for new users.

What happened:
Following the instructions in the contributing guide, I set up a new conda environment

conda env create -n dask-dev -f continuous_integration/environment-latest.yaml
conda activate dask-dev
python -m pip install --no-deps -e .

But when I run pytest (the next step from the contributing guide), many tests fail with the message "too many open files"

py.test dask --verbose --doctest-modules

I see this behavior on Ubuntu 20.04 and MacOS Catalina. Strangely, I only see this for python 3.8 environments, and not python 3.7. I don't know why this is.

Increasing the number of allowable open file handles to 2048 seems to fix the problem.

ulimit -n 2048

(My laptop running Ubuntu had a default limit of 1024, and my Mac laptop had a default limit of 256.)

What you expected to happen:
I expected pytest to complete without any failures if I followed the instructions in the contributing guide.

Anything else we need to know?:
The contributing guide also has instructions for pip installation, but when I tried this in a brand new conda environment (both with python 3.7 and 3.8) pip was unable to resolve dependency conflicts and quit.

Environment:

• Dask version: master branch
• Python version: 3.8 (pytest problem behavior not seen with python 3.7 environments)
• Operating System: Ubuntu 20.04 and MacOS Catalina 10.15.7
• Install method (conda, pip, source): conda