I believe I've found a possible file handle leak in Spack. This manifests itself when installing very large environments.
Steps to reproduce the issue
I'm trying to install the following environment:
spack:
specs:
# Shells
- bash
# Spack dependencies
- clingo
- gnupg
- graphviz
- kcov
# Newer versions of Linux tools
- watch
- wget
- unrar
# Software installation tools
- automake
- autoconf
- m4
- cmake
- ninja
- patchelf
- scons
# Version control systems
- mercurial
- subversion
# Python libraries
- python
- py-azureml-sdk
- py-black
- py-cartopy
- py-cmocean
- py-codecov
- py-dask
- py-fiona
- py-flake8
- py-flake8-import-order
- py-geocube
- py-geopandas
- py-geoplot
- py-inference-schema+numpy
- py-ipywidgets
- py-joblib
- py-jupyterlab
- py-matplotlib
- py-metpy
- py-mypy
- py-numpy
- py-openpyxl
- py-pandas
- py-pycocotools
- py-pygeos
- py-pyinstrument
- py-pytest
- py-pytest-cov
- py-pytest-mock
- py-pyyaml
- py-rarfile
- py-rasterio
- py-scikit-learn
- py-scipy
- py-seaborn
- py-setuptools
- py-shapely
- py-sphinx
- py-sphinx-rtd-theme
- py-sphinxcontrib-programoutput
- py-statsmodels
- py-tables
- py-torch
- py-torchvision
- py-twine
- py-vermin
- py-wheel
- py-xarray
- py-xgboost
# Research libraries
- gdal
- opencv
concretization: together
A few of these packages aren't yet in Spack, but you should be able to substitute them with other packages and still reproduce this.
Error Message
After successfully installing several packages, I eventually hit:
==> Installing py-pytest-cov-2.8.1-jvm5vpnwx3vgzanxse2putpvi6ft54o5
==> No binary for py-pytest-cov-2.8.1-jvm5vpnwx3vgzanxse2putpvi6ft54o5 found: installing from source
==> Warning: Missing a source id for apple-libuuid@1353.100.2
==> Error: Failed to install py-pytest-cov due to OSError: [Errno 24] Too many open files
==> Installing py-pytest-mock-1.11.1-f2yunbyvz5f7sm6p5fkunwpu3lu6zc2f
==> No binary for py-pytest-mock-1.11.1-f2yunbyvz5f7sm6p5fkunwpu3lu6zc2f found: installing from source
==> Warning: Missing a source id for apple-libuuid@1353.100.2
==> Error: Failed to install py-pytest-mock due to ProcessError: /usr/bin/git: Too many open files
Command: '/usr/bin/git' 'describe' '--tags' '--match' 'v*'
==> Installing py-requests-2.24.0-cbnrnlrvv32qea7ofousm62yr7g5o4eq
==> No binary for py-requests-2.24.0-cbnrnlrvv32qea7ofousm62yr7g5o4eq found: installing from source
==> Warning: Missing a source id for apple-libuuid@1353.100.2
==> Warning: Skipping build of py-azureml-core-1.23.0-ufhtrskom5dhrbf2ohomke6d75uaw5m5 since py-requests-2.24.0-cbnrnlrvv32qea7ofousm62yr7g5o4eq failed
==> Warning: Skipping build of py-azureml-pipeline-core-1.23.0-6o5oa3nknsfca3a2yrplbj6l2rikwwyp since py-azureml-core-1.23.0-ufhtrskom5dhrbf2ohomke6d75uaw5m5 failed
==> Error: [Errno 24] Too many open files: '/Users/Adam/spack/opt/spack/.spack-db/lock'
Information on your system
- Spack: 0.16.1-2888-a284edb852
- Python: 3.8.10
- Platform: darwin-catalina-ivybridge
- Concretizer: clingo
Additional information
I believe I've found a possible file handle leak in Spack. This manifests itself when installing very large environments.
Steps to reproduce the issue
I'm trying to install the following environment:
A few of these packages aren't yet in Spack, but you should be able to substitute them with other packages and still reproduce this.
Error Message
After successfully installing several packages, I eventually hit:
Information on your system
Additional information
spack debug reportand reported the version of Spack/Python/Platform