[BUG] setuptools does not protect against race conditions on parallel builds #3119
Description
setuptools version
setuptools == 60.9.3
Python version
3.9, although this is Python version agnostic
OS
MacOS, although this is OS-agnostic
Additional environment information
No response
Description
I have a Makefile that tries to build multiple wheels for a Python package, in parallel. This fails, with weird errors such as the following (from a pip wheel . --log out.log output):
2022-02-18T15:49:53,923 running clean
2022-02-18T15:49:53,923 'build/lib' does not exist -- can't clean it
2022-02-18T15:49:53,620 Copying test_package.egg-info to build/bdist.macosx-10.15-x86_64/wheel/test_package-1.0.0-py3.9.egg-info
2022-02-18T15:49:53,621 error: [Errno 2] No such file or directory
2022-02-18T16:12:39,443 creating build/bdist.macosx-10.15-x86_64/wheel/test_package-1.0.0.dist-info/WHEEL
2022-02-18T16:12:39,444 error: [Errno 2] No such file or directory: 'build/bdist.macosx-10.15-x86_64/wheel/test_package-1.0.0.dist-info/WHEEL'
2022-02-18T16:12:39,409 Copying test_package.egg-info to build/bdist.macosx-10.15-x86_64/wheel/test_package-1.0.0-py3.9.egg-info
2022-02-18T16:12:39,412 running install_scripts
2022-02-18T16:12:39,440 error: [Errno 17] File exists: 'build/bdist.macosx-10.15-x86_64/wheel/test_package-1.0.0.dist-info'
And various other errors related to files in the build directory.
This is because setuptools does not have any isolation/protection against parallel builds.
Expected behavior
setuptools works properly with parallel builds that use the same working directory. Either through some sort of separation within the build directory, or through filesystem based locks for the build directory.
A more targetted behaviour change would be to include the running Python version into the build directory paths, which would enable building extensions in parallel for a Python package; which is my main use case. I do think a more general solution would be nicer, eg for the example I have below. :)
How to Reproduce
- Take a basic setuptools package.
- Try to perform parallel builds on it (eg: building wheels for different Python versions, for C extensions).
- Notice that all these builds trample over each other, non-deterministically fail or have incorrect contents.
I'll take an example of a pure-Python wheel below, but the behavior for platform-specific wheels is similar.
Output
This is more of a reproducer -- examples of setuptools' failure messages are shown above and change depending on what had a race condition in each run. :)
/private/tmp/setuptools-parallel setuptools-parallel
❯ pip --version
pip 22.0.3 from /private/tmp/setuptools-parallel/.venv/lib/python3.9/site-packages/pip (python 3.9)
/private/tmp/setuptools-parallel setuptools-parallel
❯ pip list
Package Version
---------- -------
pip 22.0.3
setuptools 60.9.3
wheel 0.37.1
/private/tmp/setuptools-parallel setuptools-parallel
❯ tree .
.
├── run.sh
└── setup.py
0 directories, 2 files
/private/tmp/setuptools-parallel setuptools-parallel
❯ cat setup.py
from setuptools import setup
setup(
name="test-package",
version="1.0.0",
)
/private/tmp/setuptools-parallel setuptools-parallel
❯ cat run.sh
pip wheel . --wheel-dir one/ --log one/out.txt &
pip wheel . --wheel-dir two/ --log two/out.txt &
pip wheel . --wheel-dir three/ --log three/out.txt &
/private/tmp/setuptools-parallel setuptools-parallel
❯ . ./run.sh
[expect overlapping pip output below]
Processing /private/tmp/setuptools-parallel
Processing /private/tmp/setuptools-parallel
Processing /private/tmp/setuptools-parallel
Preparing metadata (setup.py) ... done
Preparing metadata (setup.py) ... done
Preparing metadata (setup.py) ... done
Building wheels for collected packages: test-package
Building wheels for collected packages: test-package
Building wheels for collected packages: test-package
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Building wheel for test-package (setup.py) ... error
ERROR: Failed building wheel for test-package
Running setup.py clean for test-package
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Building wheel for test-package (setup.py) ... error
Building wheel for test-package (setup.py) ... error ERROR: Failed building wheel for test-package
ERROR: Failed building wheel for test-package
Running setup.py clean for test-package
Running setup.py clean for test-package
Failed to build test-package
Failed to build test-package
Failed to build test-package
ERROR: Failed to build one or more wheels
ERROR: Failed to build one or more wheels
ERROR: Failed to build one or more wheelsSome example errors are mentioned in the description. If you run this yourself, you can see the failure output and wheels in one/, two/ and three/ directories.