-
-
Notifications
You must be signed in to change notification settings - Fork 3.2k
Description
- Platform information (e.g. Ubuntu Linux 16.04): Debian buster
- OS architecture (e.g. amd64): x86_64
- pyenv version: 3086e6e
- Python version: system 3.7.3, installing 3.9.2
- C Compiler information (e.g. gcc 7.3): 8.3.0
- Please attach verbose build log as gist: useless, see below
I'm trying to figure out how to set the number of parallel processes used for compilation in a pyenv install 3.9.2. The background is that I have a small VM on a bigger host machine; the VM sees all host CPU cores but only has a limited amount of RAM, and compiling with the default parallelism causes the loadavg to shoot into the dozens and the machine to virtually freeze within seconds. As a result, I cannot actually run a build to completion, and the partial build log contains no information whatsoever on this parallelisation. (Let me know if it's needed anyway.)
I tried to set MAKE_OPTS and/or PYTHON_MAKE_OPTS to -j 4, but this appears to be ignored for the actual CPython build; specifically, the ./python -E ./setup.py build process (as it shows up in htop) still spawns as many gcc processes as there are CPUs regardless of those environment variables.
Digging in CPython a bit, it looks like this parallel compilation was only introduced relatively recently with 3.8.0 (bpo-36786). The patch there hardcodes -j0 in CPython's Makefile.pre.in, and I don't see any obvious way to override that through an environment variable or similar. If this is correct, the proper fix would likely require an upstream change, but even if that lands immediately, it won't help with existing 3.8.x and 3.9.x versions, so a workaround in pyenv may still be necessary. However, I attempted to patch either Makefile.pre.in or Lib/compileall.py without success, so my understanding here may well be incorrect.