Skip to content

Override the number of parallel compilation processes #1857

@JustAnotherArchivist

Description

@JustAnotherArchivist
  • Platform information (e.g. Ubuntu Linux 16.04): Debian buster
  • OS architecture (e.g. amd64): x86_64
  • pyenv version: 3086e6e
  • Python version: system 3.7.3, installing 3.9.2
  • C Compiler information (e.g. gcc 7.3): 8.3.0
  • Please attach verbose build log as gist: useless, see below

I'm trying to figure out how to set the number of parallel processes used for compilation in a pyenv install 3.9.2. The background is that I have a small VM on a bigger host machine; the VM sees all host CPU cores but only has a limited amount of RAM, and compiling with the default parallelism causes the loadavg to shoot into the dozens and the machine to virtually freeze within seconds. As a result, I cannot actually run a build to completion, and the partial build log contains no information whatsoever on this parallelisation. (Let me know if it's needed anyway.)

I tried to set MAKE_OPTS and/or PYTHON_MAKE_OPTS to -j 4, but this appears to be ignored for the actual CPython build; specifically, the ./python -E ./setup.py build process (as it shows up in htop) still spawns as many gcc processes as there are CPUs regardless of those environment variables.

Digging in CPython a bit, it looks like this parallel compilation was only introduced relatively recently with 3.8.0 (bpo-36786). The patch there hardcodes -j0 in CPython's Makefile.pre.in, and I don't see any obvious way to override that through an environment variable or similar. If this is correct, the proper fix would likely require an upstream change, but even if that lands immediately, it won't help with existing 3.8.x and 3.9.x versions, so a workaround in pyenv may still be necessary. However, I attempted to patch either Makefile.pre.in or Lib/compileall.py without success, so my understanding here may well be incorrect.

Metadata

Metadata

Assignees

No one assigned

    Labels

    third-party: pythonthe problem is in the specific Python version(s)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions