Skip to content

Python 3.10 closure contains incompatible versions of CUDA and g++, causing build failures #217878

@bcdarwin

Description

@bcdarwin

Describe the bug

As per the title, the current CUDA and stdenv versions used to build torchWithCuda are seemingly incompatible and can lead to build failures stating this (specifically torchvision - see below - but likely others as well).

Likely introduced in #214887 [of course it was the actual stdenv bump], but as it's a dynamic failure it's not a triviality to add an assert to torch to catch similar errors in future.

Steps To Reproduce

  1. Built python310Packages.torchvision with CUDA-enabled torch (e.g. cudaSupport = true; in config.nix or override torch = torchWithCuda in torchvision).
  2. Observe the build fails with the following output:
  File "/nix/store/v9qczq3z12j33pnkzdfmpv7mq6zw9yvm-python3.10-setuptools-65.6.3/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
    self.build_extensions()
  File "/nix/store/z8y26ilyi7qgi9ai3m3dcn7iz6b24b2n-python3.10-torch-1.13.1/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 499, in build_extensions
    _check_cuda_version(compiler_name, compiler_version)
  File "/nix/store/z8y26ilyi7qgi9ai3m3dcn7iz6b24b2n-python3.10-torch-1.13.1/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 416, in _check_cuda_version
    raise RuntimeError(
RuntimeError: The current installed version of g++ (12.2.0) is greater than the maximum required version by CUDA 11.7 (11.5.0). Please make sure to use an adequate version
of g++ (>=6.0.0, <=11.5.0).
/nix/store/b09v23lirgvci3wzszh22mbkdfj0h0yq-stdenv-linux/setup: line 1582: pop_var_context: head of shell_variables not a function context
error: builder for '/nix/store/yv47nwpr15gggbfh7wjb5hjd6xn4pcxq-python3.10-torchvision-0.14.1.drv' failed with exit code 1

Notify maintainers

@teh @thoughtpolice @tscholak @NickCao

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 4.15.0-169-generic, Ubuntu, 18.04.6 LTS (Bionic Beaver), nobuild`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.14.0pre20230222_4a921ba`
 - channels(bcdarwin): `"home-manager, nixpkgs"`
 - nixpkgs: `/home/bcdarwin/.nix-defexpr/channels/nixpkgs`

Metadata

Metadata

Assignees

Labels

0.kind: bugSomething is broken6.topic: cudaParallel computing platform and API6.topic: pythonPython is a high-level, general-purpose programming language.
No fields configured for issues without a type.

Projects

Status

✅ Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions