I'm attempting to use spack setup to establish an out-of-stage build environment for cmake-based C++ projects, as described here. However, the binaries produced under this workflow have link errors for dependencies related to the compiler's runtime library.
Steps to reproduce the issue
$ spack setup $SPEC && mkdir build && cd build && ../spconfig.py .. && make && ./the-binary
Error description
$ ldd the-binary
...
libc++.so.1 => not found
libc++abi.so.1 => not found
These dependencies are for clang's libc++ runtime, which are configured by the extra_rpaths variable in compilers.yaml.
Information on your system
compilers.yaml:
compilers:
- compiler:
environment: {}
extra_rpaths:
- /spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/llvm-8.0.0-vcde4opt4xfhe5qcrktcagi6r7uhskgf/lib
flags: {}
modules:
- llvm-8.0.0-gcc-5.4.0-vcde4op
operating_system: ubuntu16.04
paths:
cc: /spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/llvm-8.0.0-vcde4opt4xfhe5qcrktcagi6r7uhskgf/bin/clang
cxx: /spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/llvm-8.0.0-vcde4opt4xfhe5qcrktcagi6r7uhskgf/bin/clang++
f77: /usr/bin/gfortran
fc: /usr/bin/gfortran
spec: clang@8.0.0
target: x86_64Additional Information
My CMakeLists.txt is correctly configured to use rpaths, and does so for all package-level dependencies (i.e., dependencies declared in the package's package.py file). However, the rpaths for the missing dependencies are defined by the extra_rpaths variable in the compilers.yaml file, and are completely absent from the generated spconfig.py file (attached). The bug as I understand it is that rpaths declared for the compiler with extra_rpaths are not encoded in the CMAKE_INSTALL_RPATH variable in the generated spconfig.py file.
Workarounds
I know of two workarounds.
Loading the compiler's module
The issue can be avoided by loading llvm (spack load llvm) prior to executing the binary (./the-binary). However, if the binary is installed with make install, then the issue will resurface.
Explicitly depending upon LLVM
@chuckatkins suggested in a comment regarding issue #11582 that it might be fitting to make the compiler a dependency of all packages that it builds (i.e., with depends_on() in the package.py file). I've tried this fix and can confirm that it works, but with the caveat that I have to hard-code a dependency on LLVM without knowledge of whether that's actually the compiler that the user is using. Is it possible to gain access to the spec (or the compiler) in the class definition of a package (i.e., outside of a method)? Additionally, this typically ends up reinstalling the compiler to satisfy the need for the compiler dependency to be compiled by itself.
Attachments
spconfig.py:
#!/usr/bin/python
#
import sys
import os
import subprocess
def cmdlist(str):
return list(x.strip().replace("'",'') for x in str.split('\n') if x)
env = dict(os.environ)
env['CC'] = '/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/llvm-8.0.0-vcde4opt4xfhe5qcrktcagi6r7uhskgf/bin/clang'
env['CMAKE_PREFIX_PATH'] = ":".join(cmdlist("""
/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/cmake-3.12.3-mt7wnhrl2gwab3qb3bll4nvo2pmj7idw
"""))
env['CXX'] = '/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/llvm-8.0.0-vcde4opt4xfhe5qcrktcagi6r7uhskgf/bin/clang++'
env['FC'] = '/usr/bin/gfortran'
env['PATH'] = ":".join(cmdlist("""
/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/cmake-3.12.3-mt7wnhrl2gwab3qb3bll4nvo2pmj7idw/bin
/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/cmake-3.12.3-mt7wnhrl2gwab3qb3bll4nvo2pmj7idw/bin
/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/llvm-8.0.0-vcde4opt4xfhe5qcrktcagi6r7uhskgf/bin
/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/environment-modules-3.2.10-mb3jzyhb6r5wkb64ojkr5porf2zxjie6/Modules/bin
/spack/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/snap/bin
"""))
env['SPACK_TRANSITIVE_INCLUDE_PATH'] = ";".join(cmdlist("""
/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/cmake-3.12.3-mt7wnhrl2gwab3qb3bll4nvo2pmj7idw/include
"""))
cmd = cmdlist("""
/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/cmake-3.12.3-mt7wnhrl2gwab3qb3bll4nvo2pmj7idw/bin/cmake
-G
Unix Makefiles
-DCMAKE_INSTALL_PREFIX:PATH=/spack/opt/spack/linux-ubuntu16.04-x86_64/clang-8.0.0/hdiagsutil-master-c5egaqfc2lvd4jl5ragshnp6edowy3hd
-DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo
-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=FALSE
-DCMAKE_INSTALL_RPATH:STRING=/spack/opt/spack/linux-ubuntu16.04-x86_64/clang-8.0.0/hdiagsutil-master-c5egaqfc2lvd4jl5ragshnp6edowy3hd/lib;/spack/opt/spack/linux-ubuntu16.04-x86_64/clang-8.0.0/hdiagsutil-master-c5egaqfc2lvd4jl5ragshnp6edowy3hd/lib64
-DCMAKE_PREFIX_PATH:STRING=/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/cmake-3.12.3-mt7wnhrl2gwab3qb3bll4nvo2pmj7idw
-DCMAKE_C_FLAGS=
-DCMAKE_CXX_FLAGS=
""") + sys.argv[1:]
proc = subprocess.Popen(cmd, env=env)
proc.wait()
I'm attempting to use
spack setupto establish an out-of-stage build environment for cmake-based C++ projects, as described here. However, the binaries produced under this workflow have link errors for dependencies related to the compiler's runtime library.Steps to reproduce the issue
$ spack setup $SPEC && mkdir build && cd build && ../spconfig.py .. && make && ./the-binaryError description
These dependencies are for clang's libc++ runtime, which are configured by the
extra_rpathsvariable incompilers.yaml.Information on your system
compilers.yaml:Additional Information
My
CMakeLists.txtis correctly configured to use rpaths, and does so for all package-level dependencies (i.e., dependencies declared in the package'spackage.pyfile). However, the rpaths for the missing dependencies are defined by theextra_rpathsvariable in thecompilers.yamlfile, and are completely absent from the generatedspconfig.pyfile (attached). The bug as I understand it is that rpaths declared for the compiler withextra_rpathsare not encoded in theCMAKE_INSTALL_RPATHvariable in the generatedspconfig.pyfile.Workarounds
I know of two workarounds.
Loading the compiler's module
The issue can be avoided by loading llvm (
spack load llvm) prior to executing the binary (./the-binary). However, if the binary is installed withmake install, then the issue will resurface.Explicitly depending upon LLVM
@chuckatkins suggested in a comment regarding issue #11582 that it might be fitting to make the compiler a dependency of all packages that it builds (i.e., with
depends_on()in thepackage.pyfile). I've tried this fix and can confirm that it works, but with the caveat that I have to hard-code a dependency on LLVM without knowledge of whether that's actually the compiler that the user is using. Is it possible to gain access to the spec (or the compiler) in the class definition of a package (i.e., outside of a method)? Additionally, this typically ends up reinstalling the compiler to satisfy the need for the compiler dependency to be compiled by itself.Attachments
spconfig.py: