When a package indirectly depends on python, any attempt to run EnvironmentModifications.from_sourcing_file inside its installation environment ends up in an error. The python command is broken as well (and this is probably the real cause of the observed issue).
Expected Result
To have that package installed.
Actual Result
$ spack install example-dependent
[lots of usual output...]
==> Executing phase: 'install'
==> Error: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code
RuntimeError: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code
/home/spack/spack-repo-test/packages/example-dependent/package.py:16, in install:
1 to_be_sourced = fs.join_path(prefix, 'SOURCE_ME')
2 with open(to_be_sourced, 'w') as f:
3 f.write('export SOURCE_ME_HAS_BEEN_SOURCED=1')
>> 4 EnvironmentModifications.from_sourcing_file(to_be_sourced).apply_modifications()
5 fs.touch(fs.join_path(prefix, 'INSTALL_DONE'))
See build log for details:
/home/spack/spack/var/spack/stage/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/bzip2-1.0.6/spack-build.outThe actual error raised by the call is:
$ cat /home/spack/spack/var/spack/stage/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/bzip2-1.0.6/spack-build.out
==> Executing phase: 'install'
ImportError: No module named site
Steps to reproduce the issue
I've managed to reduce the problem to a minimal set of packages that can be obtained from GitHub:
$ git clone https://github.com/nazavode/spack-repo-test.git ~/spack-repo-test
$ spack repo add ~/spack-repo-test
That Spack repo contains two packages:
example-dependency that depends on pythonexample-dependent that depends on example-dependency
Having that repo added to Spack allows us to obtain this interesting spec:
$ spack spec example-dependent
Input spec
--------------------------------
example-dependent
Concretized
--------------------------------
example-dependent@1.0.6%gcc@4.8.5 arch=linux-centos7-x86_64
^example-dependency@1.0.6%gcc@4.8.5 arch=linux-centos7-x86_64
^python@2.7.14%gcc@4.8.5 patches=123082ab3483ded78e86d7c809e98a804b3465b4683c96bd79a2fd799f572244 +pic+shared~tk~ucs4 arch=linux-centos7-x86_64
[a lot of other stuff python depends on...]This minimal dependency graph is enough to trigger the issue:
$ spack install example-dependent
[lots of usual output...]
==> Executing phase: 'install'
==> Error: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code
RuntimeError: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code
/home/spack/spack-repo-test/packages/example-dependent/package.py:16, in install:
1 to_be_sourced = fs.join_path(prefix, 'SOURCE_ME')
2 with open(to_be_sourced, 'w') as f:
3 f.write('export SOURCE_ME_HAS_BEEN_SOURCED=1')
>> 4 EnvironmentModifications.from_sourcing_file(to_be_sourced).apply_modifications()
5 fs.touch(fs.join_path(prefix, 'INSTALL_DONE'))
See build log for details:
/home/spack/spack/var/spack/stage/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/bzip2-1.0.6/spack-build.outThe root of the problem lies in the fact that the python interpreter inside the installation environment is broken:
$ spack install --only dependencies example-dependent
[installs just fine, output omitted...]
$ spack env example-dependent python
ImportError: No module named site
Inspecting the affected environment we get:
$ spack env example-dependent bash
$ which python
/usr/bin/python
$ env | grep PYTHON
PYTHONHOME=/home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/python-2.7.14-4ipldugnhasdas2pjasdevq4ruha6biu
PYTHONPATH=
$ python
ImportError: No module named site
It looks like ^python@2.7.14 messed with our environment injecting that PYTHONHOME that our system's /usr/bin/python is not standing at all. The fun part is that some distro (Ubuntu 16.04 for instance) ships with a python that actually likes the broken environment and proved to be working fine (see below for details).
Wrap up
- When a package depends indirectly on
python, it ends up with:
PYTHONHOME=<path to spack python prefix>PYTHONPATH=which python is <system python>, which is broken or not depending on its version. This in turn breaks EnvironmentModifications.from_sourcing_file as well.
- When a package depends directly on
python, it ends up with:
PYTHONHOME=<path to spack python prefix>PYTHONPATH=<path to spack python prefix>which python is <spack python>, which works correctly.
Information on your system
Reproduced this on a centos7 docker image that ships with python 2.7.5 while tested to be working fine on a ubuntu 16.04 docker image that ships with python 2.7.12. Spack was cd084d0.
A Dockefile that executes the whole issue reproduction procedure can be found here.
When a package indirectly depends on python, any attempt to run
EnvironmentModifications.from_sourcing_fileinside its installation environment ends up in an error. Thepythoncommand is broken as well (and this is probably the real cause of the observed issue).Expected Result
To have that package installed.
Actual Result
The actual error raised by the call is:
Steps to reproduce the issue
I've managed to reduce the problem to a minimal set of packages that can be obtained from GitHub:
That Spack repo contains two packages:
example-dependencythat depends onpythonexample-dependentthat depends onexample-dependencyHaving that repo added to Spack allows us to obtain this interesting spec:
This minimal dependency graph is enough to trigger the issue:
The root of the problem lies in the fact that the
pythoninterpreter inside the installation environment is broken:Inspecting the affected environment we get:
It looks like
^python@2.7.14messed with our environment injecting thatPYTHONHOMEthat our system's/usr/bin/pythonis not standing at all. The fun part is that some distro (Ubuntu 16.04 for instance) ships with a python that actually likes the broken environment and proved to be working fine (see below for details).Wrap up
python, it ends up with:PYTHONHOME=<path to spack python prefix>PYTHONPATH=which pythonis<system python>, which is broken or not depending on its version. This in turn breaksEnvironmentModifications.from_sourcing_fileas well.python, it ends up with:PYTHONHOME=<path to spack python prefix>PYTHONPATH=<path to spack python prefix>which pythonis<spack python>, which works correctly.Information on your system
Reproduced this on a centos7 docker image that ships with
python 2.7.5while tested to be working fine on a ubuntu 16.04 docker image that ships withpython 2.7.12. Spack was cd084d0.A
Dockefilethat executes the whole issue reproduction procedure can be found here.