compilers: use module-based compiler finding on Linux as well as Cray by becker33 · Pull Request #11989 · spack/spack

becker33 · 2019-07-11T18:34:14Z

Many sites deploy compilers that do not work without their modules. We should find compilers from modules for those sites.

becker33 · 2019-07-11T18:34:49Z

I'm not sure this allows us to find compilers for which there are not modules, so this is probably incomplete.

tgamblin

This looks good and is particularly impressive given that it was done in 5 minutes.

Requests:

Consolidate the module load/parse logic.
Do the module loading/parsing first, THEN do path-based compiler finding, and prefer the compilers from modules.

If we can get that done I think it is a reasonable way to get people running on TACC machines where we really need to load the modules to run the compiler.

lib/spack/spack/compiler.py

lib/spack/spack/compilers/cce.py

lib/spack/spack/operating_systems/linux_distro.py

…s of environment

…e those paths

tgamblin

@becker33: this looks very good. I have some more requests w.r.t. maintainability but this is looking great. Thanks!

tgamblin · 2019-07-20T22:33:50Z

lib/spack/spack/compilers/__init__.py

+
+        module_hints (list or None): list of modules in which to look for
+            compilers. A sensible default based on ``module avail`` will be
+            used if the values is None.


This says there are module_hints but module_hints isn't an argument.

Yeah I removed the argument and forgot to change the docstring. I'll pull it out of the docs.

lib/spack/spack/compilers/__init__.py

lib/spack/spack/util/module_cmd.py

tgamblin · 2019-07-20T23:43:37Z

lib/spack/spack/compilers/__init__.py

+    # associated with a module, it will include that module
+    module_association = {}
+    for mod in modules_to_check_paths:
+        path = spack.util.module_cmd.get_bin_path_from_module(mod)


I like this approach but see my comment on get_bin_path_from_module -- what if there's more than one thing added to PATH?

Also, one more general question about the approach: will this result in the icc modules being associated with gcc compiler entries? If so maybe we should think about this a bit. You want the right gcc to be loaded when using icc, but you likely don't want icc loaded at all when you're using gcc.

I don't think it will end up with icc modules being associated with the gcc compilers. icc modules should be adding icc to the PATH and both icc and gcc to the LD_LIBRARY_PATH. That's precisely why I'm not using the existing functionality to get a path from a module, because it would check LD_LIBRARY_PATH as well.

tgamblin · 2019-07-20T23:45:21Z

lib/spack/spack/compilers/__init__.py

+        if path:
+            if path in module_association:
+                msg = 'Multiple modules associated with path %s. ' % path
+                msg += 'Compilers may require manual configuration'


We should probably save this message for after we've tried everything, then check module_association for elements with multiple modules and print both the path and the modules. That would make the message much more useful. I realize this information will be in compilers.yaml but I think putting it in the message reduces the digging needed.

lib/spack/spack/main.py

lib/spack/spack/operating_systems/cnl.py

lib/spack/spack/util/module_cmd.py

tgamblin

Looking even better. One (hopefully last) request

tgamblin · 2019-07-23T06:34:21Z

lib/spack/spack/compilers/__init__.py

+                modules_for_paths |= module_association.get(
+                    os.path.dirname(p), set())
+
+            positive_attributes = [


@becker33: we talked about making a module ineligible to be a gcc module if it loaded gcc and any other compiler -- basically make it so that if the intel module puts icc and gcc in the path, we don't make it eligible to be included in compilers.yaml for gcc.

Did that make it in here? I'm not seeing which condition that corresponds to.

becker33 · 2019-07-24T22:53:38Z

@tgamblin should be ready for review again, comments addressed.

adrienbernede · 2019-08-26T22:33:42Z

Thanks for working on this, IMO, useful feature. Combined with recent work on compiler automated configuration, this will be awesome !

I have a few comments and questions about it:

As far as I understand it, the compiler detection will now try to find a potential path for a compiler in every single module available. Then if a compiler is found there, it will try to find the associated module. Right ?
First of all, in the tests I have done, it works ! However, this sequence of actions leads to some side effects on LC lassen and/or LC quartz:

Some fancy warnings

==> Warning: Multiple modules associated with path.
/usr/global/tools/tau/training/tau-2.26.2/ibm64linux/bin --> ['tau/2.26.3', 'tau/2.26.2']Compilers may require manual configuration

Some compilers are not detected anymore (intel compilers on lassen), since their path isn’t present in any module.
(EDIT: Looking a little deeper with @tldahlgren, the previous behavior is also covered (use of PATH), so this may be due to some change in my environment)
I guess it’s slower than it could be (because it’s parsing all the available modules). And I have seen systems with much more modules than on LC.

I would like to suggest a possible improvement:
With lmod, it is recommended to specify when a module is introducing a compiler, by adding in its definition family("compiler"). The consequence is the creation of a bunch of symlink in a path looking like /usr/tce/modulefiles/Compiler.
I couldn’t find a way to directly list those "compiler modules", but it seems possible to better sort the available modules, for improved efficiency.

(EDIT: a bit off topic)
3. Extension:
By extension, in lmod case, it is then possible to find mpi providers installed with a given compiler. IMO, the next step after improving the automated detection of compilers would be to automatically detect those MPI providers.

mwkrentel · 2019-09-29T23:13:29Z

I started looking at this and I'm finding some cases that don't fully
work.

This is on a small cluster at Rice (nots), x86_64, RedHat 7.5,
Lmod modules built with Easy Build. Spack from the
features/find-linux-compilers-by-module branch rev 92e6b4e.

(1) The GCC modules are split into two modules, GCC and GCCcore. So,
GCC/6.4.0, GCC/7.3.0, GCC/8.3.0, and GCCcore/6.4.0, GCCcore/7.3.0 and
GCCcore/8.3.0, etc.

Starting with an empty compilers.yaml and no modules loaded, spack compiler find does find GCC and PGI compilers (but not Intel or Clang).

$ spack compiler find
==> Warning: Multiple modules associated with path.
/usr/share/lmod/lmod/libexec --> ['lmod/6.6.3', 'lmod/default']Compilers may require manual configuration
==> Warning: Multiple modules associated with path.
/usr/share/lmod/lmod/settarg --> ['settarg/6.6.3', 'settarg/default']Compilers may require manual configuration
==> Added 11 new compilers to /home/krentel/.spack/linux/compilers.yaml
    pgi@19.4   pgi@17.3   gcc@7.3.0  gcc@6.4.0  gcc@5.4.0  gcc@4.8.5
    pgi@17.10  gcc@8.3.0  gcc@7.2.0  gcc@6.2.0  gcc@4.9.3
==> Compilers are defined in the following files:
    /home/krentel/.spack/linux/compilers.yaml

The problem is that spack only identifies the GCCcore module, not the
top-level GCC.

- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules:
    - GCCcore/8.3.0
    operating_system: rhel7
    paths:
      cc: /opt/apps/software/Core/GCCcore/8.3.0/bin/gcc
      cxx: /opt/apps/software/Core/GCCcore/8.3.0/bin/g++
      f77: /opt/apps/software/Core/GCCcore/8.3.0/bin/gfortran
      fc: /opt/apps/software/Core/GCCcore/8.3.0/bin/gfortran
    spec: gcc@8.3.0
    target: x86_64

This does matter and some packages fail. For example, diffutils fails
in configure checking for some system header files.

$ spack spec diffutils
diffutils@3.7%gcc@8.3.0 arch=linux-rhel7-x86_64

$ spack install diffutils
1 error found in build log:
     429    checking for a french Unicode locale...
     430    checking for a traditional japanese locale...
     431    checking for a transitional chinese locale... (cached) none
     432    checking whether sleep is declared... no
     433    checking for snprintf... (cached) no
     434    checking for socklen_t... no
  >> 435    checking for socklen_t equivalent... configure: error: Cannot find 
            a type to use in place of socklen_t

The error message is kinda misleading. I'm guessing there's something
in the -I paths that's wrong, I don't know. Anyway, the real problem
is the lack of the GCC/8.3.0 module. If I add this manually, then
spack install succeeds.

This may be a difficult case. The path to gcc@8.3.0 is actually from
GCCcore/8.3.0, /opt/apps/software/Core/GCCcore/8.3.0/bin/gcc, but
the correct, top-level module is GCC/8.3.0 (GCC loads GCCcore and
binutils). It's the same for spack or compiling directly from a
shell.

$ module show GCC/8.3.0 

whatis("Description: The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada,
 as well as libraries for these languages (libstdc++, libgcj,...).")
whatis("Homepage: http://gcc.gnu.org/")
conflict("GCC")
load("GCCcore/8.3.0")
load("binutils/.2.32")
prepend_path("MODULEPATH","/opt/apps/modules/Compiler/GCC/8.3.0")
setenv("EBROOTGCC","/opt/apps/software/Core/GCCcore/8.3.0")
setenv("EBVERSIONGCC","8.3.0")
setenv("EBDEVELGCC","/opt/apps/software/Core/GCC/8.3.0/easybuild/Core-GCC-8.3.0-easybuild-devel")
family("compiler")

$ module show GCCcore/8.3.0 

whatis("Description: The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada,
 as well as libraries for these languages (libstdc++, libgcj,...).")
whatis("Homepage: http://gcc.gnu.org/")
conflict("GCCcore")
prepend_path("MODULEPATH","/opt/apps/modules/Compiler/GCCcore/8.3.0")
prepend_path("CPATH","/opt/apps/software/Core/GCCcore/8.3.0/include")
prepend_path("LD_LIBRARY_PATH","/opt/apps/software/Core/GCCcore/8.3.0/lib")
prepend_path("LD_LIBRARY_PATH","/opt/apps/software/Core/GCCcore/8.3.0/lib64")
prepend_path("LD_LIBRARY_PATH","/opt/apps/software/Core/GCCcore/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0")
prepend_path("LIBRARY_PATH","/opt/apps/software/Core/GCCcore/8.3.0/lib")
prepend_path("LIBRARY_PATH","/opt/apps/software/Core/GCCcore/8.3.0/lib64")
prepend_path("MANPATH","/opt/apps/software/Core/GCCcore/8.3.0/share/man")
prepend_path("PATH","/opt/apps/software/Core/GCCcore/8.3.0/bin")
setenv("EBROOTGCCCORE","/opt/apps/software/Core/GCCcore/8.3.0")
setenv("EBVERSIONGCCCORE","8.3.0")
setenv("EBDEVELGCCCORE","/opt/apps/software/Core/GCCcore/8.3.0/easybuild/Core-GCCcore-8.3.0-easybuild-devel")

adrienbernede · 2019-09-30T16:26:56Z

@mwkrentel, hello !

It seems that GCCcore modules are visible to the users, are they meant to be loadable directly by users, or only indirectly by other modules ? In the latter case, could they be hidden ? I’m not sure it would solve the issue though...

I can see that only GCC modules define the family("compiler") property, which is great. This could be another way to distinguish both GCC and GCCcore. However, this is lmod specific.

mwkrentel · 2019-09-30T18:22:35Z

@adrienbernede I think these are mostly questions about Easy Build, of
which I don't know much.

The default (system) MODULEPATH is /opt/apps/modules/Core which
includes GCC, GCCcore, PGI, intel and lots more.

I'm guessing one of the main reasons for splitting the module is that
other compilers (PGI, intel) also share GCCcore. For example, loading
the PGI or intel module also pulls in GCCcore. There are modules for
(among others):

GCC/8.3.0
GCCcore/8.3.0
PGI/19.4-GCC-8.3.0
icc/2019.3.199-GCC-8.3.0

But if you want to use the compiler, then you load the top-level
module (GCC, PGI, intel) and then it loads a few other modules.

Anyway, it seems that the spack user should always check the compiler
entry to make sure it includes the same modules that module load
would pull in (or at least the top-level modules that pull in the
dependencies).

becker33 · 2020-01-21T18:44:23Z

lib/spack/spack/util/module_cmd.py

+        # shenanigans to ensure python will run.
+
+        # LD_LIBRARY_PATH under which Spack ran
+        os.environ['SPACK_LD_LIBRARY_PATH'] = spack.main.spack_ld_library_path


I think this still has a problem.

If we load multiple modules, we will only get the changes to LD_LIBRARY_PATH from the last one loaded.

We need to somehow diff the LD_LIBRARY_PATHS and apply the diff, not overwrite.

alalazo · 2021-12-17T17:50:37Z

Is this something we need to revisit at a later time? If so I think we need to figure out a way to find packages based on module file inspection (thus targeting compiler as dependencies rather than the current state of things).

mwkrentel · 2021-12-20T04:36:39Z

I previously put some comments on this in #9376. In short,
spack compiler find automatically finds compilers from modules on
Cray, so I wondered why it didn't do this on all Linux.

Also, I was hoping that when spack finds a compiler from a module that
it could automatically fill in the modules: field in compilers.yaml.
There are situations where this matters and it's a very easy mistake
to make.

The existing way to tell spack about a compiler from a module is to
load the module and run spack compiler find. In this case, spack
finds the path to the compiler but not its module.

As a compromise, I suggested adding a sub-command to spack compiler
where the user tells spack what the module is.

spack compiler module GCC/9.3.0

This would tell spack that GCC/9.3.0 is the name of a module for a
compiler and please add it to compilers.yaml and please fill in the
modules: field (so there is no extra manual step).

But note: some compiler modules depend on other modules, eg,
GCC/9.3.0 might require that module plus GCCcore/9.3.0.

becker33 · 2022-11-04T17:08:14Z

Superseded by #29392

initial implementation

2f8ad39

becker33 added the WIP label Jul 11, 2019

becker33 requested a review from tgamblin July 11, 2019 18:34

tgamblin requested changes Jul 11, 2019

View reviewed changes

lib/spack/spack/compiler.py Outdated Show resolved Hide resolved

lib/spack/spack/compilers/cce.py Outdated Show resolved Hide resolved

lib/spack/spack/operating_systems/linux_distro.py Outdated Show resolved Hide resolved

tgamblin changed the title ~~Find linux compilers from modules~~ compilers: use module-based compiler finding on Linux as well as Cray Jul 11, 2019

becker33 added 8 commits July 11, 2019 19:33

Ensure python runs with proper ld_library_path within Spack regardles…

a3ee9e6

…s of environment

bugfix: work if no LD_LIBRARY_PATH

a26ccc9

Refactor thread pool handling separate from compiler detection

eed4648

rename cray-related compiler module names

7d08117

Search modules for PATH modifications, include in path hints

74124fb

find_compilers: when path_hints provided only search modules that hav…

7210f4d

…e those paths

remove special handling for modules in linux

c56beb4

flake

b360dc3

becker33 force-pushed the features/find-linux-compilers-by-module branch from 8105269 to 031cad8 Compare July 17, 2019 15:15

sort identical compilers by likelihood we have the correct module

b37f06c

becker33 force-pushed the features/find-linux-compilers-by-module branch 3 times, most recently from ccba7f1 to 77497ea Compare July 17, 2019 20:22

becker33 closed this Jul 18, 2019

becker33 reopened this Jul 18, 2019

cleanup

87dbae3

becker33 force-pushed the features/find-linux-compilers-by-module branch from 77497ea to 87dbae3 Compare July 18, 2019 17:55

becker33 removed the WIP label Jul 18, 2019

becker33 assigned tgamblin Jul 18, 2019

becker33 force-pushed the features/find-linux-compilers-by-module branch from 7ac611b to b5477aa Compare July 18, 2019 20:16

cleanup

ba00585

becker33 force-pushed the features/find-linux-compilers-by-module branch from b5477aa to ba00585 Compare July 18, 2019 20:17

tgamblin requested changes Jul 21, 2019

View reviewed changes

address review of compiler finding logic

bfd80de

module_cmd: comments for LD_LIBRARY_PATH/sys.executable hackery

c5a03c8

tgamblin requested changes Jul 23, 2019

View reviewed changes

becker33 added 2 commits July 24, 2019 15:33

compiler detection: bugfix sort by module suitability

a236b7c

compiler modules: do not use modules for other compilers for gcc

92e6b4e

tgamblin approved these changes Jul 27, 2019

View reviewed changes

becker33 mentioned this pull request Sep 20, 2019

Difference in behavior between running configure with Spacks build-env and how Spack configures software #12888

Closed

tgamblin mentioned this pull request Jan 21, 2020

bugfix: make _source_single_file work in venvs #14569

Merged

becker33 commented Jan 21, 2020

View reviewed changes

becker33 mentioned this pull request May 27, 2020

spack module_cmd: set LD_LIBRARY_PATH for python in subshell. #16827

Merged

adamjstewart added compilers linux modules labels Aug 22, 2020

alalazo added the revisit label Dec 20, 2021

becker33 closed this Nov 4, 2022

haampie deleted the features/find-linux-compilers-by-module branch February 20, 2023 18:16

Conversation

becker33 commented Jul 11, 2019

Uh oh!

becker33 commented Jul 11, 2019

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

becker33 commented Jul 24, 2019

Uh oh!

adrienbernede commented Aug 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mwkrentel commented Sep 29, 2019

Uh oh!

adrienbernede commented Sep 30, 2019

Uh oh!

mwkrentel commented Sep 30, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alalazo commented Dec 17, 2021

Uh oh!

mwkrentel commented Dec 20, 2021

Uh oh!

becker33 commented Nov 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

adrienbernede commented Aug 26, 2019 •

edited

Loading