Allow reusing specs built with compilers not in config by alalazo · Pull Request #43539 · spack/spack

alalazo · 2024-04-08T21:04:17Z

This PR allows to reuse specs that were built with compilers that are not in the current configuration. This means that specs from buildcaches don't need to have a matching compiler locally to be reused. If a node needs to be built, only available compilers will be considered as candidates.

Below most of the specs are reused and have %gcc@=9.4.0 which is not in the configuration. The root node is built and has %gcc@=11.1.0:

This limits the implicit use of globals, and can be leveraged to have simpler unit-tests.

This commit allows reusing binaries with compilers that don't match any in the local configuration.

This is needed if we want to simulate x86_64 preferences on aarch64 host when running unit tests

alalazo · 2024-04-09T14:52:11Z

The usual performance benchmark for this test seems ok:

radiuss.develop.csv
radiuss.pr.csv
radiuss.txt

haampie · 2024-04-10T06:48:52Z

lib/spack/spack/build_environment.py

-    module.spack_cxx = os.path.join(link_dir, pkg.compiler.link_paths["cxx"])
-    module.spack_f77 = os.path.join(link_dir, pkg.compiler.link_paths["f77"])
-    module.spack_fc = os.path.join(link_dir, pkg.compiler.link_paths["fc"])
+    try:


This looks a bit too generic. Can't you catch a specific error? Maybe refactor to compiler = get_compiler(pkg) and then if compiler: module.spack_cc = ... etc?

Per DM, we need to check how to deal with mpi implementations, and similar specs that have a real run dependency on the compiler.

4d117a5 and acca63b

haampie · 2024-04-10T06:51:23Z

lib/spack/spack/test/concretize.py

+    @pytest.mark.only_clingo("Old concretizer cannot reuse")
+    def test_reuse_specs_from_non_available_compilers(self, mutable_config, mutable_database):
+        """Tests that we can reuse specs with compilers that are not configured locally."""
+        # All the specs in the mutable DB have been compiled with %gcc@=10.2.1


maybe also assert this? mutable_database.query_local(...)?

haampie · 2024-04-10T06:52:14Z

lib/spack/spack/solver/asp.py

        return False


+class CompilerParser:


Parser? 🤔

I'd leave this as is, at least here. There is also a RequirementParser - open to change both names in a following Pr.

haampie · 2024-04-10T06:53:50Z

lib/spack/spack/solver/asp.py

+        result = sorted(self.compilers, key=lambda x: (x.spec.name, x.spec.version), reverse=True)
+        # Then stable sort to prefer available compilers and account for preferences
+        ppk = spack.package_prefs.PackagePrefs("all", "compiler", all=False)
+        result = sorted(result, key=lambda x: (not x.available, ppk(x.spec)))


avoid a copy result.sort(...)

lib/spack/spack/solver/asp.py

This would happen if the mpi was installed from a buildcache, and the compiler is not available.

alalazo · 2024-04-11T06:39:44Z

@spackbot run pipeline

spackbot-app · 2024-04-11T06:39:51Z

I've started that pipeline for you!

Allow reuse of specs that were built with compilers not in the current configuration. This means that specs from build caches don't need to have a matching compiler locally to be reused. Similarly when updating a distro. If a node needs to be built, only available compilers will be considered as candidates.

The full error: ==> Installing hdf5-1.14.3-ohtopgtkb3nmwyxxgqdpwjvvnttospev [24/35] ==> No binary for hdf5-1.14.3-ohtopgtkb3nmwyxxgqdpwjvvnttospev found: installing from source ==> Error: AttributeError: module 'spack.pkg.builtin.hdf5' has no attribute 'spack_cc' The 'hdf5' package cannot find an attribute while trying to build from sources. This might be due to a change in Spack's package format to support multiple build-systems for a single package. You can fix this by updating the build recipe, and you can also report the issue as a bug. More information at https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-procedure C:\Users\dan.lipsa\projects\spack\var\spack\repos\builtin\packages\msmpi\package.py:48, in setup_dependent_package: 45 # be manually supplied to compiler by consuming package 46 # Note: This is not typical of MPI installations 47 dependent_module = dependent_spec.package.module >> 48 self.spec.mpicc = dependent_module.spack_cc 49 self.spec.mpicxx = dependent_module.spack_cxx 50 self.spec.mpifc = dependent_module.spack_fc 51 self.spec.mpif77 = dependent_module.spack_f77 This reverts changes to msmpi/package.py 1fe8e63 * Reuse specs built with compilers not in config (spack#43539)

Allow reuse of specs that were built with compilers not in the current configuration. This means that specs from build caches don't need to have a matching compiler locally to be reused. Similarly when updating a distro. If a node needs to be built, only available compilers will be considered as candidates.

Inject a Configuration object when retrieving compilers

0b1e74c

This limits the implicit use of globals, and can be leveraged to have simpler unit-tests.

spackbot-app bot added build-environment commands compilers core PR affects Spack core functionality environments stand-alone-tests Stand-alone (or smoke) tests for installed packages tests General test capability(ies) labels Apr 8, 2024

Allow reusing non-local compilers

c2d3359

This commit allows reusing binaries with compilers that don't match any in the local configuration.

alalazo force-pushed the solver/reuse-non-local-compilers branch from 8d1385e to c2d3359 Compare April 8, 2024 21:30

alalazo added this to the v0.22.0 milestone Apr 9, 2024

Always monkeypatch cpu.host()

49d61c7

This is needed if we want to simulate x86_64 preferences on aarch64 host when running unit tests

alalazo force-pushed the solver/reuse-non-local-compilers branch from 70efd09 to a679dcf Compare April 9, 2024 14:35

Add unit test for the compiler parser

d91e076

alalazo force-pushed the solver/reuse-non-local-compilers branch from a679dcf to d91e076 Compare April 9, 2024 15:02

haampie reviewed Apr 10, 2024

View reviewed changes

lib/spack/spack/solver/asp.py Show resolved Hide resolved

alalazo added 4 commits April 10, 2024 16:19

Use a more specific exception when a compiler is not available

4d117a5

Assert that mock specs are compiled with gcc@10.2.1

d4bf487

Avoid a copy on return value

780bfcc

Shorten the checks for reused specs

4364302

spackbot-app bot added intel update-package labels Apr 10, 2024

Fix mpi packages to account for possibly missing compilers

7525e7a

This would happen if the mpi was installed from a buildcache, and the compiler is not available.

alalazo force-pushed the solver/reuse-non-local-compilers branch from acca63b to 7525e7a Compare April 10, 2024 15:41

haampie approved these changes Apr 11, 2024

View reviewed changes

haampie merged commit 1fe8e63 into spack:develop Apr 11, 2024

alalazo deleted the solver/reuse-non-local-compilers branch April 11, 2024 07:24

greenc-FNAL mentioned this pull request Apr 24, 2024

Merge from upstream 2024-04-24 FNALssi/spack#332

Merged

danlipsa mentioned this pull request May 22, 2024

msmpi tries to access dependent package hdf5 spack_cc #44327

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow reusing specs built with compilers not in config#43539

Allow reusing specs built with compilers not in config#43539
haampie merged 9 commits intospack:developfrom
alalazo:solver/reuse-non-local-compilers

alalazo commented Apr 8, 2024

Uh oh!

alalazo commented Apr 9, 2024

Uh oh!

haampie Apr 10, 2024

Uh oh!

alalazo Apr 10, 2024

Uh oh!

alalazo Apr 10, 2024

Uh oh!

haampie Apr 10, 2024

Uh oh!

alalazo Apr 10, 2024

Uh oh!

haampie Apr 10, 2024 •

edited

Loading

Uh oh!

alalazo Apr 10, 2024

Uh oh!

haampie Apr 10, 2024

Uh oh!

alalazo Apr 10, 2024

Uh oh!

Uh oh!

alalazo commented Apr 11, 2024

Uh oh!

spackbot-app bot commented Apr 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

alalazo commented Apr 8, 2024

Uh oh!

alalazo commented Apr 9, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haampie Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alalazo commented Apr 11, 2024

Uh oh!

spackbot-app bot commented Apr 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

haampie Apr 10, 2024 •

edited

Loading