Summary

Right now, external specs are allowed to have missing pieces of information, and are "completed" during concretization with the bits that produce the best optimization values in clingo. This completion:

1. Is done regardless of what is really installed on a system, and
2. Creates a one (installed software) to many (concrete specs) mapping for externals

which is obviously wrong.

For Spack v1.1 we want to treat external specs as any other concrete spec. This means that each external spec will have a single, well-defined hash, a single set of immutable variants, etc External specs may also have dependencies, and properties along edges.

Rationale

This feature will solve issues where e.g. a missing variant is concretized differently for the same external, depending on the opportunity for optimizations. It would also solve any issue that arise because we trim the dependencies of externals.

From a maintenance point of view, it allows us to remove the code path for external specs in the solver, and consolidate it with the one for "reused" specs.

Description

Syntax to specify externals with dependencies

The syntax used to specify externals with dependencies is an extension of the current schema:

# Specification for the following DAG:
#
# o ascent@0.9.2
# |\
# | o adios2@2.7.1
# |/
# o bzip2@1.0.8
packages:
  ascent:
    externals:
    - spec: "ascent@0.9.2+adios2+shared"
      prefix: /user/path
      external_id: <ascent_id>
      dependencies: 
      - external_id: <adios2_id>
        deptypes: link
      - external_id: <bzip2_id>
        deptypes: "run"
  adios2:
    externals:
    - spec: "adios2@2.7.1+shared"
      prefix: /user/path
      external_id: <adios2_id>
      dependencies: 
      - external_id: <bzip2_id>
        deptypes: ["build", "link"]
  bzip2:
    externals:
    - spec: "bzip2@1.0.8+shared"
      prefix: /user/path
      external_id: <bzip2_id>

The extensions are the following:

• Each node being used as a dependency must have a unique external_id identifying it, and
• Each external node may specify dependencies, by their external_id, and by additional optional attributes that define the edge properties.

Complete missing information on an external node

To complete the pieces that are missing on each node (e.g. variants that are not specified), Spack will provide different strategies. Users will be able to select one strategy via the concretizer.yaml configuration. A tentative example of where to put that option might be:

concretizer:
  externals:
    completion: architecture_only|default_variants

The two strategies that will be implemented as a first cut are:

• Complete only the architecture information and don't add missing variants, or
• Complete architecture information and add default values for missing variants

Backward compatibility

External entries like mpich %gcc will be deprecated, since in the new model the mpich entry needs to point to a specific gcc external node.

To maintain backward compatibility in a case like the one above, Spack will:

1. Guess which gcc is the correct dependency for the mpich above, and
2. Issue a warning if the guess produced a valid result, informing the user of which gcc was selected, or error out if no gcc is available

Additional information

On top of this modification, we may want to extend the detection logic to automatically find external dependencies.

General information

• I have searched the issues of this repo and believe this is not a duplicate