This is breaking because of #22613, which I am working on.

@scottwittenburg it occurred to me that as a workaround, you can concretize with clingo but set the concretizer to the original one in the build jobs. You don't actually need to concretize in the build jobs, right? You just need to be able to refer to an already-concrete hash. So the logic after lockfile creation should be exactly the same as the old system; you should just use the new concretizer at generation time.

We can fix this in a better way after #22613 is merged, or maybe before -- I think we could tweak the concretize() method to notice when everything that needs concretizing is already concrete, and skip bootstrapping and concretization altogether (which is probably better for CI).

@tgamblin I tried your suggestion, thanks. I think it exposed the real problem, which was just a mistake on my part. I was using a new runner image with a different compiler installed, but hadn't updated the os/compiler information in the stack. So what I expect happened was that spack got started installing missing compilers, and hence needed to concretize stuff. Otherwise, I'm confused where concretization would have been needed in the rebuild jobs.

In fact, the radiuss stack is already using the clingo concretizer, and did not need to reset the concretizer for the rebuild jobs (at least not that I can see, but @tldahlgren probably knows for sure).

So now some stages of build jobs completed successfully, but eventually some failed. Running the reproducer locally for one of those, it seems that the spack.lock contains two instances of numactl with different build hashes (the key used for spack.lock entries), but with the same DAG hash. So the spack install ... command fails with:

SpackEnvironmentError: numactl@2.0.14%gcc@9.3.0 blahblahblah matches multiple specs in the environment ...:

Dependency spec
  vw2cdve  numactl@2.0.14%gcc@9.3.0 ...
Dependency spec
  vw2cdve  numactl@2.0.14%gcc@9.3.0 ...

@scottwittenburg I think this is closed by #25502 ? Please reopen if I'm incorrect.