Problem: enabling incremental make across different versions of a package

This thought was inspired by this tweet about a surprising (to me) security implication of wiping staging directories by default in alpine linux's abuild build system: https://twitter.com/RichFelker/status/1435659133223587843?s=20. In particular, when the alpine maintainers needed to rebuild firefox in response to a zero-day, they had run up against the inherent time pressure of rebuilding a very large codebase like firefox from scratch. The tweet proposes the following workflow for security-sensitive projects with a large attack surface that take a long time to rebuild:

1. retain the stage and associated build output from the prior version of the software.
2. apply the patch to fix the zero-day to the prior stage.
3. reuse the output from the prior build to take advantage of timestamp-based make.

Application to Spack

There are two points compared/contrasted to spack that I can immediately spot from the above scenario:

• In OS-specific build systems like abuild, packages usually have a single totally-ordered version lineage, so obtaining the "patch to fix the zero-day" is often relatively easy to calculate and occasionally may be provided directly by the upstream maintainer when a new source tarball is created.
  • While spack itself does not enforce a single monolithic version lineage for each package, some spack users or sites using spack may follow that protocol unofficially. This means that the patch from one install of some package to the next one (when it is upgraded) may be relatively small, and incremental rebuilds from a preserved stage directory after applying the small patch may therefore be significantly faster than a clean rebuild.
• Since spack (largely) builds on the local machine instead of pulling down binary packages from a remote server, we have the ability to retain additional data from the build process (which alpine linux would typically only be able to do on their build servers).
  • The abuild build system and aports repository allow users to build packages from source, but this isn't done in most cases (I only do this for a few packages like the kernel on my alpine install). Similarly for spack, many sites use spack buildcache create to avoid their users having to rebuild common packages from scratch each time.

Alternative Approaches

1. spack dev-build allows maintaining an external stage of a specific package, while setting the version to allow it to be used as part of a larger spack build. This can be used with spack stage -p <PATH> to retrieve the appropriate source checkout.
   • However, this doesn't work for more than one package at a time, requires spack to be executed within that stage directory, and isn't intended to be used to produce output in the spack install db (and can't be consumed by spack install).
2. spack install --keep-stage (similarly to abuild's -k flag) will retain the stage directory after a successful installation of a package.
   • However, this isn't retained across separate versions of a package, or even the same version of a package if the hash differs.

Missed opportunity: maintaining an untouched stage directory

Many/most of the underlying build systems that the codebases spack packages use to build themselves (specifically make and things that produce Makefiles) rely on timestamps to calculate what to rebuild (this is also true of abuild). While some build systems such as pants and bazel rely on recursively checksumming directory contents to know what to rebuild, those tools do not understand Makefiles or ./configure scripts, and require forking such packages in order to insert their own build setup, which then quickly diverges from upstream and imposes a significant maintenance burden. While I proposed partially adopting their model in #20407 for some cases, it remains difficult to see how to incrementally introduce the benefits of those tools into any of the packages that spack supports.

Therefore, to avoid an immense continuing effort to rewrite the build processes of upstream packages, spack often relies on timestamps to calculate what to rebuild. This actually works great in the case of spack dev-build or spack build-env, enabling package developers to use spack to pull in any dependencies, then iterate via editing and incrementally rebuilding, without doing a whole fresh build repeatedly. However, as per above, spack dev-build doesn't plug into spack install, and spack currently can't reuse the same stage directory for a new version of a previously-built package. This means that:

1. building a new version of a package to insert into a spack buildcache requires a fresh rebuild (similarly to the situation in the linked tweet at the top).
2. installing any version of a package that's not reachable from a spack buildcache requires a fresh rebuild.

As in the linked tweet at the top, (1) may widen the time range that spack users in a specific site are exposed to a zero-day, although this isn't really an issue since spack install will just build from source if a package version isn't available from a buildcache. However, (2) means users trying out any version of a package that's not already built get to wait a long time to do so, and maybe more significantly, users doing a spack dev-build of a previously-built package still have to wait for a clean rebuild before being able to do their work. This is especially tragic if their site e.g. reserves beefy machines for building LLVM from source, but not for individual spack users (as with alpine linux and abuild/apk install). To make the point stark, none of these issues arise when people build everything from source by hand without spack, which to me implies there is an opportunity here to solve a more general problem than the one mentioned in the linked tweet.

Proposal: specify a persistent staging directory for a package to reuse across builds

1. Specify which packages should have their stage directory persisted with spack install --persist-stage=package1,package2,....
   • We may decide to make this into a config option instead.
2. For those packages, move the stage directory to the install prefix after installation instead of deleting it, and make a note of its location in ~/.spack/caches/persisted-stages/<checksum>.json.
   • A maximum of one persisted stage is retained per package to avoid unbounded disk usage, so just a single json file is retained in the cache directory, renaming the file whenever its contents change.
3. Make spack buildcache create --propagate-persisted-stages --persist-stage=package1,package2,... retain the stage directory in the install prefix for the specified packages, preserving the files' modification times.
4. Allow spack install --persist-stage=package1,package2,... to look for a persisted stage directory in a buildcache or install db for the specified packages (after looking in ~/.spack/caches/persisted-stages/<checksum>.json for those packages).
   • If a persisted stage is found, calculate the diff between spack stage of the new package vs the persisted stage (diffing over just the files that exist in the new package's stage), apply the diff, then build within the persisted stage directory.
     • If the persisted stage is located in a spack buildcache, then it is extracted to a temporary directory for the duration of the install, before the temporary directory is moved to the install prefix as in (2).
   • If a persisted stage is not found, perform (2) for that package, creating a persisted stage for it and noting that in ~/.spack/caches/persisted-stages/<checksum>.json.