Skip to content

Support keeping old refs around after the layout change#17

Merged
scottwittenburg merged 5 commits intospack:mainfrom
scottwittenburg:avoid-unnecessary-processing
Feb 13, 2025
Merged

Support keeping old refs around after the layout change#17
scottwittenburg merged 5 commits intospack:mainfrom
scottwittenburg:avoid-unnecessary-processing

Conversation

@scottwittenburg
Copy link
Copy Markdown
Collaborator

The goal of this PR is to prepare for an upcoming change to spack buildcache layout. Rather than have this repo maintain code to sift through buildcaches of two different layouts, the idea is to get it to stop re-processing the same immutable stacks on a daily basis, ahead of making the change to support the new buildcache layout. This will serve two purposes:

  1. Allow the website to maintain listings for old spack refs that are unchanging anyway (tagged releases and develop snapshots), both now and after we update spack and the cache indexer to the new buildcache layout.
  2. Avoid a lot of unnecessary fetching of spec files after the layout update. Once we update to the new buildcache layout, we can no longer directly compute the url to a tarball from just the concrete spec and mirror url. Instead, it will at that point be necessary to retrieve the spec metadata (.spec.json.sig or spec.json) file for each spec, in order to read the tarball checksum first. In contrast, currently the website generation process only needs to fetch the buildcache index of each top-level or stack-specific mirror, so the layout change will significantly increase the number of files we have to fetch per-ref.

This change is accomplished by introducing a MUTABLE_REFS list, and avoiding processing for any refs that already exist in the repo and do not appear in that list. In order to avoid losing information, we must now load some of the existing information from cache/ and _data/ in order to preserve data for refs we skip processing.

One consequence of this change is that removal of old tags/refs will no longer be sufficient to remove the corresponding material from the website. Instead we can do the following when we are ready to remove the old refs that still have the old buildcache layout:

  • merge a PR here to remove data associated with those refs from _cache/, _data/, and pages/tags/
  • remove those refs from the mirror bucket
  • remove (or regenerate) the cache_spack_io_index.json from the root of the bucket

We can merge this now or soon and keep an eye on the github actions to make sure changes to develop and new snapshots are properly processed, while old refs are left untouched. Then we will need a subsequent PR to handle the layout change, to be timed appropriately with merging the spack PR linked above.

@scottwittenburg
Copy link
Copy Markdown
Collaborator Author

Is github broken? I can't seem to add reviewers, so @zackgalbreath, here is my request for your review 😁

@scottwittenburg scottwittenburg merged commit a7b0047 into spack:main Feb 13, 2025
@scottwittenburg scottwittenburg deleted the avoid-unnecessary-processing branch February 13, 2025 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants