Support keeping old refs around after the layout change#17
Merged
scottwittenburg merged 5 commits intospack:mainfrom Feb 13, 2025
Merged
Conversation
Collaborator
Author
|
Is github broken? I can't seem to add reviewers, so @zackgalbreath, here is my request for your review 😁 |
zackgalbreath
approved these changes
Feb 5, 2025
21 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The goal of this PR is to prepare for an upcoming change to spack buildcache layout. Rather than have this repo maintain code to sift through buildcaches of two different layouts, the idea is to get it to stop re-processing the same immutable stacks on a daily basis, ahead of making the change to support the new buildcache layout. This will serve two purposes:
.spec.json.sigorspec.json) file for each spec, in order to read the tarball checksum first. In contrast, currently the website generation process only needs to fetch the buildcache index of each top-level or stack-specific mirror, so the layout change will significantly increase the number of files we have to fetch per-ref.This change is accomplished by introducing a
MUTABLE_REFSlist, and avoiding processing for any refs that already exist in the repo and do not appear in that list. In order to avoid losing information, we must now load some of the existing information fromcache/and_data/in order to preserve data for refs we skip processing.One consequence of this change is that removal of old tags/refs will no longer be sufficient to remove the corresponding material from the website. Instead we can do the following when we are ready to remove the old refs that still have the old buildcache layout:
_cache/,_data/, andpages/tags/cache_spack_io_index.jsonfrom the root of the bucketWe can merge this now or soon and keep an eye on the github actions to make sure changes to
developand new snapshots are properly processed, while old refs are left untouched. Then we will need a subsequent PR to handle the layout change, to be timed appropriately with merging the spack PR linked above.