prune: Better handling of eventually consistent backend#2685
Closed
MichaelEischer wants to merge 3 commits intorestic:masterfrom
Closed
prune: Better handling of eventually consistent backend#2685MichaelEischer wants to merge 3 commits intorestic:masterfrom
MichaelEischer wants to merge 3 commits intorestic:masterfrom
Conversation
prune listed all pack files again after repacking the packs. This can be a problem on eventual consistent backends where new files may show up with some delay. This could cause the index rebuild to miss a few packs. This change lets the prune command keep track of existing, removed and newly created packs and specifically rebuild the index with these files. This guarantees that the index will contain all relevant pack files.
7 tasks
11 tasks
8 tasks
Member
Author
|
This PR is obsolete now that #2842 is merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of this change? What does it change?
The prune command rebuilds the repository index immediately after repacking unused packs. Rebuilding the index currently requires listing all packs in the repository. For an eventually consistent backend this can be a problem, as files may show up with some delay.
This PR lets the prune command itself keep track of all packs that should be contained in the rebuilt index. This side-steps the problems associated with listing the pack files.
Remaining design issues:
selectedPacksRepo, which wraps a normal repository object but just returns the packs to be included in the new index, feels a bit like a hack. I've also made an attempt to explicitly pass in the list of packs to include in the index, however that resulted in special cases all the way betweenrunPruneandindex/index.New.packHookwhich directly reaches into thepacker_manager.savePacker. While this works it seems rather hacky, to directly reach into the repository object like that.map[restic.ID]int64used to map packs to their size, could probably use a proper type definition.#2674 is a prerequisite to use this PR safely.
Was the change discussed in an issue or in the forum before?
No, but there have been some reports in the forum e.g. in https://forum.restic.net/t/fail-restic-stats-mode-raw-data/2601/16 .
Checklist
[ ] I have added documentation for the changes (in the manual)changelog/unreleased/that describes the changes for our users (template here)gofmton the code in all commits