backup: verify blobs before upload#4681
Merged
MichaelEischer merged 13 commits intorestic:masterfrom Feb 4, 2024
Merged
Conversation
8 tasks
This only covers the blobs themselves, the pack header is not verified so far. Unpacked files are also not covered by the integrity check.
Some tests have to explicitly create pack files with blobs that don't match their ID. For those blobs the builtin verification of the repository must be disabled.
098562c to
29c2575
Compare
Verifying all blobs before upload comes with a notable performance impact. Allow users to skip it if necessary.
29c2575 to
c32e5e2
Compare
72b23f8 to
86b38a0
Compare
d5f7d4a to
5957417
Compare
This was referenced Feb 4, 2024
Closed
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR change? What problem does it solve?
The repository struct now verifies blobs before uploading them. This systematically prevents corrupted blobs like in #4677 from being stored in the repository. Thereby, backups gain protection from random bitflips that causes data corruption or bugs in the encryption/compression libraries.
The verification now covers blobs, the pack header as well as unpacked files.
Was the change previously discussed in an issue or on the forum?
Fixes #4529 .
Checklist
changelog/unreleased/that describes the changes for our users (see template).gofmton the code in all commits.