backupccl: reintroduce previously offline tables with manifest.DescriptorChanges#89981
Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom Oct 20, 2022
Merged
Conversation
Member
|
This change is |
a61551e to
f23ce37
Compare
Collaborator
Author
|
this is a trivial forward port of the 22.2 patch to master. just needs a stamp! |
stevendanna
approved these changes
Oct 19, 2022
Collaborator
Author
|
TFTR! bors r=stevendanna |
Contributor
|
Build failed: |
Collaborator
Author
|
bor r- |
Collaborator
Author
|
bors r- |
…ptorChanges getReintroducedSpans finds all tables included in the current and previous backup that may have undergone a non-mvcc operation. The current backup will then back up these tables' spans from ts = 0, as the previous backup may have missed certain non-mvcc operations (e.g. ClearRange, non-mvcc AddSSTable). To find these tables, getReintroducedSpans must find all tables covered in the previous backup that were in the offline state at previous backup start time. This function assumed that the previous backup's manifest.Descriptors field would contain all tables covered in the previous backup; however, while investigating cockroachdb#88042, we discovered that this assumption is not correct in pre 22.2 branches. During revision history backups, a table with an in-progress import (e.g. offline at backup time) can get backed up and included in manifest.DescriptorChanges but not in manifest.Descriptors. This implies getReintroducedSpans missed reintroducing spans from this table, implying that backup chains have missed backing up some data. It's worth noting this bug could only affect a backup chain in 22.2 iff the backup that observed the table go offline was taken on a 22.1 cluster. Because 22.2 backups explicitly back up all offline tables, all tables in manifest.DescriptorChanges should also be in manifest.Descriptors. In other words, before this patch, the following sequence could lead to a corrupt backup chain: - t0: begin IMPORT on foo, on a pre 22.2 cluster - t1: conduct a revision history backup of foo - t2: complete or cancel the import of foo, via a non-mvcc operation - t3: upgrade the cluster to 22.2 - t4: conduct a revision history incremental backup of foo - foo's span will not get re-introduced This patch fixes getReintroducedSpans to ensure it reintroduces tables included in manifest.DescriptorChanges whose last revision brought the table offline. In addition, this patch adds a check to restore planning which causes the restore to fail immedidiately if any restore target is missing an introduced span. The patch tests this check on a corrupt backup taken on v21.2. Release note(bug fix): fix apart of Tech Advisory https://cockroachlabs.atlassian.net/browse/TSE-198 Release justification: bug fix
f23ce37 to
d80f505
Compare
Collaborator
Author
|
bors r=adityamaru |
Contributor
|
Build succeeded: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
getReintroducedSpans finds all tables included in the current and previous
backup that may have undergone a non-mvcc operation. The current backup will
then back up these tables' spans from ts = 0, as the previous backup may have
missed certain non-mvcc operations (e.g. ClearRange, non-mvcc AddSSTable).
To find these tables, getReintroducedSpans must find all tables covered in the
previous backup that were in the offline state at previous backup start time.
This function assumed that the previous backup's
manifest.Descriptors field would contain all tables covered in the previous
backup; however, while investigating #88042, we discovered that this assumption
is not correct in pre 22.2 branches. During revision history backups, a table
with an in-progress import (e.g. offline at backup time) can get backed up and
included in manifest.DescriptorChanges but not in manifest.Descriptors. This
implies getReintroducedSpans missed reintroducing spans from this table,
implying that backup chains have missed backing up some data.
It's worth noting this bug could only affect a backup chain post 22.1 iff the
backup that observed the table go offline was taken on a 22.1
cluster. Because post 22.1 backups explicitly back up all offline tables, all tables
in manifest.DescriptorChanges should also be in manifest.Descriptors. In other
words, before this patch, the following sequence could lead to a corrupt backup
chain:
This patch fixes getReintroducedSpans to ensure it reintroduces tables included
in manifest.DescriptorChanges whose last revision brought the table offline. In
addition, this patch adds a check to restore planning which causes the restore
to fail immedidiately if any restore target is missing an introduced span. The
patch tests this check on a corrupt backup taken on v21.2.
Fixes #88042
Release note(bug fix): fix apart of Tech Advisory
https://cockroachlabs.atlassian.net/browse/TSE-198