Skip to content

restore: skip_localities_check no longer works#101682

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
fqazi:restoreLocalityBug
Apr 18, 2023
Merged

restore: skip_localities_check no longer works#101682
craig[bot] merged 1 commit intocockroachdb:masterfrom
fqazi:restoreLocalityBug

Conversation

@fqazi
Copy link
Copy Markdown
Collaborator

@fqazi fqazi commented Apr 17, 2023

Previously, there was no additional validation for generating zone configurations so if regions were
missing during the process no errors would be generated. Later on the generation process had validation added to validate that all regions reference existed within the zone configuration. To address this, this patch adds an option to skip validation of regions when
generating zone configurations.

Fixes: #100913
Release note (bug fix): Previously, RESTORE with skip_localities_check would still fail with errors if regions were missing on a cluster.

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@fqazi fqazi added backport-22.2.x backport-23.1.x PAST MAINTENANCE SUPPORT: 23.1 patch releases via ER request only labels Apr 17, 2023
@fqazi fqazi force-pushed the restoreLocalityBug branch from 4423664 to dcc0404 Compare April 17, 2023 20:15
@fqazi fqazi marked this pull request as ready for review April 17, 2023 20:54
@fqazi fqazi requested review from a team as code owners April 17, 2023 20:54
@fqazi fqazi requested review from a team and msbutler and removed request for a team April 17, 2023 20:54
Copy link
Copy Markdown
Contributor

@chengxiong-ruan chengxiong-ruan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice~

Copy link
Copy Markdown
Collaborator

@msbutler msbutler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could you clean up the commit title to: "fix skip_localities_check"? Also, i think you could clarify in the commit message that a restore on a cluster with mismatched localities was failing even when the the skip_localities check was passed.

The restore_job and jobs.proto changes lgtm though!

pq: detected a mismatch in regions between the restore cluster and the backup cluster, missing regions detected: us-east-1, us-west-1.
HINT: there are two ways you can resolve this issue: 1) update the cluster to which you're restoring to ensure that the regions present on the nodes' --locality flags match those present in the backup image, or 2) restore with the "skip_localities_check" option

exec-sql
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I'd run the cluster restore first, then the database restore with the new_db_name option. Else, the cluster restore may flake because it detects database that's in the process of getting dropped, even after the DROP DATABASE command returns.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point let me change this

@fqazi
Copy link
Copy Markdown
Collaborator Author

fqazi commented Apr 18, 2023

pkg/ccl/backupccl/testdata/backup-restore/multiregion

Updated this to:

    restore: fix skip_localities_check

    Previously, there was no additional validation for
    generating zone configurations during restore, so nothing 
    bad would happen if regions were missing during restore.
    Later on, the zone config generation process had validation
    to ensure all regions existed. This was problematic for
    restores with skip_localities_check since we would still
    incorrectly check localities.
    To address this, this patch adds an option to skip validation
    of regions when generating zone configurations.

    Fixes: #100913

    Release note (bug fix): Previously, RESTORE with skip_localities_check
    would still fail with errors if regions were missing on a cluster.

Previously, there was no additional validation for
generating zone configurations during restore, so nothing
bad would happen if regions were missing during restore.
Later on, the zone config generation process had validation
to ensure all regions existed. This was problematic for
restores with skip_localities_check since we would still
incorrectly check localities.
To address this, this patch adds an option to skip validation
of regions when generating zone configurations.

Fixes: cockroachdb#100913

Release note (bug fix): Previously, RESTORE with skip_localities_check
would still fail with errors if regions were missing on a cluster.
@fqazi fqazi force-pushed the restoreLocalityBug branch from dcc0404 to ddbac17 Compare April 18, 2023 20:21
@fqazi
Copy link
Copy Markdown
Collaborator Author

fqazi commented Apr 18, 2023

TFTR @msbutler @chengxiong-ruan

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Apr 18, 2023

Build succeeded:

@blathers-crl
Copy link
Copy Markdown

blathers-crl bot commented Apr 18, 2023

Encountered an error creating backports. Some common things that can go wrong:

  1. The backport branch might have already existed.
  2. There was a merge conflict.
  3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.


error creating merge commit from ddbac17 to blathers/backport-release-22.2-101682: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.2.x failed. See errors above.


🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-23.1.x PAST MAINTENANCE SUPPORT: 23.1 patch releases via ER request only

Projects

None yet

Development

Successfully merging this pull request may close these issues.

restore: skip_localities_check no longer works in v22.2

4 participants