Skip to content

ddl: Fix the storage instance may be physically dropped when the region is not removed#8721

Merged
ti-chi-bot[bot] merged 10 commits intopingcap:masterfrom
JaySon-Huang:fix_inconsis
Jan 23, 2024
Merged

ddl: Fix the storage instance may be physically dropped when the region is not removed#8721
ti-chi-bot[bot] merged 10 commits intopingcap:masterfrom
JaySon-Huang:fix_inconsis

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Jan 22, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #8710

Problem Summary:
There are some case that a region may not be removed from TiFlash when there are unhealth peers rule_checker @ pd. However, after the tiflash replica is set to zero, the data still be physically dropped after gc safepoint exceed.

What is changed and how it works?

  • Prevent the storage instance being physically dropped when the region is not removed
    * Throw exception when the storage instance is not exist after schema synced

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Fix the issue that the tiflash replica data may be corrupted after setting the tiflash replica to 0 and add it back later

@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 22, 2024
@JaySon-Huang JaySon-Huang self-assigned this Jan 22, 2024
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jan 22, 2024

/run-all-tests

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jan 22, 2024
hongyunyan
hongyunyan approved these changes Jan 22, 2024
View reviewed changes
Copy link
Contributor

@hongyunyan hongyunyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 22, 2024
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 22, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hongyunyan, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [Lloyd-Pottiger,hongyunyan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 22, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-01-22 12:20:44.856149059 +0000 UTC m=+790486.420446763: ☑️ agreed by Lloyd-Pottiger.
  • 2024-01-22 14:11:46.532058325 +0000 UTC m=+797148.096356029: ☑️ agreed by hongyunyan.

@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jan 23, 2024

/run-all-tests

@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jan 23, 2024

/hold

@ti-chi-bot ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 23, 2024
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jan 23, 2024

/run-all-tests

@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jan 23, 2024

/run-unit-test

@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 23, 2024
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jan 23, 2024

/run-all-tests

@JaySon-Huang JaySon-Huang removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 23, 2024
@ti-chi-bot ti-chi-bot bot merged commit 6058d19 into pingcap:master Jan 23, 2024
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jan 23, 2024

/unhold

@JaySon-Huang JaySon-Huang deleted the fix_inconsis branch January 23, 2024 17:18
@ti-chi-bot ti-chi-bot added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Jan 24, 2024
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jan 24, 2024

/cherry-pick release-7.5

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Jan 24, 2024
@JaySon-Huang @ti-chi-bot
This is an automated cherry-pick of pingcap#8721
c327a00 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Jan 24, 2024
Merged
12 tasks
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jan 24, 2024

In response to a cherrypick label: new pull request created to branch release-7.5: #8724.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jan 24, 2024

@ti-chi-bot: new pull request could not be created: failed to create pull request against pingcap/tiflash#release-7.5 from head ti-chi-bot:cherry-pick-8721-to-release-7.5: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for ti-chi-bot:cherry-pick-8721-to-release-7.5."}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request"}

Details

In response to this:

/cherry-pick release-7.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Jan 24, 2024
@JaySon-Huang @ti-chi-bot
This is an automated cherry-pick of pingcap#8721
b23daac 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
JaySon-Huang added a commit to ti-chi-bot/tiflash that referenced this pull request Jan 26, 2024
@JaySon-Huang
This is an automated cherry-pick of pingcap#8721
fc0c4e1 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot bot pushed a commit that referenced this pull request Jan 26, 2024
@ti-chi-bot
ddl: Fix the storage instance may be physically dropped when the regi…
70ef408 
…on is not removed (#8721) (#8724)

close #8710
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Feb 27, 2024 •
edited
Loading

This PR also prevent data being unexpected dropped when the region is not removed. Ref #8777

This was referenced Apr 8, 2024
Closed
Merged
@ti-chi-bot ti-chi-bot mentioned this pull request Apr 9, 2024
Merged
12 tasks
@JaySon-Huang JaySon-Huang mentioned this pull request Sep 20, 2024
Merged
12 tasks
@ti-chi-bot ti-chi-bot mentioned this pull request Jan 22, 2025
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hongyunyan hongyunyan hongyunyan approved these changes

@Lloyd-Pottiger Lloyd-Pottiger Lloyd-Pottiger approved these changes

@JinheLin JinheLin Awaiting requested review from JinheLin

Assignees

@JaySon-Huang JaySon-Huang

Labels

approved lgtm needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Data may be dropped silently after replica is set to 0 but peer not removed

4 participants

@JaySon-Huang @ti-chi-bot @hongyunyan @Lloyd-Pottiger