Skip to content

ddl: Fix the storage instance may be physically dropped when the region is not removed (#8721)#8724

Merged
ti-chi-bot[bot] merged 5 commits intopingcap:release-7.5from
ti-chi-bot:cherry-pick-8721-to-release-7.5
Jan 26, 2024
Merged

ddl: Fix the storage instance may be physically dropped when the region is not removed (#8721)#8724
ti-chi-bot[bot] merged 5 commits intopingcap:release-7.5from
ti-chi-bot:cherry-pick-8721-to-release-7.5

Conversation

@ti-chi-bot
Copy link
Member

@ti-chi-bot ti-chi-bot commented Jan 24, 2024

This is an automated cherry-pick of #8721

What problem does this PR solve?

Issue Number: close #8710

Problem Summary:
There are some case that a region may not be removed from TiFlash when there are unhealth peers rule_checker @ pd. However, after the tiflash replica is set to zero, the data still be physically dropped after gc safepoint exceed.

What is changed and how it works?

  • Prevent the storage instance being physically dropped when the region is not removed
    * Throw exception when the storage instance is not exist after schema synced
  • Do not physically remove IStorage instance when the replica is set to 0
  • Add the reason why an IStorage instance is recovered ddl: Add the reason why an IStorage instance is recovered #8735

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Fix the issue that the tiflash replica data may be corrupted after setting the tiflash replica to 0 and add it back later

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-7.5 This PR is cherry-picked to release-7.5 from a source PR. labels Jan 24, 2024
@ti-chi-bot ti-chi-bot added the cherry-pick-approved Cherry pick PR approved by release team. label Jan 24, 2024
@ti-chi-bot ti-chi-bot force-pushed the cherry-pick-8721-to-release-7.5 branch from c327a00 to b23daac Compare January 24, 2024 01:49
@JaySon-Huang
Copy link
Contributor

/run-all-tests

1 similar comment
@JaySon-Huang
Copy link
Contributor

/run-all-tests

@JaySon-Huang JaySon-Huang force-pushed the cherry-pick-8721-to-release-7.5 branch from b655d72 to 2c41f99 Compare January 24, 2024 17:26
@JaySon-Huang
Copy link
Contributor

/run-all-tests

@JaySon-Huang
Copy link
Contributor

/hold

@ti-chi-bot ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 25, 2024
@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jan 26, 2024
@JaySon-Huang JaySon-Huang force-pushed the cherry-pick-8721-to-release-7.5 branch from f5efc1f to 1f77eda Compare January 26, 2024 06:51
@JaySon-Huang
Copy link
Contributor

/run-all-tests

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 26, 2024

@Lloyd-Pottiger: Your lgtm message is repeated, so it is ignored.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Comment on lines +401 to +403
updateTiFlashReplicaNumOnStorage(database_id, table_id, storage, table_info);
// FIXME: Do not drop the table under release-7.5 branch, need more testing
// applyDropTable(database_id, table_id, "SetTiFlashReplica-0");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the tiflash replica is set to 0, do not drop the IStorage instance under release-7.5 branch

@JaySon-Huang
Copy link
Contributor

[2024-01-26T07:25:38.553Z] [2024/01/26 15:24:58.892 +08:00] [FATAL] [Exception.cpp:106] ["Poco::Exception. Code: 1000, e.code() = 2, e.displayText() = Exception: Receive TsoResponse failed, e.what() = Exception"] [source="DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap *, DB::WriteCmdsView, DB::RaftCmdHeader)"] [thread_id=430]

@JaySon-Huang
Copy link
Contributor

/run-integration-test

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 26, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [JaySon-Huang,Lloyd-Pottiger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 26, 2024
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 26, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-01-26 02:49:16.351583459 +0000 UTC m=+1101797.915881166: ☑️ agreed by Lloyd-Pottiger.
  • 2024-01-26 08:36:45.811303994 +0000 UTC m=+1122647.375601699: ☑️ agreed by JaySon-Huang.

@JaySon-Huang
Copy link
Contributor

/unhold

@ti-chi-bot ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 26, 2024
@ti-chi-bot ti-chi-bot bot merged commit 70ef408 into pingcap:release-7.5 Jan 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved cherry-pick-approved Cherry pick PR approved by release team. lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-7.5 This PR is cherry-picked to release-7.5 from a source PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants