Fix potential data inconsistency under heavy ddl operation#5044
Merged
lidezhu merged 3 commits intopingcap:masterfrom Jun 2, 2022
Merged
Fix potential data inconsistency under heavy ddl operation#5044lidezhu merged 3 commits intopingcap:masterfrom
lidezhu merged 3 commits intopingcap:masterfrom
Conversation
Member
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. DetailsReviewer can indicate their review by submitting an approval review. |
12 tasks
Contributor
Author
|
/run-all-tests |
cce601f to
e67c854
Compare
Contributor
Author
|
/run-all-tests |
e67c854 to
07356bc
Compare
Contributor
Author
|
/run-all-tests |
1cd0213 to
1bfc7b7
Compare
Collaborator
Coverage for changed filesCoverage summaryfull coverage report (for internal network access only) |
JaySon-Huang
reviewed
Jun 2, 2022
1bfc7b7 to
b11d58b
Compare
Contributor
Author
|
/run-all-tests |
Collaborator
Coverage for changed filesCoverage summaryfull coverage report (for internal network access only) |
flowbehappy
approved these changes
Jun 2, 2022
Contributor
Author
|
/merge |
Member
|
@lidezhu: It seems you want to merge this PR, I will help you trigger all the tests: /run-all-tests You only need to trigger If you have any questions about the PR merge process, please refer to pr process. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
Member
|
This pull request has been accepted and is ready to merge. DetailsCommit hash: ee5ccce |
Contributor
Author
|
/run-all-tests |
Collaborator
Coverage for changed filesCoverage summaryfull coverage report (for internal network access only) |
ti-chi-bot
pushed a commit
to ti-chi-bot/tiflash
that referenced
this pull request
Jun 2, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
12 tasks
Member
|
In response to a cherrypick label: new pull request created: #5050. |
ti-chi-bot
pushed a commit
to ti-chi-bot/tiflash
that referenced
this pull request
Jun 2, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
12 tasks
Member
|
In response to a cherrypick label: new pull request created: #5051. |
Member
|
In response to a cherrypick label: failed to apply #5044 on top of branch "release-6.1": |
Contributor
Author
ti-chi-bot
added a commit
that referenced
this pull request
Jun 2, 2022
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #5032
Problem Summary: Currently we cache decoding schema for decoding raft data if a table schema doesn't change. And we judge whether a table schema has changed based on the table schema version.
But the schema version is not strictly consistent with the actual table schema which can be seen at https://github.com/pingcap/tiflash/blob/master/dbms/src/TiDB/Schema/SchemaBuilder.cpp#L362. That is when applying different schema changes in a diff, the table schema version will be set to the latest schema version after the first schema change is applied.
More concretely, when a lossy ddl change occurs, it will trigger
drop columnandadd columnschema changes and also rewrite some data at the same time. The schema changes will be applied one by one, and because tiflash updates the table schema version ahead of time when applying schema diff, after applying thedrop columnschema change, it will update the schema version to the latest schema version.And if the decode thread tries to obtain the current schema for decoding data before the
add columnis applied, the current schema and the latest schema version will be cached. Then after the subsequentadd columnoperation is applied, the table schema version will not be updated, so the cache of the decode thread will not be invalidated.Therefore, the decode thread will decode the new data with an older schema, considering that the new added column is a dropped column and discarding it.
In addition, there is a lower chance of triggering this problem in the case of frequent add column and insert data.
What is changed and how it works?
Add an internal schema version for
DecodingStorageSchemaSnapshot.Check List
Tests
Side effects
Documentation
Release note