Skip to content

lightning: add back table empty check and add a switch config (#30887)#33881

Closed
ti-srebot wants to merge 6 commits intopingcap:release-5.2from
ti-srebot:release-5.2-393415782452
Closed

lightning: add back table empty check and add a switch config (#30887)#33881
ti-srebot wants to merge 6 commits intopingcap:release-5.2from
ti-srebot:release-5.2-393415782452

Conversation

@ti-srebot
Copy link
Contributor

@ti-srebot ti-srebot commented Apr 12, 2022 •
edited
Loading

cherry-pick #30887 to release-5.2
You can switch your code base to this Pull Request by using git-extras:

# In tidb repo:
git pr https://github.com/pingcap/tidb/pull/33881

After apply modifications, you can push your change to this PR via:

git push git@github.com:ti-srebot/tidb.git pr/33881:release-5.2-393415782452

What problem does this PR solve?

Issue Number: close #27919

Problem Summary:

What is changed and how it works?

  • Add a new config tikv-importer.incremental-import to control whether allow lightning to import to table that already contains data
  • When tikv-importer.incremental-import = false, tidb-lightning will check target tables' vacancy. If the tables already contains data, lightning will directly report an error and exit. In this mode, lightning will also avoid create and write data into lightning_meta in target tidb.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility
    • Add a new config to control the table empty check, so this is not compatible with v5.2 and v5.3.

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Add a new config to control whether support incremental import.

@glorv @ti-srebot
cherry pick pingcap#30887 to release-5.2
cf28319 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Apr 12, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • glorv

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/cherry-pick-not-approved do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Apr 12, 2022
@ti-srebot
Copy link
Contributor Author

ti-srebot commented Apr 12, 2022

/run-all-tests

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Apr 12, 2022
@ti-srebot ti-srebot mentioned this pull request Apr 12, 2022
Merged
12 tasks
@ti-srebot ti-srebot added component/lightning This issue is related to Lightning of TiDB. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. type/5.2-cherry-pick type/bugfix This PR fixes a bug. labels Apr 12, 2022
@ti-chi-bot ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 12, 2022
@ti-srebot
Copy link
Contributor Author

ti-srebot commented Apr 12, 2022

@glorv you're already a collaborator in bot's repo.

@ti-chi-bot ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 12, 2022
@sre-bot
Copy link
Contributor

sre-bot commented Apr 12, 2022 •
edited
Loading

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/b54887173a492f419c00682c32e31fcd400393ee

@glorv
Copy link
Contributor

glorv commented Apr 12, 2022

@kennytm @gozssky PTAL

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 12, 2022
@sleepymole
Copy link
Contributor

sleepymole commented Apr 13, 2022

lint error

[](https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_check/detail/tidb_ghpr_check/58657/pipeline#step-113-log-154)[2022-04-12T08:51:46.852Z] br/pkg/lightning/restore/check_info.go

[2022-04-12T08:51:46.852Z] make: *** [fmt] Error 1

@glorv
Copy link
Contributor

glorv commented Apr 13, 2022

/run-unit-test

@glorv
Copy link
Contributor

glorv commented Apr 13, 2022

/run-mysql-test

@glorv
Copy link
Contributor

glorv commented Apr 13, 2022

@gozssky PTAL again

@sleepymole
Copy link
Contributor

sleepymole commented Apr 13, 2022

Integration test fail. I'm not sure if it is caused by this PR.
https://ci.pingcap.net/job/br_ghpr_unit_and_integration_test/11159/display/redirect

@sleepymole
Copy link
Contributor

sleepymole commented Apr 13, 2022

/run-integration-br-test

@VelocityLight VelocityLight added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved cherry-pick-approved Cherry pick PR approved by release team. labels Apr 14, 2022
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Apr 14, 2022

@ti-srebot: This cherry pick PR is for a release branch and has not yet been approved by release team.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick, it must first be approved by the collaborators.

AFTER it has been approved by collaborators, please ping the release team in a comment to request a cherry pick review.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@VelocityLight
Copy link

VelocityLight commented Sep 19, 2022

This pull request is closed because it's related version has closed automatic cherry-picking.
If it's still needed, you can reopen it or just regenerate it using bot,
see:

You can find more details at:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@glorv glorv glorv approved these changes

@3pointer 3pointer Awaiting requested review from 3pointer

@sleepymole sleepymole Awaiting requested review from sleepymole

@kennytm kennytm Awaiting requested review from kennytm

@Little-Wallace Little-Wallace Awaiting requested review from Little-Wallace

Assignees

@glorv glorv

Labels

component/lightning This issue is related to Lightning of TiDB. do-not-merge/cherry-pick-not-approved release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/LGT1 Indicates that a PR has LGTM 1. type/bugfix This PR fixes a bug. type/5.2-cherry-pick

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ti-srebot @ti-chi-bot @sre-bot @glorv @sleepymole @VelocityLight