lightning: job range keys and region split keys are 2 concepts (part 2) by lance6716 · Pull Request #55759 · pingcap/tidb

lance6716 · 2024-08-30T03:02:49Z

What problem does this PR solve?

Issue Number: close #55374

Problem Summary:

What changed and how does it work?

common.Engine interface has changed. Now it doesn't have a region split size parameter. Implementations can choose split keys or range keys when constructed, or set the region split size parameter as a member.
LoadIngestData now returns multiple ranges instead of one range. These ranges should be considered with actual regions in cluster and convereted to regionJob
LoadIngestData now return data and ranges based on implementation's internal information rather than arguments. For example, for external engine it will use jobKeys.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: lance6716 <lance6716@gmail.com>

tiprow · 2024-08-30T03:03:13Z

Hi @lance6716. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-08-30T04:09:02Z

Codecov Report

Attention: Patch coverage is 88.93443% with 27 lines in your changes missing coverage. Please review.

Project coverage is 58.2626%. Comparing base (7ea6526) to head (e891f64).
Report is 736 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #55759         +/-   ##
=================================================
- Coverage   72.8318%   58.2626%   -14.5693%     
=================================================
  Files          1586       1751        +165     
  Lines        443354     651052     +207698     
=================================================
+ Hits         322903     379320      +56417     
- Misses       100590     245978     +145388     
- Partials      19861      25754       +5893

Flag	Coverage Δ
integration	`41.3092% <84.0163%> (?)`
unit	`73.4254% <70.1244%> (+1.4740%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`54.5253% <ø> (+1.5686%)`	⬆️
parser	`∅ <ø> (∅)`
br	`64.5259% <ø> (+19.1285%)`	⬆️

Signed-off-by: lance6716 <lance6716@gmail.com>

pkg/ddl/backfilling_import_cloud.go

pkg/lightning/backend/external/engine.go

pkg/lightning/backend/local/region_job.go

lance6716 · 2024-09-04T05:00:49Z

/retest

tiprow · 2024-09-04T05:01:12Z

@lance6716: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

GMHDBJD

LGTM

ti-chi-bot · 2024-09-04T06:42:44Z

[LGTM Timeline notifier]

Timeline:

2024-09-03 09:28:40.166063506 +0000 UTC m=+350244.684116426: ☑️ agreed by D3Hunter.
2024-09-04 06:42:44.021379304 +0000 UTC m=+426688.539432229: ☑️ agreed by GMHDBJD.

yudongusa

LGTM

ti-chi-bot · 2024-09-04T14:43:56Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, GMHDBJD, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~lightning/OWNERS~~ [yudongusa]
~~pkg/ddl/OWNERS~~ [D3Hunter,GMHDBJD]
~~pkg/disttask/OWNERS~~ [D3Hunter,GMHDBJD]
~~pkg/lightning/OWNERS~~ [D3Hunter,GMHDBJD]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

lance6716 · 2024-09-04T23:32:34Z

/retest

tiprow · 2024-09-04T23:32:55Z

@lance6716: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lance6716 · 2024-09-05T00:19:15Z

/retest

tiprow · 2024-09-05T00:19:37Z

@lance6716: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2024-09-05T01:32:18Z

In response to a cherrypick label: new pull request created to branch release-8.3: #55859.

ti-chi-bot · 2024-12-11T05:26:40Z

In response to a cherrypick label: new pull request created to branch release-8.1: #58152.

lance6716 · 2024-12-11T13:52:22Z

/cherry-pick release-8.1

…2) (pingcap#55759) close pingcap#55374

ti-chi-bot · 2024-12-11T13:53:05Z

@lance6716: new pull request created to branch release-8.1: #58180.

Details

In response to this:

/cherry-pick release-8.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

…2) (#55759) (#58180) close #55374

lance6716 added 4 commits August 26, 2024 16:43

external: range job keys and region split keys are 2 concepts

000ade6

Signed-off-by: lance6716 <lance6716@gmail.com>

save more

286708f

Signed-off-by: lance6716 <lance6716@gmail.com>

address comment

dcbd685

Signed-off-by: lance6716 <lance6716@gmail.com>

--wip-- [skip ci]

26b3bdb

--wip-- [skip ci]

2d1dac1

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 30, 2024

lance6716 added 3 commits August 30, 2024 14:56

--wip-- [skip ci]

04ed97e

--wip-- [skip ci]

92dea94

Merge branch 'master' of github.com:pingcap/tidb into fix55374-2

359cd9d

Signed-off-by: lance6716 <lance6716@gmail.com>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 30, 2024

lance6716 added 2 commits September 2, 2024 11:33

save work

625abc1

Signed-off-by: lance6716 <lance6716@gmail.com>

need to add test

2961738

Signed-off-by: lance6716 <lance6716@gmail.com>

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 2, 2024

fix some build

6b13af8

Signed-off-by: lance6716 <lance6716@gmail.com>

lance6716 changed the title ~~[WIP]external: range job keys and region split keys are 2 concepts (part 2)~~ [WIP]lightning: job range keys and region split keys are 2 concepts (part 2) Sep 2, 2024

lance6716 added 3 commits September 2, 2024 19:03

add UT

471cb44

Signed-off-by: lance6716 <lance6716@gmail.com>

add more test

05b6038

Signed-off-by: lance6716 <lance6716@gmail.com>

fix bazel

4fe77f3

Signed-off-by: lance6716 <lance6716@gmail.com>

ti-chi-bot bot removed do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked labels Sep 2, 2024

D3Hunter approved these changes Sep 3, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 3, 2024

lance6716 commented Sep 3, 2024

View reviewed changes

pkg/ddl/backfilling_import_cloud.go Outdated Show resolved Hide resolved

pkg/lightning/backend/external/engine.go Outdated Show resolved Hide resolved

pkg/lightning/backend/local/region_job.go Outdated Show resolved Hide resolved

lance6716 added 2 commits September 3, 2024 20:37

Apply suggestions from code review

f39ff90

Merge branch 'master' of github.com:pingcap/tidb into fix55374-2

e891f64

GMHDBJD approved these changes Sep 4, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 4, 2024

yudongusa approved these changes Sep 4, 2024

View reviewed changes

ti-chi-bot bot added the approved label Sep 4, 2024

ti-chi-bot bot merged commit 3578e29 into pingcap:master Sep 5, 2024

ti-chi-bot mentioned this pull request Sep 5, 2024

lightning: job range keys and region split keys are 2 concepts (part 2) (#55759) #55859

Closed

13 tasks

ti-chi-bot bot added the needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. label Dec 11, 2024

ti-chi-bot mentioned this pull request Dec 11, 2024

lightning: job range keys and region split keys are 2 concepts (part 2) (#55759) #58152

Closed

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Dec 11, 2024

lightning: job range keys and region split keys are 2 concepts (part …

175c269

…2) (pingcap#55759) close pingcap#55374

ti-chi-bot mentioned this pull request Dec 11, 2024

lightning: job range keys and region split keys are 2 concepts (part 2) (#55759) #58180

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Dec 13, 2024

lightning: job range keys and region split keys are 2 concepts (part …

df4f619

…2) (#55759) (#58180) close #55374

lance6716 mentioned this pull request Dec 17, 2024

lightning: handle full engine range after separate region split keys #58317

Closed

13 tasks

Conversation

lance6716 commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Aug 30, 2024

Uh oh!

codecov bot commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lance6716 commented Sep 4, 2024

Uh oh!

tiprow bot commented Sep 4, 2024

Uh oh!

GMHDBJD left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Sep 4, 2024

[LGTM Timeline notifier]

Uh oh!

yudongusa left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Sep 4, 2024

Uh oh!

lance6716 commented Sep 4, 2024

Uh oh!

tiprow bot commented Sep 4, 2024

Uh oh!

lance6716 commented Sep 5, 2024

Uh oh!

tiprow bot commented Sep 5, 2024

Uh oh!

ti-chi-bot commented Sep 5, 2024

Uh oh!

ti-chi-bot commented Dec 11, 2024

Uh oh!

lance6716 commented Dec 11, 2024

Uh oh!

ti-chi-bot commented Dec 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lance6716 commented Aug 30, 2024 •

edited

Loading

codecov bot commented Aug 30, 2024 •

edited

Loading