Skip to content

lightning: job range keys and region split keys are 2 concepts (part 2) (#55759)#55859

Closed
ti-chi-bot wants to merge 23 commits intopingcap:release-8.3from
ti-chi-bot:cherry-pick-55759-to-release-8.3
Closed

lightning: job range keys and region split keys are 2 concepts (part 2) (#55759)#55859
ti-chi-bot wants to merge 23 commits intopingcap:release-8.3from
ti-chi-bot:cherry-pick-55759-to-release-8.3

Conversation

@ti-chi-bot
Copy link
Member

@ti-chi-bot ti-chi-bot commented Sep 5, 2024

This is an automated cherry-pick of #55759

What problem does this PR solve?

Issue Number: close #55374

Problem Summary:

What changed and how does it work?

  1. common.Engine interface has changed. Now it doesn't have a region split size parameter. Implementations can choose split keys or range keys when constructed, or set the region split size parameter as a member.
  2. LoadIngestData now returns multiple ranges instead of one range. These ranges should be considered with actual regions in cluster and convereted to regionJob
  3. LoadIngestData now return data and ranges based on implementation's internal information rather than arguments. For example, for external engine it will use jobKeys.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@lance6716 @ti-chi-bot
external: range job keys and region split keys are 2 concepts
12adbc7 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
save more
0c72884 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
address comment
6d5fedc 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
--wip-- [skip ci]
aa1bdd5
@lance6716 @ti-chi-bot
--wip-- [skip ci]
952f87f
@lance6716 @ti-chi-bot
--wip-- [skip ci]
d7f78d7
@lance6716 @ti-chi-bot
--wip-- [skip ci]
67cde8f
@lance6716 @ti-chi-bot
save work
fd7c7c6 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
need to add test
5345224 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix some build
582a9c3 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
add UT
531a6bc 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
add more test
91f99f2 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix bazel
59a6074 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix lightning UT
9d98abb 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix UT
ac1e773 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix some CI
62f8642 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix some test
a49d476 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix UT
ef66c86 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
revert a debug change
3347bfc 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix a UT
4f01b6c 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix CI
5387a24 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
address comment
f18e0ee 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
Apply suggestions from code review
03c841b
@ti-chi-bot ti-chi-bot mentioned this pull request Sep 5, 2024
Merged
13 tasks
@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-8.3 labels Sep 5, 2024
@ti-chi-bot
Copy link

ti-chi-bot bot commented Sep 5, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign benjamin2037, easonn7 for approval, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Sep 5, 2024
@codecov
Copy link

codecov bot commented Sep 5, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 77.61194% with 75 lines in your changes missing coverage. Please review.

Please upload report for BASE (release-8.3@1a0c3ac). Learn more about missing BASE report.

Additional details and impacted files 
@@               Coverage Diff                @@
##             release-8.3     #55859   +/-   ##
================================================
  Coverage               ?   50.8751%           
================================================
  Files                  ?       1700           
  Lines                  ?     610243           
  Branches               ?          0           
================================================
  Hits                   ?     310462           
  Misses                 ?     276273           
  Partials               ?      23508
Flag Coverage Δ
integration 22.7713% <77.6119%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 55.1894% <0.0000%> (?)
parser ∅ <0.0000%> (?)
br 48.4820% <0.0000%> (?)

@lance6716 lance6716 closed this Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@lance6716 lance6716

Labels

cherry-pick-approved Cherry pick PR approved by release team. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-8.3

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ti-chi-bot @lance6716