Skip to content

lightning: job range keys and region split keys are 2 concepts (part 2) (#55759)#58152

Closed
ti-chi-bot wants to merge 23 commits intopingcap:release-8.1from
ti-chi-bot:cherry-pick-55759-to-release-8.1
Closed

lightning: job range keys and region split keys are 2 concepts (part 2) (#55759)#58152
ti-chi-bot wants to merge 23 commits intopingcap:release-8.1from
ti-chi-bot:cherry-pick-55759-to-release-8.1

Conversation

@ti-chi-bot
Copy link
Member

@ti-chi-bot ti-chi-bot commented Dec 11, 2024

This is an automated cherry-pick of #55759

What problem does this PR solve?

Issue Number: close #55374

Problem Summary:

What changed and how does it work?

  1. common.Engine interface has changed. Now it doesn't have a region split size parameter. Implementations can choose split keys or range keys when constructed, or set the region split size parameter as a member.
  2. LoadIngestData now returns multiple ranges instead of one range. These ranges should be considered with actual regions in cluster and convereted to regionJob
  3. LoadIngestData now return data and ranges based on implementation's internal information rather than arguments. For example, for external engine it will use jobKeys.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@lance6716 @ti-chi-bot
external: range job keys and region split keys are 2 concepts
e5bf6fa 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
save more
4c41f24 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
address comment
d71a02f 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
--wip-- [skip ci]
8482d51
@lance6716 @ti-chi-bot
--wip-- [skip ci]
2574438
@lance6716 @ti-chi-bot
--wip-- [skip ci]
44ceaac
@lance6716 @ti-chi-bot
--wip-- [skip ci]
873c878
@lance6716 @ti-chi-bot
save work
7c9d7a6 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
need to add test
13ccdfa 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix some build
8366249 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
add UT
47d4be7 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
add more test
3acac50 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix bazel
aa1587e 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix lightning UT
31b3e86 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix UT
b793613 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix some CI
bda01a2 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix some test
36e52ea 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix UT
b122fd8 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
revert a debug change
215a970 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix a UT
1d788a0 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
fix CI
b12ab7c 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
address comment
512b25a 
Signed-off-by: lance6716 <lance6716@gmail.com>
@lance6716 @ti-chi-bot
Apply suggestions from code review
f75b1ee
@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-8.1 This PR is cherry-picked to release-8.1 from a source PR. labels Dec 11, 2024
@ti-chi-bot ti-chi-bot mentioned this pull request Dec 11, 2024
Merged
13 tasks
@ti-chi-bot ti-chi-bot bot added do-not-merge/cherry-pick-not-approved cherry-pick-approved Cherry pick PR approved by release team. labels Dec 11, 2024
@ti-chi-bot ti-chi-bot bot added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Dec 11, 2024
@ti-chi-bot
Copy link

ti-chi-bot bot commented Dec 11, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign benjamin2037, yudongusa for approval, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@codecov
Copy link

codecov bot commented Dec 11, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 68.05970% with 107 lines in your changes missing coverage. Please review.

Please upload report for BASE (release-8.1@1e2bb37). Learn more about missing BASE report.

Additional details and impacted files 
@@               Coverage Diff                @@
##             release-8.1     #58152   +/-   ##
================================================
  Coverage               ?   71.2146%           
================================================
  Files                  ?       1467           
  Lines                  ?     423562           
  Branches               ?          0           
================================================
  Hits                   ?     301638           
  Misses                 ?     101423           
  Partials               ?      20501
Flag Coverage Δ
unit 71.2146% <68.0597%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.9656% <0.0000%> (?)
parser ∅ <0.0000%> (?)
br 41.2849% <0.0000%> (?)

@lance6716
Copy link
Contributor

lance6716 commented Dec 11, 2024

close and trigger cherry-pick again

@lance6716 lance6716 closed this Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@lance6716 lance6716

Labels

cherry-pick-approved Cherry pick PR approved by release team. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-8.1 This PR is cherry-picked to release-8.1 from a source PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ti-chi-bot @lance6716