Skip to content

lightning: job range keys and region split keys are 2 concepts (part 2) (#55759)#58180

Merged
ti-chi-bot[bot] merged 1 commit intopingcap:release-8.1from
ti-chi-bot:cherry-pick-55759-to-release-8.1
Dec 13, 2024
Merged

lightning: job range keys and region split keys are 2 concepts (part 2) (#55759)#58180
ti-chi-bot[bot] merged 1 commit intopingcap:release-8.1from
ti-chi-bot:cherry-pick-55759-to-release-8.1

Conversation

@ti-chi-bot
Copy link
Member

@ti-chi-bot ti-chi-bot commented Dec 11, 2024

This is an automated cherry-pick of #55759

What problem does this PR solve?

Issue Number: close #55374

Problem Summary:

What changed and how does it work?

  1. common.Engine interface has changed. Now it doesn't have a region split size parameter. Implementations can choose split keys or range keys when constructed, or set the region split size parameter as a member.
  2. LoadIngestData now returns multiple ranges instead of one range. These ranges should be considered with actual regions in cluster and convereted to regionJob
  3. LoadIngestData now return data and ranges based on implementation's internal information rather than arguments. For example, for external engine it will use jobKeys.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot force-pushed the cherry-pick-55759-to-release-8.1 branch from f75b1ee to 175c269 Compare December 11, 2024 13:53
@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-8.1 This PR is cherry-picked to release-8.1 from a source PR. labels Dec 11, 2024
@ti-chi-bot ti-chi-bot mentioned this pull request Dec 11, 2024
Merged
13 tasks
@lance6716
Copy link
Contributor

lance6716 commented Dec 12, 2024

/retest

@lance6716 lance6716 requested a review from Copilot December 12, 2024 02:22
Copilot AI reviewed Dec 12, 2024
View reviewed changes
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 5 out of 16 changed files in this pull request and generated 1 suggestion.

Files not reviewed (11)
  • pkg/lightning/backend/external/BUILD.bazel: Language not supported
  • pkg/lightning/backend/local/region_job_test.go: Evaluated as low risk
  • pkg/lightning/backend/external/engine.go: Evaluated as low risk
  • pkg/lightning/backend/external/split.go: Evaluated as low risk
  • pkg/lightning/backend/local/engine_mgr.go: Evaluated as low risk
  • lightning/tidb-lightning.toml: Evaluated as low risk
  • pkg/lightning/backend/external/reader.go: Evaluated as low risk
  • pkg/lightning/common/ingest_data.go: Evaluated as low risk
  • pkg/disttask/importinto/planner.go: Evaluated as low risk
  • pkg/ddl/backfilling_dist_scheduler.go: Evaluated as low risk
  • pkg/lightning/backend/local/localhelper.go: Evaluated as low risk
Comments skipped due to low confidence (1)

pkg/lightning/backend/local/engine.go:1100

  • [nitpick] The variable name 'jobRangeKeys' is ambiguous. It should be renamed to 'regionSplitKeys'.
jobRangeKeys := e.regionSplitKeysCache

pkg/lightning/backend/local/engine.go
}
}

prev := jobRangeKeys[0]
Copy link

Copilot AI Dec 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no check to ensure that 'jobRangeKeys' is not empty before accessing its first element. Add a check to avoid potential runtime errors.

Copilot uses AI. Check for mistakes.
@codecov
Copy link

codecov bot commented Dec 12, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 73.33333% with 60 lines in your changes missing coverage. Please review.

Please upload report for BASE (release-8.1@02bb564). Learn more about missing BASE report.

Additional details and impacted files 
@@               Coverage Diff                @@
##             release-8.1     #58180   +/-   ##
================================================
  Coverage               ?   71.2230%           
================================================
  Files                  ?       1467           
  Lines                  ?     423645           
  Branches               ?          0           
================================================
  Hits                   ?     301733           
  Misses                 ?     101408           
  Partials               ?      20504
Flag Coverage Δ
unit 71.2230% <73.3333%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.9656% <0.0000%> (?)
parser ∅ <0.0000%> (?)
br 41.2877% <0.0000%> (?)

@lance6716
Copy link
Contributor

lance6716 commented Dec 12, 2024

/retest

@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Dec 12, 2024
@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 12, 2024
@ti-chi-bot
Copy link

ti-chi-bot bot commented Dec 12, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-12-12 08:51:22.596900001 +0000 UTC m=+514872.685702759: ☑️ agreed by lance6716.
  • 2024-12-12 08:52:19.775487692 +0000 UTC m=+514929.864290235: ☑️ agreed by D3Hunter.

Benjamin2037
Benjamin2037 approved these changes Dec 12, 2024
View reviewed changes
Copy link
Collaborator

@Benjamin2037 Benjamin2037 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot
Copy link

ti-chi-bot bot commented Dec 13, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Benjamin2037, D3Hunter, lance6716, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the approved label Dec 13, 2024
@lance6716
Copy link
Contributor

lance6716 commented Dec 13, 2024

/retest

@ti-chi-bot ti-chi-bot bot merged commit df4f619 into pingcap:release-8.1 Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@lance6716 lance6716 lance6716 approved these changes

@D3Hunter D3Hunter D3Hunter approved these changes

@Benjamin2037 Benjamin2037 Benjamin2037 approved these changes

+1 more reviewer

@yudongusa yudongusa yudongusa approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@lance6716 lance6716

Labels

approved cherry-pick-approved Cherry pick PR approved by release team. lgtm release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-8.1 This PR is cherry-picked to release-8.1 from a source PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ti-chi-bot @lance6716 @D3Hunter @Benjamin2037 @yudongusa