Skip to content

ddl: support pre-split index regions before creating index#57553

Merged
ti-chi-bot[bot] merged 17 commits intopingcap:masterfrom
tangenta:presplit-index
Dec 18, 2024
Merged

ddl: support pre-split index regions before creating index#57553
ti-chi-bot[bot] merged 17 commits intopingcap:masterfrom
tangenta:presplit-index

Conversation

@tangenta
Copy link
Contributor

@tangenta tangenta commented Nov 20, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #57552, close #57551

Problem Summary:

See #57552.

What changed and how does it work?

  • Support specifying the split option in the index option at parser.
  • Split the target index region by keys or ranges before data reorganization when adding an index.
  • Extract common code and tests from pkg/executor/split.go to pkg/util/split.go.
  • Add two functional tests.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot bot added do-not-merge/invalid-title do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 20, 2024
@tiprow
Copy link

tiprow bot commented Nov 20, 2024

Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@tangenta tangenta changed the title Support pre-split index regions before creating index ddl: support pre-split index regions before creating index Nov 20, 2024
@codecov
Copy link

codecov bot commented Nov 20, 2024 •
edited
Loading

Codecov Report

❌ Patch coverage is 25.18519% with 303 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.9483%. Comparing base (4ec2b23) to head (d86e2ef).
⚠️ Report is 2329 commits behind head on master.

Additional details and impacted files 
@@               Coverage Diff                @@
##             master     #57553        +/-   ##
================================================
+ Coverage   73.1893%   74.9483%   +1.7589%     
================================================
  Files          1675       1722        +47     
  Lines        461929     471007      +9078     
================================================
+ Hits         338083     353012     +14929     
+ Misses       103081      95784      -7297     
- Partials      20765      22211      +1446
Flag Coverage Δ
integration 48.9990% <17.0940%> (?)
unit 72.5363% <25.1851%> (-0.0206%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling ∅ <ø> (∅)
parser ∅ <ø> (∅)
br 62.4801% <ø> (+15.3236%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

lance6716
lance6716 reviewed Nov 21, 2024
View reviewed changes
D3Hunter
D3Hunter reviewed Nov 25, 2024
View reviewed changes
D3Hunter
D3Hunter reviewed Nov 25, 2024
View reviewed changes
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 25, 2024
bb7133
bb7133 reviewed Dec 10, 2024
View reviewed changes
@ti-chi-bot ti-chi-bot mentioned this pull request Nov 28, 2025
Open
13 tasks
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Nov 28, 2025

In response to a cherrypick label: new pull request created to branch release-8.5: #64751.
But this PR has conflicts, please resolve them!

@Benjamin2037 Benjamin2037 added needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. labels Feb 5, 2026
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Feb 5, 2026
@tangenta @ti-chi-bot
This is an automated cherry-pick of pingcap#57553
2e54d14 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Feb 5, 2026
Open
13 tasks
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Feb 5, 2026

In response to a cherrypick label: new pull request created to branch release-8.1: #66059.
But this PR has conflicts, please resolve them!

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Feb 5, 2026
@tangenta @ti-chi-bot
This is an automated cherry-pick of pingcap#57553
8c92f11 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Feb 5, 2026
Open
13 tasks
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Feb 5, 2026

In response to a cherrypick label: new pull request created to branch release-7.5: #66060.
But this PR has conflicts, please resolve them!

expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 6, 2026
@tangenta @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
525490b 
…7553)

close pingcap#57551, close pingcap#57552
bb7133 pushed a commit to bb7133/tidb that referenced this pull request Feb 6, 2026
@tangenta @bb7133
ddl: support pre-split index regions before creating index (pingcap#5…
2a9d375 
…7553)

close pingcap#57551, close pingcap#57552
@bb7133 bb7133 mentioned this pull request Feb 6, 2026
Merged
13 tasks
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 7, 2026
@tangenta @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
a31b41b 
…7553)

close pingcap#57551, close pingcap#57552
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 7, 2026
@tangenta @cursoragent @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
77394fe 
…7553)

close pingcap#57551, close pingcap#57552

Co-authored-by: Cursor <cursoragent@cursor.com>
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 7, 2026
@tangenta @cursoragent @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
f1e20a8 
…7553)

close pingcap#57551, close pingcap#57552

Co-authored-by: Cursor <cursoragent@cursor.com>
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 8, 2026
@tangenta @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
4cac8b8 
…7553)

close pingcap#57551, close pingcap#57552
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 8, 2026
@tangenta @cursoragent @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
2640e3a 
…7553)

close pingcap#57551, close pingcap#57552

Co-authored-by: Cursor <cursoragent@cursor.com>
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 8, 2026
@tangenta @cursoragent @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
a12d73e 
…7553)

close pingcap#57551, close pingcap#57552

Co-authored-by: Cursor <cursoragent@cursor.com>
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 8, 2026
@tangenta @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
fca07af 
…7553)

close pingcap#57551, close pingcap#57552
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 8, 2026
@tangenta @cursoragent @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
d22252f 
…7553)

close pingcap#57551, close pingcap#57552

Co-authored-by: Cursor <cursoragent@cursor.com>
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 8, 2026
@tangenta @cursoragent @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
2470f1c 
…7553)

close pingcap#57551, close pingcap#57552

Co-authored-by: Cursor <cursoragent@cursor.com>
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 8, 2026
@tangenta @cursoragent @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
1cd6371 
…7553)

close pingcap#57551, close pingcap#57552

Co-authored-by: Cursor <cursoragent@cursor.com>
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 9, 2026
@tangenta @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
d42f4b0 
…7553)

close pingcap#57551, close pingcap#57552
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 9, 2026
@tangenta @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
b00df30 
…7553)

close pingcap#57551, close pingcap#57552
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 9, 2026
@tangenta @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
95c6ac9 
…7553)

close pingcap#57551, close pingcap#57552
expxiaoli pushed a commit to expxiaoli/tidb that referenced this pull request Feb 9, 2026
@tangenta @expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
4674563 
…7553)

close pingcap#57551, close pingcap#57552
expxiaoli added a commit to expxiaoli/tidb that referenced this pull request Feb 10, 2026
@expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
9994550 
…7553)

- align string-to-BIT conversion with MySQL behavior for BIT(1), preserving truncation warning in INSERT IGNORE paths

- remove special-case parsing for string literals in BIT conversion and reuse byte->integer conversion path

- expand unit tests to cover both truncation and non-truncation cases (true/false/0/1 and bit-literal strings)
expxiaoli added a commit to expxiaoli/tidb that referenced this pull request Feb 10, 2026
@expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
61afe3d 
…7553)

- fix unit-test regression in LOAD DATA path for BIT(1) values

- keep MySQL-compatible BIT cast behavior for regular INSERT while restoring legacy compatibility for LOAD DATA inputs true/false/0/1

- only apply compatibility rewrite when InLoadDataStmt and target column is BIT(1), avoiding broader behavior changes
expxiaoli added a commit to expxiaoli/tidb that referenced this pull request Feb 10, 2026
@expxiaoli
ddl: support pre-split index regions before creating index (pingcap#5…
1689ac6 
…7553)

- fix flaky failure in TestBackfillingSchedulerGlobalSortMode by making cleanup nil-safe when fake GCS server creation fails

- remove dependency on external fake-GCS data for this scheduler unit test; keep focus on step transitions in global-sort mode

- make merge-sort expectation deterministic by using explicit overlapping metadata and pinning mock TS failpoint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bb7133 bb7133 bb7133 left review comments

@lance6716 lance6716 lance6716 left review comments

@D3Hunter D3Hunter D3Hunter approved these changes

@Benjamin2037 Benjamin2037 Benjamin2037 approved these changes

+3 more reviewers

@benmeadowcroft benmeadowcroft benmeadowcroft approved these changes

@yudongusa yudongusa yudongusa approved these changes

@Frank945946 Frank945946 Frank945946 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved lgtm needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support pre-split index regions before creating index Cluster suffering write hotspot after creating index

9 participants

@tangenta @ti-chi-bot @bb7133 @benmeadowcroft @lance6716 @D3Hunter @Benjamin2037 @yudongusa @Frank945946