Skip to content

diff: REORGANIZE PARTITION for per-partition definition rewrite of RANGE/LIST (split/merge still unsupported)#53

Merged
winebarrel merged 26 commits into
mainfrom
partition-reorganize-value-change
May 4, 2026
Merged

diff: REORGANIZE PARTITION for per-partition definition rewrite of RANGE/LIST (split/merge still unsupported)#53
winebarrel merged 26 commits into
mainfrom
partition-reorganize-value-change

Conversation

@winebarrel

@winebarrel winebarrel commented May 3, 2026

Copy link
Copy Markdown
Owner

Summary

Builds on PR #50's RANGE/LIST suffix add + subset drop and PR #52's HASH/KEY count grow + shrink. When the catalog and desired partition name lists line up position-by-position (every partition stays in the same slot, every name matches case-insensitively), any per-partition definition difference is now generated as one or more ALTER TABLE … REORGANIZE PARTITION p_i, p_{i+1}, … INTO (PARTITION p_i …, PARTITION p_{i+1} …, …) statements. The most common shape is a VALUES LESS THAN / VALUES IN boundary tweak (e.g. p2020 LESS THAN (2021)p2020 LESS THAN (2025)), but COMMENT / MAX_ROWS / TABLESPACE and other per-partition options that round-trip through vitess's PartitionDefinition formatter are picked up here too. Two semantic no-ops are intentionally suppressed even though they would otherwise look byte-different: case-only partition-name diffs (pAB → PAB — MySQL identifiers are case-insensitive) and LIST / LIST COLUMNS VALUES IN (…) permutations (the value list is a set semantically, so reordering is a no-op). MySQL redistributes existing rows into the new boundaries (row-preserving), so no --allow-drop gating.

How the REORGANIZE statements are sliced depends on the partition strategy because the safety constraints differ:

  • RANGE / RANGE COLUMNS — one REORGANIZE per run of consecutive changed slots. RANGE values are continuous and bound by per-slot boundaries, so a value can only move to an adjacent slot when its boundary shifts — values can't cross over an unchanged slot in between. A "p1 + p3 boundary edit" 5-partition RANGE table emits two REORGANIZE statements rather than one giant REORGANIZE p1, p2, p3, p4 … that drags unchanged p2 into the rewrite. Each run additionally extends by one slot when that run's last changed slot's VALUES LESS THAN actually moved AND that slot isn't the final partition (the VALUES LESS THAN <x> of slot i is also the implicit lower bound of slot i+1, so MySQL needs slot i+1 in the REORGANIZE — otherwise it rejects with "VALUES less than value must be strictly increasing"). Per-partition option-only edits don't trigger that cascade, so a metadata-only change stays a one-slot REORGANIZE.

  • LIST / LIST COLUMNS — per-run when no value leaves any changed slot, single span otherwise. The diff layer checks each changed slot's NEW VALUES IN (…) against its OLD set; when every changed slot's NEW is a superset of OLD (no value removed anywhere), per-run is safe (no value can flow to a different slot, so no MySQL Error 1495 "Multiple definition of same constant in list partitioning" risk) and we emit one REORGANIZE per consecutive run — pure-addition disjoint diffs like "p0 gains 9, p3 gains 10 in a 100-partition table" stay proportional to the diff instead of dragging 98 unchanged in-betweens through a data-moving rewrite. When at least one changed slot drops a value (whether the value moved to another slot or was discarded), fall back to a single span over [first..last]; the matched in-betweens get re-stated unchanged. The check uses sqlparser.String keying so it's tuple-safe for LIST COLUMNS for free.

Other "drops AND adds both non-empty" shapes (split / merge / reorder, retention roll-forward with disjoint names, interior insert in front of a catch-all) still error out — inferring the right REORGANIZE boundaries from a name-only diff isn't safe. The error message reads split / merge / reorder to distinguish from the supported per-partition definition branch.

The appendAlterHints rewrite that injects --alter-algorithm / --alter-lock clauses got a related fix in this PR. MySQL's parser rejects the trailing-comma form on partition operations (REORGANIZE / ADD / DROP / COALESCE / TRUNCATE / EXCHANGE PARTITION) — Error 1064, the comma after the closing ) of the partition list (or after the partition_names list) is treated as a new partition_definition. The hint inserter now detects partition ops by skipping ALTER TABLE <table-name> (back-tick / qualified-name aware) and matching the alter-spec keyword that follows, then splices the hints in the leading position MySQL accepts (ALTER TABLE t ALGORITHM=…, LOCK=…, REORGANIZE PARTITION …). Column / index / constraint operations keep the trailing-comma format.

Behaviour matrix delta vs PR #52

Scenario Result
RANGE / RANGE COLUMNS: same-name same-order, one or more per-partition definitions differ One REORGANIZE PARTITION … per run of consecutive changed slots, each run extended by one slot when its last's VALUES LESS THAN boundary actually moved AND last isn't the final partition
LIST / LIST COLUMNS: same-name same-order, every changed slot's NEW VALUES IN ⊇ OLD VALUES IN (pure addition) One REORGANIZE PARTITION … per run of consecutive changed slots — matched in-betweens stay alone
LIST / LIST COLUMNS: same-name same-order, some changed slot drops a value One single-span REORGANIZE PARTITION p_first, …, p_last INTO (…) covering [first..last] (matched in-betweens re-stated unchanged)
Either: case-only name diffs (pAB → PAB) and/or LIST VALUES IN (…) permutations no DDL (semantic no-op)
Either: drops AND adds non-empty, name lists don't line up (split / merge / reorder / retention roll-forward / interior insert in front of catch-all) error: split / merge / reorder
Any partition op above + --alter-algorithm / --alter-lock Hints spliced in the leading position (ALTER TABLE t ALGORITHM=…, LOCK=…, <KEYWORD> …) — MySQL rejects the trailing-comma form for partition ops

Implementation

  • diff/partitions.go adds partitionNameListEqual (positional case-insensitive name compare), partitionDefEqual (case-insensitive name + bytewise body compare via parser.FormatPartitionDefinition, with temporarilySortListValues folding VALUES IN permutations as no-ops), partitionValueRangeEqual (boundary-only compare for the RANGE cascade gate, also LIST-permutation-safe via temporarilySortValueRangeListValues so it's reusable), partitionListChangedSlotsAreSuperset (the LIST per-run-vs-single-span gate), and formatReorganizeRun (renders one REORGANIZE for one run with model.Ident quoting on the OLD-name list, vitess's formatter on the INTO bodies, and a precomputed matchedSlots mask so cascaded / matched-in-between slots use the catalog's body and changed slots use desired's). The matched-name-list arm of diffPartitions walks both sides exactly once to build matchedSlots, then dispatches to RANGE per-run + cascade or LIST per-run-or-single-span; mismatched lists fall through to the unsupported error.
  • parser/partition.go exports FormatPartitionDefinition (used by the new REORGANIZE generator to render the per-partition body) and ParsePartitionClause (re-parses the normalised partition clause back into the AST so the diff layer can inspect individual definitions).
  • diff_all.go reworks appendHintsTo so partition operations get the leading-position splice and other ALTER TABLE alter-specs keep the trailing-comma form. The detector skips ALTER TABLE <name> (back-tick / qualified-name aware) and only looks at the alter-spec keyword right after the table name — it can't be fooled by a comment or default literal that mentions ADD PARTITION. Indices are computed on raw stmt so non-ASCII byte-length shifts under ToUpper can't displace the splice point.

Tests

  • Plan + apply fixtures for the basic value-change shape: partition_value_change_reorganize, partition_value_change_reorganize_list, partition_value_change_reorganize_range_columns, partition_value_change_reorganize_list_columns. Apply runs against MySQL and verify_no_drift: true confirms convergence in one apply.
  • Minimal-span / cascade pins (RANGE only): partition_value_change_reorganize_minimal_range (scalar RANGE, p1's boundary moves, cascade pulls in p2) and partition_value_change_reorganize_minimal_range_columns (RANGE COLUMNS tuple-boundary equivalent).
  • RANGE per-run pin: partition_value_change_reorganize_range_disjoint (RANGE: p1 + p3 → two REORGANIZEs each with their own cascade extension).
  • LIST per-run vs single-span pins: partition_value_change_reorganize_list_disjoint (LIST p0 + p3 pure-addition → two per-run REORGANIZEs, p1 / p2 stay alone) and partition_value_change_reorganize_list_crossflow (LIST value 4 moves p3 → p0 → single-span REORGANIZE because p3 drops 4 → confirms MySQL accepts the single-span and rejects per-run with Error 1495).
  • Cascade case-only-name pin (RANGE only): partition_value_change_reorganize_cascade_case_only_name (RANGE p1 boundary moves AND cascaded p2 is pB → PB rename → INTO body uses catalog's pB, not desired's PB).
  • Per-partition option-only pins: partition_per_partition_option_change_reorganize (LIST, COMMENT-only), partition_per_partition_option_change_reorganize_range (RANGE, COMMENT-only — also pins cascade-NOT-firing), partition_per_partition_option_change_reorganize_range_columns (RANGE COLUMNS, COMMENT-only — same), partition_per_partition_option_change_reorganize_max_rows (LIST, MAX_ROWS-only — confirms the broader claim isn't COMMENT-specific).
  • Semantic-no-op pins: partition_name_case_only_diff_no_op (pAB → PAB), partition_list_value_permutation_no_op ((1,2) → (2,1)), partition_list_columns_value_permutation_no_op (LIST COLUMNS tuple permutation).
  • Mixed case-only name + body diff: partition_value_change_reorganize_case_only_name_plus_body (pAB → PAB + (1,2) → (1,2,7) at the same slot — pins that the DROP side names the OLD partition and the INTO side names the NEW one without triggering a perpetual rename loop).
  • alter-hints + partition-op pins (apply, end-to-end): partition_value_change_reorganize_alter_hints (REORGANIZE PARTITION) and partition_add_range_with_alter_hints (ADD PARTITION) — both verify MySQL accepts the leading-position splice. Plus extended TestAppendAlterHints table tests in diff_all_internal_test.go covering all four partition-op keywords, the comment-mentioning-keyword regression, and the back-ticked-non-ASCII-name regression for the splice index.
  • Reserved-word quoting on REORGANIZE: TestDiffPartitionsQuotesReservedPartitionNamesOnReorganize in diff/partitions_test.go (companion to the existing ADD/DROP quoting pins).
  • Existing partition_mid_mismatch_errors.yml (which pinned the previous "value-change is unsupported" error) renamed to partition_value_change_reorganize.yml and rewritten as the value-change pin.
  • partition_drop_and_add_disjoint_errors.yml and partition_maxvalue_catchall_insert_errors.yml updated to expect the new split / merge / reorder error string — both shapes still fall through because their name lists don't line up.

Docs

  • CAVEATS.md "Partitioning" moves "per-partition definition change" to the generated bullets, with the RANGE-vs-LIST slicing trade-off (per-run + cascade vs. per-run-with-superset-gate vs. single-span) split into separate sub-bullets, plus the operational warning that REORGANIZE is data-moving (matches the HASH/KEY count-change section's wording).
  • AGENTS.md "In scope" / "Not yet implemented" updated to match.
  • TODO.md entry rewritten — what's left is the split / merge / reorder REORGANIZE shapes, scheme/expression changes, and adding/removing partitioning entirely.

Test plan

  • make lint
  • make test (8.0)
  • make test-scenario

Builds on PR #50's RANGE/LIST suffix add + subset drop and
PR #52's HASH/KEY count grow + shrink. Pure value-change of
existing partitions — same names on both sides, only the
`VALUES LESS THAN` / `VALUES IN` clauses differ — now produces
a single `ALTER TABLE … REORGANIZE PARTITION p1, p2, …
INTO (PARTITION p1 VALUES …, PARTITION p2 VALUES …, …)`. MySQL
redistributes existing rows into the new boundaries, so this
is row-preserving and doesn't need `--allow-drop` gating.

Split / merge / reorder shapes (drops and adds with
non-aligned name lists, including the `[p2020,p2021] →
[p2021,p2022]` retention roll-forward and the
`[p0<10,p1<20,p2<30] → [p0<10,q1<40]` merge that motivated
keeping these unsupported in PR #50) still error — inferring
the right REORGANIZE boundaries from a name-only diff isn't
safe. Error message now reads `split / merge / reorder` to
distinguish from the value-change branch that handles itself.

Implementation:
- `diff/partitions.go` adds `partitionNameListEqual` (positional
  case-insensitive name compare) and uses it to gate the
  REORGANIZE branch in the existing both-non-empty arm of
  diffPartitions.

Tests:
- New plan + apply fixtures (`partition_value_change_reorganize`)
  pin a 2-partition value change → REORGANIZE PARTITION p1, p2
  INTO (… , …); apply runs against MySQL and verify_no_drift
  confirms convergence.
- Existing `partition_mid_mismatch_errors.yml` (which pinned
  the previous error for value-change) is renamed and
  rewritten as the new value-change pin.
- `partition_drop_and_add_disjoint_errors.yml` and
  `partition_maxvalue_catchall_insert_errors.yml` updated to
  expect the new "split / merge / reorder" error string —
  both shapes still fall through to the unsupported branch
  because their name lists don't line up.

Docs:
- CAVEATS.md "Partitioning" moves "pure value-change" to the
  generated bullets and reframes the unsupported entry as
  "split / merge / reorder".
- AGENTS.md "In scope" / "Not yet implemented" updated to
  match.
- TODO.md entry rewritten — what's left is the split / merge /
  reorder REORGANIZE shapes, scheme/expression changes, and
  adding/removing partitioning entirely.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 3, 2026 16:02
@codecov

codecov Bot commented May 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.63636% with 104 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.83%. Comparing base (9004a12) to head (8e04387).
⚠️ Report is 29 commits behind head on main.

Files with missing lines Patch % Lines
diff/partitions.go 59.36% 63 Missing and 26 partials ⚠️
diff_all.go 86.66% 5 Missing and 3 partials ⚠️
diff/tables.go 0.00% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #53      +/-   ##
==========================================
- Coverage   72.74%   71.83%   -0.92%     
==========================================
  Files          28       28              
  Lines        2741     3014     +273     
==========================================
+ Hits         1994     2165     +171     
- Misses        541      615      +74     
- Partials      206      234      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends myschema's partition diffing so RANGE/LIST partition definitions with the same partition names but changed VALUES clauses can be applied automatically via ALTER TABLE ... REORGANIZE PARTITION ... INTO (...), instead of always erroring out. It fits into the repository's ongoing work to expand partition-diff support beyond the earlier ADD/DROP and HASH/KEY count-change cases.

Changes:

  • Adds a new RANGE/LIST diff branch that emits REORGANIZE PARTITION when mixed drop/add diffs appear to be same-name value changes.
  • Adds plan/apply fixtures for the new value-change path and updates existing error fixtures to the new unsupported-shape wording.
  • Updates partitioning docs in CAVEATS.md, AGENTS.md, and TODO.md to describe the new supported case and remaining unsupported shapes.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
testdata/plan/partition_value_change_reorganize.yml New plan fixture for a RANGE value-change that should emit REORGANIZE PARTITION.
testdata/plan/partition_maxvalue_catchall_insert_errors.yml Updates expected error text for unsupported catch-all insertion shape.
testdata/plan/partition_drop_and_add_disjoint_errors.yml Updates expected error text for unsupported mixed add/drop with disjoint names.
testdata/apply/partition_value_change_reorganize.yml New apply fixture validating the REORGANIZE path converges without drift.
diff/partitions.go Implements same-name mixed add/drop detection and emits REORGANIZE PARTITION.
TODO.md Refreshes partitioning roadmap to reflect the newly supported value-change case.
CAVEATS.md Documents generated REORGANIZE behavior and reframes unsupported cases as split/merge/reorder.
AGENTS.md Updates contributor guidance on the current partition-diff support matrix.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/partitions.go Outdated
Comment thread diff/partitions.go Outdated
Comment thread CAVEATS.md
Comment thread diff/partitions.go Outdated
One real bug + one coverage hole + two stale docs.

Bug: tighten the value-change gate. Comparing only the drops
and adds name lists let a reorder like `[p0,p1,p2] →
[p0,p2,p1]` slip through — the subset walk drops `p1`, matches
`p2`, leaves `adds=[p1]`, the drops/adds names line up
positionally, and we'd emit `REORGANIZE PARTITION p1 INTO (p1
…)` even though REORGANIZE can't physically reorder partitions
on disk. Compare the *full* catalog and desired name lists
position-by-position instead — value-change is "every partition
in the same place, only the values differ"; reorder fails the
check and falls through to the split / merge / reorder error.

Pinned with a new plan fixture (`partition_reorder_errors`)
that uses LIST partitioning (so the values can legally appear
out of order) to exercise exactly the [p0,p1,p2] → [p0,p2,p1]
shape Copilot called out.

Coverage: add LIST counterparts to the value-change pin
(`partition_value_change_reorganize_list`, plan + apply with
verify_no_drift). The REORGANIZE branch is shared by RANGE
and LIST, but only RANGE `VALUES LESS THAN` had a fixture.
The LIST fixture exercises `VALUES IN (...)` end-to-end.

Docs:
- diffPartitions's function-header docblock still claimed
  "value change → error". Reword the RANGE/LIST bullet to
  spell out the value-change generation path and the new
  full-list reorder check.
- CAVEATS.md "Workaround for the two systems of record"
  block still listed mid-list value changes and HASH/KEY
  count changes as cases the user has to run by hand.
  Rewrite the block: only "split / merge / reorder", scheme
  changes, add/remove partitioning, and SUBPARTITION are
  manual now; the supported shapes don't need a workaround.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/partitions.go
Comment thread diff/partitions.go Outdated
Comment thread diff/partitions.go
Comment thread CAVEATS.md Outdated
Comment thread diff/partitions.go Outdated
Two real bugs + one wording fix + two coverage / docs:

Bug 1: name comparison must be case-insensitive end-to-end.
`partitionNameListEqual` was already case-insensitive, but
`partitionDefEqual` (used by the subset walk that feeds the
both-non-empty branch) compared names byte-for-byte. So a
`pAB → PAB` style diff failed the per-slot equality, the walk
produced drops=[pAB] adds=[PAB], the name lists lined up
positionally, and we'd emit a REORGANIZE for what should have
been a no-op. Switch partitionDefEqual's name compare to
`strings.EqualFold` so the two predicates agree.

Bug 2: REORGANIZE only the minimal contiguous span. The old
code reused the subset walk's drops/adds, which dropped every
catalog partition after the first per-slot mismatch. So a
`p1<20 → p1<25` edit on `[p1, p2, p3]` emitted
`REORGANIZE PARTITION p1, p2, p3 INTO (...)` even though p2
and p3 didn't change. The new value-change branch walks the
full partition list, finds the first and last differing slot,
and emits a REORGANIZE that covers exactly that span — small
edits stay small, only the slots between first and last
mismatch get re-stated (matched partitions in between have to
be included so the new value range still covers the old one,
which is a MySQL requirement).

Wording: the both-non-empty error message used to call out
"split / merge / reorder" but the same branch fires for
interior inserts in front of a catch-all and for retention
roll-forward too. Reword to "the catalog and desired
partition name lists don't line up position-by-position, so
this is some shape of split / merge / reorder / interior
insert / retention roll-forward" — matches what the diff
actually rejects.

Coverage: add a RANGE COLUMNS REORGANIZE plan fixture
(tuple boundaries `VALUES LESS THAN (a, b)`) and a
minimal-range fixture (`p1 < 20` shifts to `p1 < 25` on a
3-partition table → `REORGANIZE PARTITION p1` only). The
existing fixtures were RANGE / LIST scalar only and always
spanned the whole partition list, so neither path was pinned.

Docs: CAVEATS "Per-partition definition change" bullet
rewritten — describe the supported shape as "any per-partition
definition difference" (since `parser.FormatPartitionDefinition`
only strips ENGINE, options like COMMENT / MAX_ROWS /
TABLESPACE are also picked up), and call out the minimal-span
behaviour explicitly so users know small edits don't expand
into whole-tail rewrites.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/partitions.go
Comment thread testdata/plan/partition_value_change_reorganize_minimal_range.yml Outdated
Comment thread diff/partitions.go
Comment thread CAVEATS.md Outdated
Two real bugs + one fixture follow-up + one typo:

Bug 1: extend the REORGANIZE span to the next partition for
RANGE. Each RANGE / RANGE COLUMNS partition's `LESS THAN <x>`
is also the implicit lower bound of the next partition, so
changing slot i's upper bound silently changes slot i+1's
lower bound. MySQL rejects a REORGANIZE whose new partitions
don't cover the same value range as the old ones, so the
diff has to include the next partition whenever the last
changed slot isn't already the very last in the list. LIST
partitions don't have this cascade (`VALUES IN (...)` is a
closed set per slot), so the extension only fires for RANGE.
Pinned with the existing minimal-range plan fixture (now
expects `REORGANIZE p1, p2` instead of `REORGANIZE p1`) plus
a new apply fixture that runs the resulting ALTER against
MySQL and verify_no_drifts the next plan.

Bug 2: strip the partition name from the formatted-body
comparison in `partitionDefEqual`. Round 3 made the *name*
check case-insensitive, but the existing
`FormatPartitionDefinition(a) == FormatPartitionDefinition(b)`
check still includes the name in the formatted output, so a
`pAB → PAB` desired SQL change still produced different
formatted strings and tripped a spurious REORGANIZE.
Temporarily make b's name match a's around the format call
(restored via defer) so the comparison covers only the
definition body.

Typo: a stray `/` in CAVEATS.md "Per-partition definition
change" bullet (`/and` → `and`).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…d 4 #3 fix)

Round 4's partitionDefEqual fix strips the partition name from
the formatted-body comparison so a `pAB → PAB` desired SQL
change doesn't produce different formatted strings and trip a
spurious REORGANIZE. The fix landed without an explicit
regression test — pin it now with a plan fixture that asserts
the case-only diff produces no executable SQL.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 7 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread testdata/plan/partition_value_change_reorganize_range_columns.yml
Comment thread diff/partitions.go
Comment thread diff/partitions.go Outdated
Comment thread AGENTS.md Outdated
Comment thread TODO.md Outdated
Comment thread AGENTS.md Outdated
Comment thread diff/partitions.go Outdated
#1 Pin the RANGE COLUMNS REORGANIZE end-to-end. The plan-only
   fixture (partition_value_change_reorganize_range_columns.yml)
   covered the generator side, but nothing confirmed MySQL accepts
   `REORGANIZE PARTITION ... INTO (...)` with tuple boundaries or
   that the rewritten boundaries round-trip cleanly. Add the
   matching apply / verify_no_drift fixture.

#2 The case-only partition-name no-op already has a regression test
   (`partition_name_case_only_diff_no_op.yml`, commit 4c6998f).

#3 Pin the per-partition *option*-only diff (here: COMMENT-only)
   end-to-end. partitionDefEqual delegates to
   `parser.FormatPartitionDefinition`, which preserves COMMENT /
   MAX_ROWS / TABLESPACE / etc., so any byte-different definition
   already routed through the REORGANIZE branch — but no fixture
   pinned that MySQL accepts the generated statement or that the
   rewritten options round-trip. New apply fixture
   `partition_per_partition_option_change_reorganize.yml` covers
   both. LIST avoids the RANGE upper-bound cascade so the
   REORGANIZE only touches the one slot whose option actually
   changed.

#4 / #5 / #6 Widen the Coverage / Not-yet-implemented sections of
   AGENTS.md (in scope + v1 cuts) and the TODO.md "Already shipped"
   block from "pure value-change (same names, only `VALUES`
   differs)" to "per-partition definition rewrite ... when both
   sides have the same partition names in the same order — covers
   `VALUES …` boundary tweaks plus COMMENT / MAX_ROWS /
   TABLESPACE / other per-partition option changes that round-trip
   through vitess's PartitionDefinition formatter". Brings the
   docs in line with what diffPartitions actually does (and with
   what CAVEATS.md "Per-partition definition change" already
   describes).

#7 Fix the misleading comment in partitionNameListEqual.
   `parser.NormalizePartitionOption` only lower-cases function /
   column-reference identifiers inside the partition expression —
   it does NOT lower-case partition names. The case-insensitive
   `EqualFold` here is needed precisely because the round-trip
   preserves whatever case the user wrote on the partition name.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/partitions.go Outdated
Comment thread diff/partitions.go Outdated
Comment thread diff/partitions.go Outdated
Comment thread CAVEATS.md Outdated
Comment thread diff/partitions.go Outdated
Round 6 raised one behavior bug + one coverage gap, both around
the RANGE upper-bound cascade in the per-partition definition
rewrite branch.

#1 The cascade `last++` fired unconditionally for RANGE tables
   whenever the last changed slot wasn't the final partition.
   That was right for *boundary* edits (changing slot i's
   `VALUES LESS THAN <x>` silently moves slot i+1's implicit
   lower bound, so MySQL needs slot i+1 in the REORGANIZE), but
   wrong for per-partition *option*-only edits (COMMENT /
   MAX_ROWS / TABLESPACE / …) where no boundary moves and
   pulling p_{last+1} in just to restate it unchanged turns a
   metadata-only edit into a second partition rewrite.

   Fix: gate the cascade on the actual boundary moving. New
   `partitionValueRangeEqual` compares only the
   `Options.ValueRange` of the two PartitionDefinitions; the
   cascade now fires only when the boundary at `last`
   genuinely changed.

#2 No apply / verify_no_drift fixture exercised the
   RANGE-specific span-expansion path for option-only diffs, so
   a regression in the cascade logic would have slipped past
   tests even though the docs claimed RANGE option diffs were
   supported. New
   `testdata/apply/partition_per_partition_option_change_reorganize_range.yml`
   pins the narrower behaviour end-to-end (LIST counterpart
   already existed). The fixture was written first and confirmed
   failing on the unfixed code (REORGANIZE PARTITION p2020,
   p2021 …) before the gate was tightened.
@winebarrel winebarrel changed the title diff: REORGANIZE PARTITION for pure RANGE/LIST value-change (split/merge still unsupported) diff: REORGANIZE PARTITION for per-partition definition rewrite of RANGE/LIST (split/merge still unsupported) May 3, 2026
Two more comments from review #4216592781:

#D CAVEATS.md "MySQL requires the new partitions to cover exactly
   the same value range as the old ones" was inaccurate. The
   value-change apply fixture extends the overall RANGE coverage
   from <2022 to <2026 and still converges, so REORGANIZE clearly
   does not require collective-range equality. The actual MySQL
   constraint is narrower:
     - REORGANIZE'd partitions must be consecutive in the partition
       ordering (Error 1519, fires for both RANGE and LIST), which
       is why matched in-between slots have to be re-stated
       unchanged in the partition_names list.
     - For RANGE, the boundary where the REORGANIZE'd group meets
       the unchanged partitions has to align (otherwise MySQL
       rejects with "VALUES less than value must be strictly
       increasing"), which is why the cascade extends `last` by one
       when the boundary at `last` moved AND `last` isn't the final
       partition.
   Reworded the caveat (and the diff/partitions.go comment) to
   spell out both constraints separately.

#E "Disjoint LIST edits get coalesced into one large REORGANIZE"
   prompted a refactor that split RANGE (contiguous span +
   cascade) from LIST (only-changed slots). Wrote the failing
   apply fixture first, refactored, and the new RANGE/LIST split
   produced the expected `REORGANIZE PARTITION p0, p3 INTO (...)`
   for LIST — but MySQL rejected it with Error 1519. Turns out
   the consecutive-ordering rule applies to LIST too, not just
   RANGE; LIST slots being independent doesn't matter to the
   `partition_names` enumeration, only to whether the new
   `VALUES IN (...)` sets have to add up to the old union. So the
   refactor was wrong; reverted.

   Repurposed the fixture as a pin: the same "p0 + p3 changed in a
   4-partition LIST" diff now asserts the contiguous span
   (`REORGANIZE PARTITION p0, p1, p2, p3 INTO (...)` with p1 / p2
   restated unchanged) and verifies MySQL accepts it via
   verify_no_drift. Future contributors looking at this LIST shape
   and tempted to "skip the in-between matched slots" hit Error
   1519 in tests instead of in production.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread CAVEATS.md Outdated
Comment thread diff/partitions.go Outdated
Comment thread testdata/plan/partition_value_change_reorganize_range_columns.yml Outdated
Three docs-only nits — all about places where the
"value-change" wording / "not pinned end-to-end" tense was
left over after the broader per-partition definition framing
landed in earlier rounds.

#A CAVEATS.md "Diffs that still error" recap line still listed
   "pure value-change REORGANIZE" alongside the supported shapes.
   Widened to "per-partition definition rewrite via REORGANIZE
   PARTITION (VALUES boundary tweaks plus COMMENT / MAX_ROWS /
   TABLESPACE / other per-partition options that round-trip
   through vitess's PartitionDefinition formatter)" so the
   recap matches the "Per-partition definition change"
   explainer above and the actual diff layer behaviour.

#B diff/partitions.go's `diffPartitions` overview comment for
   the "drops AND adds both non-empty" outcome still framed the
   matched-name-list branch as "the only thing that changed is
   each partition's value clause — a pure value-change". The
   actual code routes any byte-different per-partition
   definition through here. Reworded the bullet to "every
   partition stays in the same slot and the diff is necessarily
   an in-place rewrite of one or more per-partition
   definitions — a `VALUES …` boundary tweak, a COMMENT /
   MAX_ROWS / TABLESPACE change, or any other per-partition
   option that round-trips through vitess's
   PartitionDefinition formatter".

#C testdata/plan/partition_value_change_reorganize_range_columns.yml
   header said "this case wasn't pinned end-to-end" — stale
   since round 5 added the matching apply / verify_no_drift
   fixture. Updated to point at
   testdata/apply/partition_value_change_reorganize_range_columns.yml
   so the next reader can find the end-to-end pin.
@winebarrel winebarrel requested a review from Copilot May 3, 2026 17:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 36 out of 36 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff_all.go Outdated
Comment thread testdata/plan/partition_add_range_with_alter_hints.yml Outdated
Comment thread diff_all.go
#A `skipQualifiedIdentifier` のコメント中の literal `` (escape
   for embedded backtick) が round 18 の `make fix` で typographic
   left double-quote (U+201C) に書き換えられていて、説明として
   読めなくなっていた。リテラルバッククォートを使わない描写
   (「embedded backtick is doubled inside the back-ticked
   identifier」) に書き直して、再度の auto-format でも壊れない
   形にした。

#B round 17 で書いた cross-reference
   `partition_add_range_with_alter_hints_apply` は実際には
   存在しないファイル名だった (apply fixture の実名は
   `testdata/apply/partition_add_range_with_alter_hints.yml`)。
   path-prefix 付きで正しい名前に修正。

#C `partitionOpKeywords` には TRUNCATE PARTITION と EXCHANGE
   PARTITION も入っているが、`TestAppendAlterHints` は
   REORGANIZE / ADD / DROP / COALESCE しか pin していなかった。
   両キーワードの leading-splice をテーブルテストに追加 —
   keyword リスト上の typo や future refactor が trailing-comma
   形式 (MySQL 1064) に silent fallback するのを防ぐ。

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 36 out of 36 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/partitions.go
`partitionListChangedSlotsAreSuperset` only checks that each
*changed* slot's NEW `VALUES IN` is a superset of its OLD set.
It doesn't check whether a value newly added to a changed
slot is already owned by an *unchanged* sibling partition. So
a desired schema like `p0: (1) -> (1, 4)` while unchanged p3
still has `4` slips through the gate, the per-run REORGANIZE
emits `REORGANIZE p0 INTO (p0 IN (1, 4))`, and apply fails
with MySQL Error 1495 "Multiple definition of same constant
in list partitioning" — the desired schema itself is invalid
(two partitions own the same LIST constant), but the operator
only finds out at apply time.

New `findDuplicateListValue` walks `desPO.Definitions` once
and returns the first value (formatted via `sqlparser.String`,
so tuple-safe for LIST COLUMNS) that appears in more than one
partition, plus both partition names. The check runs right
after re-parsing the desired clause, before any of the per-
run / single-span / cascade machinery, so the failure surfaces
at plan time with an actionable message ("Remove the
duplicate from one of the partitions") instead of as a deep
apply-time MySQL error.

Test-first: new
`testdata/plan/partition_list_value_overlap_errors.yml` pins
the error string for the `p0 (1) -> (1, 4)` + unchanged
`p3 (4)` shape. Confirmed failing before the check was added
(no error returned, then apply would have hit Error 1495).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/partitions.go Outdated
Comment thread CAVEATS.md
#A round 20's `findDuplicateListValue` only ran on the
   modified-table path through `diffPartitions`. Brand-new
   partitioned tables go through `DiffTables`'s create-table
   branch (`dt.SQL()`) and skip `diffPartitions` entirely, so
   a `CREATE TABLE … PARTITION BY LIST …` whose desired
   schema assigns the same constant to two partitions slipped
   past plan and failed at apply with MySQL Error 1495 —
   defeating the purpose of the round-20 plan-time guard.

   Extracted `validateDesiredListValuesAreDisjoint(fqtn,
   defs)` as a shared helper (formats the same actionable
   error from both call sites). `diffPartitions` calls it on
   its already-parsed `desPO.Definitions`; the create-table
   branch in `DiffTables` parses the desired clause once,
   calls this helper, then runs the existing
   `partitionRequiredColumns` / `uniqueKeyPartitionCoverGap`
   pair (kept separate so the cover-gap check can grow
   independently). The two checks together mirror what
   diffTable enforces on the modified-table side.

   Test-first: new
   `testdata/plan/partition_list_value_overlap_on_create_errors.yml`
   pins the create-path equivalent of
   `partition_list_value_overlap_errors.yml` (catalog empty,
   desired CREATE TABLE has overlapping LIST constants).
   Confirmed failing before the wiring landed.

#B CAVEATS.md "Diffs that still error" listed the existing
   PRIMARY KEY / UNIQUE INDEX cover-gap and SUBPARTITION
   guards, but didn't document the new
   `desired LIST partition definitions assign value V to both
   partition pX and partition pY` plan-time error. Added a
   bullet that explains the rule (MySQL Error 1495), the
   coverage (both modified-table and create-table paths),
   the LIST COLUMNS tuple-safety, and the workaround (remove
   the duplicate from one partition).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 39 out of 39 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/partitions.go
Comment thread diff_all.go
#A `findDuplicateListValue` keys with `sqlparser.String` so it
   works on tuple values for `LIST COLUMNS` too, but only the
   scalar `LIST` overlap was pinned (round 20 fixture). New
   `partition_list_columns_value_overlap_errors.yml`
   exercises the same plan-time error message for tuple
   values (`(3, 3)` ends up assigned to both p0 and p1) so a
   regression in tuple formatting / equality surfaces here
   instead of as MySQL Error 1495 at apply time.

#B Apply-side coverage of the `appendHintsTo` leading-position
   splice was REORGANIZE + ADD PARTITION only; DROP PARTITION
   and COALESCE PARTITION (HASH/KEY shrink) weren't pinned
   end-to-end. Added two apply fixtures so MySQL has to accept
   the leading-position splice for those keywords too:
   - `partition_drop_range_with_alter_hints.yml`: DROP
     PARTITION p2020 + ALGORITHM=INPLACE LOCK=NONE under
     `--allow-drop=partition`.
   - `partition_coalesce_hash_with_alter_hints.yml`: HASH
     PARTITIONS 4 -> PARTITIONS 2 under
     `--alter-algorithm=COPY --alter-lock=SHARED` (also
     `--allow-drop=partition`-gated).
   Both run with verify_no_drift so a keyword-specific splice
   bug fails the suite instead of escaping to apply-time
   1064.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/partitions.go
Comment thread CAVEATS.md Outdated
Comment thread diff/partitions.go
#A The LIST single-span fallback assumed any leftover-value
   shape was safe to auto-generate. It isn't: when desired
   removes a `VALUES IN` constant *without* re-assigning it
   to another partition, MySQL silently DROPs any rows for
   that value on REORGANIZE — no apply-time error, no
   warning, just data gone (verified against MySQL 8.0.45 in
   a probe). Detect the catalog-vs-desired discard at plan
   time and gate it behind `--allow-drop=partition`, the
   same flag DROP PARTITION / COALESCE PARTITION already use
   for partition-level data destruction.

   New `findDiscardedListValue` walks the catalog `VALUES IN
   (…)` constants once and returns the first one missing
   from desired. `sqlparser.String` keying makes it tuple-
   safe for `LIST COLUMNS` for free. Without
   `--allow-drop=partition`, plan errors with `desired LIST
   partition layout discards value V (catalog partition pX);
   MySQL silently drops any matching rows on REORGANIZE`.
   With the flag, the discard goes through as the operator's
   acknowledged data loss.

   Test-first:
   - `partition_list_value_discard_disallowed.yml` (plan,
     no flag) pins the new error.
   - `partition_list_value_discard_with_allow.yml` (apply,
     `allow_drop: [partition]`) pins that the gate lets the
     REORGANIZE through and verify_no_drift confirms MySQL
     accepts the resulting layout.

#B CAVEATS.md "Per-partition definition change" claimed
   REORGANIZE is unconditionally row-preserving. Updated to
   spell out the LIST-discard exception (silent row drop on
   value removal) and the new `--allow-drop=partition` gate
   so operators don't read the section and assume there's
   never a need to think about row safety on REORGANIZE.

#C The RANGE cascade only pulled in `last+1`; it never
   verified that the desired-side `VALUES LESS THAN`
   sequence is globally monotonic. A diff like
   `p0 < 10, p1 < 20` -> `p0 < 25, p1 < 20` is rejected by
   MySQL with "VALUES less than value must be strictly
   increasing", but vitess's parser accepts it, so the diff
   layer was happy to emit a REORGANIZE that can never
   apply.

   New `validateDesiredRangeMonotonic` walks the desired-side
   RANGE definitions once. Integer-literal boundaries are
   compared numerically; MAXVALUE is treated as +∞ (allowed
   at most once and only as the final partition). Non-
   integer / non-literal boundaries (function calls, RANGE
   COLUMNS tuples) only surface here on consecutive bytewise
   duplicates — deeper ordering for those falls back to
   MySQL's own error at apply time, because the diff layer
   doesn't have a general expression evaluator.

   Test-first:
   `partition_range_non_monotonic_errors.yml` pins the
   integer-literal case (`p0 < 25, p1 < 20`) with the
   new error message. CAVEATS.md "Diffs that still error"
   gets a corresponding bullet.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 45 out of 45 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/partitions.go Outdated
Round 23's `findDiscardedListValue` ran *before* the diff
layer decided which path the diff would take, so it hard-
errored on two shapes the existing dedicated paths already
handle correctly:

1. **Supported LIST DROP PARTITION without
   `--allow-drop=partition`.** When `p3` is dropped entirely
   (every value in `p3.VALUES IN (…)` disappears from
   desired), the existing DROP PARTITION code further down
   already gates on `--allow-drop=partition` and surfaces a
   `-- skipped: ALTER TABLE … DROP PARTITION p3;` line for
   the unauthorised drop. The early discard guard pre-empted
   that and returned a hard error referencing REORGANIZE
   instead — wrong path and wrong message.

2. **LIST → HASH (or other) scheme change.** When desired
   moves the table to a non-LIST strategy, every catalog
   LIST constant is "discarded" by definition. The strategy
   mismatch is caught further down by `partitionHeaderEqual`
   with the dedicated `partition strategy / expression
   differs` error. The early discard guard pre-empted that
   too.

Fix: move the discard check inside the matched-name-list
REORGANIZE branch (which is the only path that actually
generates a REORGANIZE that could silently drop rows).
Gated additionally on `curPO.Type == ListType` so the check
is only reachable when the catalog itself is LIST. The
original protection (round 23) for the genuine REORGANIZE-
discard shape is preserved — `partition_list_value_discard_*`
fixtures still pass — but the supported DROP / strategy-
change paths now fall through correctly.

Test-first regression pins:
- `partition_drop_list_disallowed.yml`: LIST table, p3
  dropped, no `--allow-drop=partition` → empty plan +
  `-- skipped: ALTER TABLE … DROP PARTITION p3;` line.
- `partition_scheme_change_list_to_hash_errors.yml`:
  LIST → HASH → `partition strategy / expression differs`
  error (not the REORGANIZE-discard message).

Both confirmed failing on the round-23 code (REORGANIZE-
discard error fired instead of the right path) before the
move.
@winebarrel winebarrel requested a review from Copilot May 4, 2026 04:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 47 out of 47 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/tables.go
Round 23's `validateDesiredRangeMonotonic` was only wired
into `diffPartitions` (modified-table path). Brand-new
partitioned tables go through `DiffTables`'s create-table
branch (`dt.SQL()` directly, skipping `diffPartitions`), so
a `CREATE TABLE … PARTITION BY RANGE …` whose
`VALUES LESS THAN` sequence isn't strictly increasing
slipped past plan and only surfaced at apply time with
MySQL's "VALUES less than value must be strictly
increasing" error — defeating the purpose of the round-23
plan-time guard.

This is the same shape of gap round 21 closed for the LIST
overlap guard. Wire `validateDesiredRangeMonotonic` into
the existing create-table partition validation block so
both desired-side schema checks (LIST overlap + RANGE
monotonic) fire from both paths. The unique-key cover
check stays separate because it depends on `dt.Indexes`
(the model), not just the parsed partition AST.

Test-first: new
`testdata/plan/partition_range_non_monotonic_on_create_errors.yml`
pins the create-path equivalent of
`partition_range_non_monotonic_errors.yml` (catalog empty,
desired CREATE TABLE has `p0 < 25, p1 < 20`). Confirmed
failing before the wiring landed.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 48 out of 48 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants