diff: REORGANIZE PARTITION for per-partition definition rewrite of RANGE/LIST (split/merge still unsupported)#53
Conversation
Builds on PR #50's RANGE/LIST suffix add + subset drop and PR #52's HASH/KEY count grow + shrink. Pure value-change of existing partitions — same names on both sides, only the `VALUES LESS THAN` / `VALUES IN` clauses differ — now produces a single `ALTER TABLE … REORGANIZE PARTITION p1, p2, … INTO (PARTITION p1 VALUES …, PARTITION p2 VALUES …, …)`. MySQL redistributes existing rows into the new boundaries, so this is row-preserving and doesn't need `--allow-drop` gating. Split / merge / reorder shapes (drops and adds with non-aligned name lists, including the `[p2020,p2021] → [p2021,p2022]` retention roll-forward and the `[p0<10,p1<20,p2<30] → [p0<10,q1<40]` merge that motivated keeping these unsupported in PR #50) still error — inferring the right REORGANIZE boundaries from a name-only diff isn't safe. Error message now reads `split / merge / reorder` to distinguish from the value-change branch that handles itself. Implementation: - `diff/partitions.go` adds `partitionNameListEqual` (positional case-insensitive name compare) and uses it to gate the REORGANIZE branch in the existing both-non-empty arm of diffPartitions. Tests: - New plan + apply fixtures (`partition_value_change_reorganize`) pin a 2-partition value change → REORGANIZE PARTITION p1, p2 INTO (… , …); apply runs against MySQL and verify_no_drift confirms convergence. - Existing `partition_mid_mismatch_errors.yml` (which pinned the previous error for value-change) is renamed and rewritten as the new value-change pin. - `partition_drop_and_add_disjoint_errors.yml` and `partition_maxvalue_catchall_insert_errors.yml` updated to expect the new "split / merge / reorder" error string — both shapes still fall through to the unsupported branch because their name lists don't line up. Docs: - CAVEATS.md "Partitioning" moves "pure value-change" to the generated bullets and reframes the unsupported entry as "split / merge / reorder". - AGENTS.md "In scope" / "Not yet implemented" updated to match. - TODO.md entry rewritten — what's left is the split / merge / reorder REORGANIZE shapes, scheme/expression changes, and adding/removing partitioning entirely. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #53 +/- ##
==========================================
- Coverage 72.74% 71.83% -0.92%
==========================================
Files 28 28
Lines 2741 3014 +273
==========================================
+ Hits 1994 2165 +171
- Misses 541 615 +74
- Partials 206 234 +28 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR extends myschema's partition diffing so RANGE/LIST partition definitions with the same partition names but changed VALUES clauses can be applied automatically via ALTER TABLE ... REORGANIZE PARTITION ... INTO (...), instead of always erroring out. It fits into the repository's ongoing work to expand partition-diff support beyond the earlier ADD/DROP and HASH/KEY count-change cases.
Changes:
- Adds a new RANGE/LIST diff branch that emits
REORGANIZE PARTITIONwhen mixed drop/add diffs appear to be same-name value changes. - Adds plan/apply fixtures for the new value-change path and updates existing error fixtures to the new unsupported-shape wording.
- Updates partitioning docs in
CAVEATS.md,AGENTS.md, andTODO.mdto describe the new supported case and remaining unsupported shapes.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
testdata/plan/partition_value_change_reorganize.yml |
New plan fixture for a RANGE value-change that should emit REORGANIZE PARTITION. |
testdata/plan/partition_maxvalue_catchall_insert_errors.yml |
Updates expected error text for unsupported catch-all insertion shape. |
testdata/plan/partition_drop_and_add_disjoint_errors.yml |
Updates expected error text for unsupported mixed add/drop with disjoint names. |
testdata/apply/partition_value_change_reorganize.yml |
New apply fixture validating the REORGANIZE path converges without drift. |
diff/partitions.go |
Implements same-name mixed add/drop detection and emits REORGANIZE PARTITION. |
TODO.md |
Refreshes partitioning roadmap to reflect the newly supported value-change case. |
CAVEATS.md |
Documents generated REORGANIZE behavior and reframes unsupported cases as split/merge/reorder. |
AGENTS.md |
Updates contributor guidance on the current partition-diff support matrix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
One real bug + one coverage hole + two stale docs. Bug: tighten the value-change gate. Comparing only the drops and adds name lists let a reorder like `[p0,p1,p2] → [p0,p2,p1]` slip through — the subset walk drops `p1`, matches `p2`, leaves `adds=[p1]`, the drops/adds names line up positionally, and we'd emit `REORGANIZE PARTITION p1 INTO (p1 …)` even though REORGANIZE can't physically reorder partitions on disk. Compare the *full* catalog and desired name lists position-by-position instead — value-change is "every partition in the same place, only the values differ"; reorder fails the check and falls through to the split / merge / reorder error. Pinned with a new plan fixture (`partition_reorder_errors`) that uses LIST partitioning (so the values can legally appear out of order) to exercise exactly the [p0,p1,p2] → [p0,p2,p1] shape Copilot called out. Coverage: add LIST counterparts to the value-change pin (`partition_value_change_reorganize_list`, plan + apply with verify_no_drift). The REORGANIZE branch is shared by RANGE and LIST, but only RANGE `VALUES LESS THAN` had a fixture. The LIST fixture exercises `VALUES IN (...)` end-to-end. Docs: - diffPartitions's function-header docblock still claimed "value change → error". Reword the RANGE/LIST bullet to spell out the value-change generation path and the new full-list reorder check. - CAVEATS.md "Workaround for the two systems of record" block still listed mid-list value changes and HASH/KEY count changes as cases the user has to run by hand. Rewrite the block: only "split / merge / reorder", scheme changes, add/remove partitioning, and SUBPARTITION are manual now; the supported shapes don't need a workaround. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Two real bugs + one wording fix + two coverage / docs: Bug 1: name comparison must be case-insensitive end-to-end. `partitionNameListEqual` was already case-insensitive, but `partitionDefEqual` (used by the subset walk that feeds the both-non-empty branch) compared names byte-for-byte. So a `pAB → PAB` style diff failed the per-slot equality, the walk produced drops=[pAB] adds=[PAB], the name lists lined up positionally, and we'd emit a REORGANIZE for what should have been a no-op. Switch partitionDefEqual's name compare to `strings.EqualFold` so the two predicates agree. Bug 2: REORGANIZE only the minimal contiguous span. The old code reused the subset walk's drops/adds, which dropped every catalog partition after the first per-slot mismatch. So a `p1<20 → p1<25` edit on `[p1, p2, p3]` emitted `REORGANIZE PARTITION p1, p2, p3 INTO (...)` even though p2 and p3 didn't change. The new value-change branch walks the full partition list, finds the first and last differing slot, and emits a REORGANIZE that covers exactly that span — small edits stay small, only the slots between first and last mismatch get re-stated (matched partitions in between have to be included so the new value range still covers the old one, which is a MySQL requirement). Wording: the both-non-empty error message used to call out "split / merge / reorder" but the same branch fires for interior inserts in front of a catch-all and for retention roll-forward too. Reword to "the catalog and desired partition name lists don't line up position-by-position, so this is some shape of split / merge / reorder / interior insert / retention roll-forward" — matches what the diff actually rejects. Coverage: add a RANGE COLUMNS REORGANIZE plan fixture (tuple boundaries `VALUES LESS THAN (a, b)`) and a minimal-range fixture (`p1 < 20` shifts to `p1 < 25` on a 3-partition table → `REORGANIZE PARTITION p1` only). The existing fixtures were RANGE / LIST scalar only and always spanned the whole partition list, so neither path was pinned. Docs: CAVEATS "Per-partition definition change" bullet rewritten — describe the supported shape as "any per-partition definition difference" (since `parser.FormatPartitionDefinition` only strips ENGINE, options like COMMENT / MAX_ROWS / TABLESPACE are also picked up), and call out the minimal-span behaviour explicitly so users know small edits don't expand into whole-tail rewrites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Two real bugs + one fixture follow-up + one typo: Bug 1: extend the REORGANIZE span to the next partition for RANGE. Each RANGE / RANGE COLUMNS partition's `LESS THAN <x>` is also the implicit lower bound of the next partition, so changing slot i's upper bound silently changes slot i+1's lower bound. MySQL rejects a REORGANIZE whose new partitions don't cover the same value range as the old ones, so the diff has to include the next partition whenever the last changed slot isn't already the very last in the list. LIST partitions don't have this cascade (`VALUES IN (...)` is a closed set per slot), so the extension only fires for RANGE. Pinned with the existing minimal-range plan fixture (now expects `REORGANIZE p1, p2` instead of `REORGANIZE p1`) plus a new apply fixture that runs the resulting ALTER against MySQL and verify_no_drifts the next plan. Bug 2: strip the partition name from the formatted-body comparison in `partitionDefEqual`. Round 3 made the *name* check case-insensitive, but the existing `FormatPartitionDefinition(a) == FormatPartitionDefinition(b)` check still includes the name in the formatted output, so a `pAB → PAB` desired SQL change still produced different formatted strings and tripped a spurious REORGANIZE. Temporarily make b's name match a's around the format call (restored via defer) so the comparison covers only the definition body. Typo: a stray `/` in CAVEATS.md "Per-partition definition change" bullet (`/and` → `and`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…d 4 #3 fix) Round 4's partitionDefEqual fix strips the partition name from the formatted-body comparison so a `pAB → PAB` desired SQL change doesn't produce different formatted strings and trip a spurious REORGANIZE. The fix landed without an explicit regression test — pin it now with a plan fixture that asserts the case-only diff produces no executable SQL. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated 7 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
#1 Pin the RANGE COLUMNS REORGANIZE end-to-end. The plan-only fixture (partition_value_change_reorganize_range_columns.yml) covered the generator side, but nothing confirmed MySQL accepts `REORGANIZE PARTITION ... INTO (...)` with tuple boundaries or that the rewritten boundaries round-trip cleanly. Add the matching apply / verify_no_drift fixture. #2 The case-only partition-name no-op already has a regression test (`partition_name_case_only_diff_no_op.yml`, commit 4c6998f). #3 Pin the per-partition *option*-only diff (here: COMMENT-only) end-to-end. partitionDefEqual delegates to `parser.FormatPartitionDefinition`, which preserves COMMENT / MAX_ROWS / TABLESPACE / etc., so any byte-different definition already routed through the REORGANIZE branch — but no fixture pinned that MySQL accepts the generated statement or that the rewritten options round-trip. New apply fixture `partition_per_partition_option_change_reorganize.yml` covers both. LIST avoids the RANGE upper-bound cascade so the REORGANIZE only touches the one slot whose option actually changed. #4 / #5 / #6 Widen the Coverage / Not-yet-implemented sections of AGENTS.md (in scope + v1 cuts) and the TODO.md "Already shipped" block from "pure value-change (same names, only `VALUES` differs)" to "per-partition definition rewrite ... when both sides have the same partition names in the same order — covers `VALUES …` boundary tweaks plus COMMENT / MAX_ROWS / TABLESPACE / other per-partition option changes that round-trip through vitess's PartitionDefinition formatter". Brings the docs in line with what diffPartitions actually does (and with what CAVEATS.md "Per-partition definition change" already describes). #7 Fix the misleading comment in partitionNameListEqual. `parser.NormalizePartitionOption` only lower-cases function / column-reference identifiers inside the partition expression — it does NOT lower-case partition names. The case-insensitive `EqualFold` here is needed precisely because the round-trip preserves whatever case the user wrote on the partition name.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Round 6 raised one behavior bug + one coverage gap, both around the RANGE upper-bound cascade in the per-partition definition rewrite branch. #1 The cascade `last++` fired unconditionally for RANGE tables whenever the last changed slot wasn't the final partition. That was right for *boundary* edits (changing slot i's `VALUES LESS THAN <x>` silently moves slot i+1's implicit lower bound, so MySQL needs slot i+1 in the REORGANIZE), but wrong for per-partition *option*-only edits (COMMENT / MAX_ROWS / TABLESPACE / …) where no boundary moves and pulling p_{last+1} in just to restate it unchanged turns a metadata-only edit into a second partition rewrite. Fix: gate the cascade on the actual boundary moving. New `partitionValueRangeEqual` compares only the `Options.ValueRange` of the two PartitionDefinitions; the cascade now fires only when the boundary at `last` genuinely changed. #2 No apply / verify_no_drift fixture exercised the RANGE-specific span-expansion path for option-only diffs, so a regression in the cascade logic would have slipped past tests even though the docs claimed RANGE option diffs were supported. New `testdata/apply/partition_per_partition_option_change_reorganize_range.yml` pins the narrower behaviour end-to-end (LIST counterpart already existed). The fixture was written first and confirmed failing on the unfixed code (REORGANIZE PARTITION p2020, p2021 …) before the gate was tightened.
Two more comments from review #4216592781:
#D CAVEATS.md "MySQL requires the new partitions to cover exactly
the same value range as the old ones" was inaccurate. The
value-change apply fixture extends the overall RANGE coverage
from <2022 to <2026 and still converges, so REORGANIZE clearly
does not require collective-range equality. The actual MySQL
constraint is narrower:
- REORGANIZE'd partitions must be consecutive in the partition
ordering (Error 1519, fires for both RANGE and LIST), which
is why matched in-between slots have to be re-stated
unchanged in the partition_names list.
- For RANGE, the boundary where the REORGANIZE'd group meets
the unchanged partitions has to align (otherwise MySQL
rejects with "VALUES less than value must be strictly
increasing"), which is why the cascade extends `last` by one
when the boundary at `last` moved AND `last` isn't the final
partition.
Reworded the caveat (and the diff/partitions.go comment) to
spell out both constraints separately.
#E "Disjoint LIST edits get coalesced into one large REORGANIZE"
prompted a refactor that split RANGE (contiguous span +
cascade) from LIST (only-changed slots). Wrote the failing
apply fixture first, refactored, and the new RANGE/LIST split
produced the expected `REORGANIZE PARTITION p0, p3 INTO (...)`
for LIST — but MySQL rejected it with Error 1519. Turns out
the consecutive-ordering rule applies to LIST too, not just
RANGE; LIST slots being independent doesn't matter to the
`partition_names` enumeration, only to whether the new
`VALUES IN (...)` sets have to add up to the old union. So the
refactor was wrong; reverted.
Repurposed the fixture as a pin: the same "p0 + p3 changed in a
4-partition LIST" diff now asserts the contiguous span
(`REORGANIZE PARTITION p0, p1, p2, p3 INTO (...)` with p1 / p2
restated unchanged) and verifies MySQL accepts it via
verify_no_drift. Future contributors looking at this LIST shape
and tempted to "skip the in-between matched slots" hit Error
1519 in tests instead of in production.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Three docs-only nits — all about places where the "value-change" wording / "not pinned end-to-end" tense was left over after the broader per-partition definition framing landed in earlier rounds. #A CAVEATS.md "Diffs that still error" recap line still listed "pure value-change REORGANIZE" alongside the supported shapes. Widened to "per-partition definition rewrite via REORGANIZE PARTITION (VALUES boundary tweaks plus COMMENT / MAX_ROWS / TABLESPACE / other per-partition options that round-trip through vitess's PartitionDefinition formatter)" so the recap matches the "Per-partition definition change" explainer above and the actual diff layer behaviour. #B diff/partitions.go's `diffPartitions` overview comment for the "drops AND adds both non-empty" outcome still framed the matched-name-list branch as "the only thing that changed is each partition's value clause — a pure value-change". The actual code routes any byte-different per-partition definition through here. Reworded the bullet to "every partition stays in the same slot and the diff is necessarily an in-place rewrite of one or more per-partition definitions — a `VALUES …` boundary tweak, a COMMENT / MAX_ROWS / TABLESPACE change, or any other per-partition option that round-trips through vitess's PartitionDefinition formatter". #C testdata/plan/partition_value_change_reorganize_range_columns.yml header said "this case wasn't pinned end-to-end" — stale since round 5 added the matching apply / verify_no_drift fixture. Updated to point at testdata/apply/partition_value_change_reorganize_range_columns.yml so the next reader can find the end-to-end pin.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 36 out of 36 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
#A `skipQualifiedIdentifier` のコメント中の literal `` (escape for embedded backtick) が round 18 の `make fix` で typographic left double-quote (U+201C) に書き換えられていて、説明として 読めなくなっていた。リテラルバッククォートを使わない描写 (「embedded backtick is doubled inside the back-ticked identifier」) に書き直して、再度の auto-format でも壊れない 形にした。 #B round 17 で書いた cross-reference `partition_add_range_with_alter_hints_apply` は実際には 存在しないファイル名だった (apply fixture の実名は `testdata/apply/partition_add_range_with_alter_hints.yml`)。 path-prefix 付きで正しい名前に修正。 #C `partitionOpKeywords` には TRUNCATE PARTITION と EXCHANGE PARTITION も入っているが、`TestAppendAlterHints` は REORGANIZE / ADD / DROP / COALESCE しか pin していなかった。 両キーワードの leading-splice をテーブルテストに追加 — keyword リスト上の typo や future refactor が trailing-comma 形式 (MySQL 1064) に silent fallback するのを防ぐ。
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 36 out of 36 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
`partitionListChangedSlotsAreSuperset` only checks that each
*changed* slot's NEW `VALUES IN` is a superset of its OLD set.
It doesn't check whether a value newly added to a changed
slot is already owned by an *unchanged* sibling partition. So
a desired schema like `p0: (1) -> (1, 4)` while unchanged p3
still has `4` slips through the gate, the per-run REORGANIZE
emits `REORGANIZE p0 INTO (p0 IN (1, 4))`, and apply fails
with MySQL Error 1495 "Multiple definition of same constant
in list partitioning" — the desired schema itself is invalid
(two partitions own the same LIST constant), but the operator
only finds out at apply time.
New `findDuplicateListValue` walks `desPO.Definitions` once
and returns the first value (formatted via `sqlparser.String`,
so tuple-safe for LIST COLUMNS) that appears in more than one
partition, plus both partition names. The check runs right
after re-parsing the desired clause, before any of the per-
run / single-span / cascade machinery, so the failure surfaces
at plan time with an actionable message ("Remove the
duplicate from one of the partitions") instead of as a deep
apply-time MySQL error.
Test-first: new
`testdata/plan/partition_list_value_overlap_errors.yml` pins
the error string for the `p0 (1) -> (1, 4)` + unchanged
`p3 (4)` shape. Confirmed failing before the check was added
(no error returned, then apply would have hit Error 1495).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 37 out of 37 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
#A round 20's `findDuplicateListValue` only ran on the modified-table path through `diffPartitions`. Brand-new partitioned tables go through `DiffTables`'s create-table branch (`dt.SQL()`) and skip `diffPartitions` entirely, so a `CREATE TABLE … PARTITION BY LIST …` whose desired schema assigns the same constant to two partitions slipped past plan and failed at apply with MySQL Error 1495 — defeating the purpose of the round-20 plan-time guard. Extracted `validateDesiredListValuesAreDisjoint(fqtn, defs)` as a shared helper (formats the same actionable error from both call sites). `diffPartitions` calls it on its already-parsed `desPO.Definitions`; the create-table branch in `DiffTables` parses the desired clause once, calls this helper, then runs the existing `partitionRequiredColumns` / `uniqueKeyPartitionCoverGap` pair (kept separate so the cover-gap check can grow independently). The two checks together mirror what diffTable enforces on the modified-table side. Test-first: new `testdata/plan/partition_list_value_overlap_on_create_errors.yml` pins the create-path equivalent of `partition_list_value_overlap_errors.yml` (catalog empty, desired CREATE TABLE has overlapping LIST constants). Confirmed failing before the wiring landed. #B CAVEATS.md "Diffs that still error" listed the existing PRIMARY KEY / UNIQUE INDEX cover-gap and SUBPARTITION guards, but didn't document the new `desired LIST partition definitions assign value V to both partition pX and partition pY` plan-time error. Added a bullet that explains the rule (MySQL Error 1495), the coverage (both modified-table and create-table paths), the LIST COLUMNS tuple-safety, and the workaround (remove the duplicate from one partition).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 39 out of 39 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
#A `findDuplicateListValue` keys with `sqlparser.String` so it
works on tuple values for `LIST COLUMNS` too, but only the
scalar `LIST` overlap was pinned (round 20 fixture). New
`partition_list_columns_value_overlap_errors.yml`
exercises the same plan-time error message for tuple
values (`(3, 3)` ends up assigned to both p0 and p1) so a
regression in tuple formatting / equality surfaces here
instead of as MySQL Error 1495 at apply time.
#B Apply-side coverage of the `appendHintsTo` leading-position
splice was REORGANIZE + ADD PARTITION only; DROP PARTITION
and COALESCE PARTITION (HASH/KEY shrink) weren't pinned
end-to-end. Added two apply fixtures so MySQL has to accept
the leading-position splice for those keywords too:
- `partition_drop_range_with_alter_hints.yml`: DROP
PARTITION p2020 + ALGORITHM=INPLACE LOCK=NONE under
`--allow-drop=partition`.
- `partition_coalesce_hash_with_alter_hints.yml`: HASH
PARTITIONS 4 -> PARTITIONS 2 under
`--alter-algorithm=COPY --alter-lock=SHARED` (also
`--allow-drop=partition`-gated).
Both run with verify_no_drift so a keyword-specific splice
bug fails the suite instead of escaping to apply-time
1064.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 42 out of 42 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
#A The LIST single-span fallback assumed any leftover-value
shape was safe to auto-generate. It isn't: when desired
removes a `VALUES IN` constant *without* re-assigning it
to another partition, MySQL silently DROPs any rows for
that value on REORGANIZE — no apply-time error, no
warning, just data gone (verified against MySQL 8.0.45 in
a probe). Detect the catalog-vs-desired discard at plan
time and gate it behind `--allow-drop=partition`, the
same flag DROP PARTITION / COALESCE PARTITION already use
for partition-level data destruction.
New `findDiscardedListValue` walks the catalog `VALUES IN
(…)` constants once and returns the first one missing
from desired. `sqlparser.String` keying makes it tuple-
safe for `LIST COLUMNS` for free. Without
`--allow-drop=partition`, plan errors with `desired LIST
partition layout discards value V (catalog partition pX);
MySQL silently drops any matching rows on REORGANIZE`.
With the flag, the discard goes through as the operator's
acknowledged data loss.
Test-first:
- `partition_list_value_discard_disallowed.yml` (plan,
no flag) pins the new error.
- `partition_list_value_discard_with_allow.yml` (apply,
`allow_drop: [partition]`) pins that the gate lets the
REORGANIZE through and verify_no_drift confirms MySQL
accepts the resulting layout.
#B CAVEATS.md "Per-partition definition change" claimed
REORGANIZE is unconditionally row-preserving. Updated to
spell out the LIST-discard exception (silent row drop on
value removal) and the new `--allow-drop=partition` gate
so operators don't read the section and assume there's
never a need to think about row safety on REORGANIZE.
#C The RANGE cascade only pulled in `last+1`; it never
verified that the desired-side `VALUES LESS THAN`
sequence is globally monotonic. A diff like
`p0 < 10, p1 < 20` -> `p0 < 25, p1 < 20` is rejected by
MySQL with "VALUES less than value must be strictly
increasing", but vitess's parser accepts it, so the diff
layer was happy to emit a REORGANIZE that can never
apply.
New `validateDesiredRangeMonotonic` walks the desired-side
RANGE definitions once. Integer-literal boundaries are
compared numerically; MAXVALUE is treated as +∞ (allowed
at most once and only as the final partition). Non-
integer / non-literal boundaries (function calls, RANGE
COLUMNS tuples) only surface here on consecutive bytewise
duplicates — deeper ordering for those falls back to
MySQL's own error at apply time, because the diff layer
doesn't have a general expression evaluator.
Test-first:
`partition_range_non_monotonic_errors.yml` pins the
integer-literal case (`p0 < 25, p1 < 20`) with the
new error message. CAVEATS.md "Diffs that still error"
gets a corresponding bullet.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 45 out of 45 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Round 23's `findDiscardedListValue` ran *before* the diff layer decided which path the diff would take, so it hard- errored on two shapes the existing dedicated paths already handle correctly: 1. **Supported LIST DROP PARTITION without `--allow-drop=partition`.** When `p3` is dropped entirely (every value in `p3.VALUES IN (…)` disappears from desired), the existing DROP PARTITION code further down already gates on `--allow-drop=partition` and surfaces a `-- skipped: ALTER TABLE … DROP PARTITION p3;` line for the unauthorised drop. The early discard guard pre-empted that and returned a hard error referencing REORGANIZE instead — wrong path and wrong message. 2. **LIST → HASH (or other) scheme change.** When desired moves the table to a non-LIST strategy, every catalog LIST constant is "discarded" by definition. The strategy mismatch is caught further down by `partitionHeaderEqual` with the dedicated `partition strategy / expression differs` error. The early discard guard pre-empted that too. Fix: move the discard check inside the matched-name-list REORGANIZE branch (which is the only path that actually generates a REORGANIZE that could silently drop rows). Gated additionally on `curPO.Type == ListType` so the check is only reachable when the catalog itself is LIST. The original protection (round 23) for the genuine REORGANIZE- discard shape is preserved — `partition_list_value_discard_*` fixtures still pass — but the supported DROP / strategy- change paths now fall through correctly. Test-first regression pins: - `partition_drop_list_disallowed.yml`: LIST table, p3 dropped, no `--allow-drop=partition` → empty plan + `-- skipped: ALTER TABLE … DROP PARTITION p3;` line. - `partition_scheme_change_list_to_hash_errors.yml`: LIST → HASH → `partition strategy / expression differs` error (not the REORGANIZE-discard message). Both confirmed failing on the round-23 code (REORGANIZE- discard error fired instead of the right path) before the move.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 47 out of 47 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Round 23's `validateDesiredRangeMonotonic` was only wired into `diffPartitions` (modified-table path). Brand-new partitioned tables go through `DiffTables`'s create-table branch (`dt.SQL()` directly, skipping `diffPartitions`), so a `CREATE TABLE … PARTITION BY RANGE …` whose `VALUES LESS THAN` sequence isn't strictly increasing slipped past plan and only surfaced at apply time with MySQL's "VALUES less than value must be strictly increasing" error — defeating the purpose of the round-23 plan-time guard. This is the same shape of gap round 21 closed for the LIST overlap guard. Wire `validateDesiredRangeMonotonic` into the existing create-table partition validation block so both desired-side schema checks (LIST overlap + RANGE monotonic) fire from both paths. The unique-key cover check stays separate because it depends on `dt.Indexes` (the model), not just the parsed partition AST. Test-first: new `testdata/plan/partition_range_non_monotonic_on_create_errors.yml` pins the create-path equivalent of `partition_range_non_monotonic_errors.yml` (catalog empty, desired CREATE TABLE has `p0 < 25, p1 < 20`). Confirmed failing before the wiring landed.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 48 out of 48 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Summary
Builds on PR #50's RANGE/LIST suffix add + subset drop and PR #52's HASH/KEY count grow + shrink. When the catalog and desired partition name lists line up position-by-position (every partition stays in the same slot, every name matches case-insensitively), any per-partition definition difference is now generated as one or more
ALTER TABLE … REORGANIZE PARTITION p_i, p_{i+1}, … INTO (PARTITION p_i …, PARTITION p_{i+1} …, …)statements. The most common shape is aVALUES LESS THAN/VALUES INboundary tweak (e.g.p2020 LESS THAN (2021)→p2020 LESS THAN (2025)), but COMMENT / MAX_ROWS / TABLESPACE and other per-partition options that round-trip through vitess's PartitionDefinition formatter are picked up here too. Two semantic no-ops are intentionally suppressed even though they would otherwise look byte-different: case-only partition-name diffs (pAB → PAB— MySQL identifiers are case-insensitive) and LIST / LIST COLUMNSVALUES IN (…)permutations (the value list is a set semantically, so reordering is a no-op). MySQL redistributes existing rows into the new boundaries (row-preserving), so no--allow-dropgating.How the REORGANIZE statements are sliced depends on the partition strategy because the safety constraints differ:
RANGE / RANGE COLUMNS — one REORGANIZE per run of consecutive changed slots. RANGE values are continuous and bound by per-slot boundaries, so a value can only move to an adjacent slot when its boundary shifts — values can't cross over an unchanged slot in between. A "p1 + p3 boundary edit" 5-partition RANGE table emits two REORGANIZE statements rather than one giant
REORGANIZE p1, p2, p3, p4 …that drags unchanged p2 into the rewrite. Each run additionally extends by one slot when that run's last changed slot'sVALUES LESS THANactually moved AND that slot isn't the final partition (theVALUES LESS THAN <x>of slot i is also the implicit lower bound of slot i+1, so MySQL needs slot i+1 in the REORGANIZE — otherwise it rejects with "VALUES less than value must be strictly increasing"). Per-partition option-only edits don't trigger that cascade, so a metadata-only change stays a one-slot REORGANIZE.LIST / LIST COLUMNS — per-run when no value leaves any changed slot, single span otherwise. The diff layer checks each changed slot's NEW
VALUES IN (…)against its OLD set; when every changed slot's NEW is a superset of OLD (no value removed anywhere), per-run is safe (no value can flow to a different slot, so no MySQL Error 1495 "Multiple definition of same constant in list partitioning" risk) and we emit one REORGANIZE per consecutive run — pure-addition disjoint diffs like "p0 gains 9, p3 gains 10 in a 100-partition table" stay proportional to the diff instead of dragging 98 unchanged in-betweens through a data-moving rewrite. When at least one changed slot drops a value (whether the value moved to another slot or was discarded), fall back to a single span over [first..last]; the matched in-betweens get re-stated unchanged. The check usessqlparser.Stringkeying so it's tuple-safe for LIST COLUMNS for free.Other "drops AND adds both non-empty" shapes (split / merge / reorder, retention roll-forward with disjoint names, interior insert in front of a catch-all) still error out — inferring the right REORGANIZE boundaries from a name-only diff isn't safe. The error message reads
split / merge / reorderto distinguish from the supported per-partition definition branch.The
appendAlterHintsrewrite that injects--alter-algorithm/--alter-lockclauses got a related fix in this PR. MySQL's parser rejects the trailing-comma form on partition operations (REORGANIZE / ADD / DROP / COALESCE / TRUNCATE / EXCHANGE PARTITION) — Error 1064, the comma after the closing)of the partition list (or after the partition_names list) is treated as a new partition_definition. The hint inserter now detects partition ops by skippingALTER TABLE <table-name>(back-tick / qualified-name aware) and matching the alter-spec keyword that follows, then splices the hints in the leading position MySQL accepts (ALTER TABLE t ALGORITHM=…, LOCK=…, REORGANIZE PARTITION …). Column / index / constraint operations keep the trailing-comma format.Behaviour matrix delta vs PR #52
REORGANIZE PARTITION …per run of consecutive changed slots, each run extended by one slot when itslast'sVALUES LESS THANboundary actually moved ANDlastisn't the final partitionVALUES IN⊇ OLDVALUES IN(pure addition)REORGANIZE PARTITION …per run of consecutive changed slots — matched in-betweens stay aloneREORGANIZE PARTITION p_first, …, p_last INTO (…)covering [first..last] (matched in-betweens re-stated unchanged)pAB → PAB) and/or LISTVALUES IN (…)permutationssplit / merge / reorder--alter-algorithm/--alter-lockALTER TABLE t ALGORITHM=…, LOCK=…, <KEYWORD> …) — MySQL rejects the trailing-comma form for partition opsImplementation
diff/partitions.goaddspartitionNameListEqual(positional case-insensitive name compare),partitionDefEqual(case-insensitive name + bytewise body compare viaparser.FormatPartitionDefinition, withtemporarilySortListValuesfoldingVALUES INpermutations as no-ops),partitionValueRangeEqual(boundary-only compare for the RANGE cascade gate, also LIST-permutation-safe viatemporarilySortValueRangeListValuesso it's reusable),partitionListChangedSlotsAreSuperset(the LIST per-run-vs-single-span gate), andformatReorganizeRun(renders one REORGANIZE for one run withmodel.Identquoting on the OLD-name list, vitess's formatter on the INTO bodies, and a precomputedmatchedSlotsmask so cascaded / matched-in-between slots use the catalog's body and changed slots use desired's). The matched-name-list arm ofdiffPartitionswalks both sides exactly once to buildmatchedSlots, then dispatches to RANGE per-run + cascade or LIST per-run-or-single-span; mismatched lists fall through to the unsupported error.parser/partition.goexportsFormatPartitionDefinition(used by the new REORGANIZE generator to render the per-partition body) andParsePartitionClause(re-parses the normalised partition clause back into the AST so the diff layer can inspect individual definitions).diff_all.goreworksappendHintsToso partition operations get the leading-position splice and other ALTER TABLE alter-specs keep the trailing-comma form. The detector skipsALTER TABLE <name>(back-tick / qualified-name aware) and only looks at the alter-spec keyword right after the table name — it can't be fooled by a comment or default literal that mentionsADD PARTITION. Indices are computed on rawstmtso non-ASCII byte-length shifts under ToUpper can't displace the splice point.Tests
partition_value_change_reorganize,partition_value_change_reorganize_list,partition_value_change_reorganize_range_columns,partition_value_change_reorganize_list_columns. Apply runs against MySQL andverify_no_drift: trueconfirms convergence in one apply.partition_value_change_reorganize_minimal_range(scalar RANGE, p1's boundary moves, cascade pulls in p2) andpartition_value_change_reorganize_minimal_range_columns(RANGE COLUMNS tuple-boundary equivalent).partition_value_change_reorganize_range_disjoint(RANGE: p1 + p3 → two REORGANIZEs each with their own cascade extension).partition_value_change_reorganize_list_disjoint(LIST p0 + p3 pure-addition → two per-run REORGANIZEs, p1 / p2 stay alone) andpartition_value_change_reorganize_list_crossflow(LIST value4moves p3 → p0 → single-span REORGANIZE because p3 drops 4 → confirms MySQL accepts the single-span and rejects per-run with Error 1495).partition_value_change_reorganize_cascade_case_only_name(RANGE p1 boundary moves AND cascaded p2 ispB → PBrename → INTO body uses catalog'spB, not desired'sPB).partition_per_partition_option_change_reorganize(LIST, COMMENT-only),partition_per_partition_option_change_reorganize_range(RANGE, COMMENT-only — also pins cascade-NOT-firing),partition_per_partition_option_change_reorganize_range_columns(RANGE COLUMNS, COMMENT-only — same),partition_per_partition_option_change_reorganize_max_rows(LIST, MAX_ROWS-only — confirms the broader claim isn't COMMENT-specific).partition_name_case_only_diff_no_op(pAB → PAB),partition_list_value_permutation_no_op((1,2) → (2,1)),partition_list_columns_value_permutation_no_op(LIST COLUMNS tuple permutation).partition_value_change_reorganize_case_only_name_plus_body(pAB → PAB+(1,2) → (1,2,7)at the same slot — pins that the DROP side names the OLD partition and the INTO side names the NEW one without triggering a perpetual rename loop).partition_value_change_reorganize_alter_hints(REORGANIZE PARTITION) andpartition_add_range_with_alter_hints(ADD PARTITION) — both verify MySQL accepts the leading-position splice. Plus extendedTestAppendAlterHintstable tests indiff_all_internal_test.gocovering all four partition-op keywords, the comment-mentioning-keyword regression, and the back-ticked-non-ASCII-name regression for the splice index.TestDiffPartitionsQuotesReservedPartitionNamesOnReorganizeindiff/partitions_test.go(companion to the existing ADD/DROP quoting pins).partition_mid_mismatch_errors.yml(which pinned the previous "value-change is unsupported" error) renamed topartition_value_change_reorganize.ymland rewritten as the value-change pin.partition_drop_and_add_disjoint_errors.ymlandpartition_maxvalue_catchall_insert_errors.ymlupdated to expect the newsplit / merge / reordererror string — both shapes still fall through because their name lists don't line up.Docs
CAVEATS.md"Partitioning" moves "per-partition definition change" to the generated bullets, with the RANGE-vs-LIST slicing trade-off (per-run + cascade vs. per-run-with-superset-gate vs. single-span) split into separate sub-bullets, plus the operational warning that REORGANIZE is data-moving (matches the HASH/KEY count-change section's wording).AGENTS.md"In scope" / "Not yet implemented" updated to match.TODO.mdentry rewritten — what's left is the split / merge / reorder REORGANIZE shapes, scheme/expression changes, and adding/removing partitioning entirely.Test plan
make lintmake test(8.0)make test-scenario