diff: auto-detect RANGE catch-all interior insert (REORGANIZE pmax)#56
Conversation
Catalog ends in `VALUES LESS THAN MAXVALUE` and desired slots
one or more new partitions strictly between the catalog
prefix and the catch-all (catch-all stays in place with the
same body, prefix unchanged). Until now this hit the generic
"split / merge / reorder" plan-time error and forced the
operator to run REORGANIZE PARTITION by hand. The shape is
unambiguous though — the new partitions slice the catch-all's
range without changing total coverage — so myschema can
auto-emit the right REORGANIZE.
`detectCatchAllInteriorInsert` recognises the trigger envelope
and `formatCatchAllInteriorInsertReorganize` builds the
single-statement REORGANIZE that MySQL accepts (Error 1481
on naive ADD PARTITION and Error 1520 on narrowing both
stay clear because total coverage is preserved).
Trigger is intentionally narrow:
- scalar RANGE only (RANGE COLUMNS punted: vitess can't
round-trip `VALUES LESS THAN (MAXVALUE, MAXVALUE)`, and
the monotonicity check punts on tuple boundaries anyway);
- catch-all body byte-equal between catalog and desired
(case-only name folding is allowed via partitionDefEqual);
- catalog prefix `[0..n-2]` byte-equal to desired prefix
(no compound diffs — body changes on prefix partitions
disqualify);
- extras' names don't collide with catalog (rules out
reorder / dup-name desired SQL).
Anything outside the envelope falls through to the existing
split / merge / reorder error so the operator runs the right
ALTER by hand.
Spec coverage (12 fixtures total):
Positive (auto-emits, apply + verify_no_drift):
- partition_catchall_insert_one.yml — canonical 1-extra
case (incl. pre-existing rows in pmax that redistribute);
- partition_catchall_insert_multiple.yml — multiple extras
in one REORGANIZE;
- partition_catchall_insert_only_catchall.yml — catalog
has only the catch-all (n=1 boundary of prefix-equal
check);
- partition_catchall_insert_case_only_rename.yml —
case-only catch-all name diff (`pmax` vs `PMAX`) still
triggers via `partitionDefEqual`'s EqualFold.
Negative — auto-detect must NOT fire, existing error
preserved (plan):
- partition_interior_insert_no_catchall_errors.yml —
catalog has no MAXVALUE (T3);
- partition_catchall_drop_with_replacement_errors.yml —
desired drops catch-all entirely (T4 drop arm);
- partition_catchall_renamed_errors.yml — catch-all name
differs beyond case (T4 rename arm);
- partition_catchall_body_change_with_insert_errors.yml —
catch-all body differs (T4 body arm — strict spec, no
compound diffs);
- partition_catchall_prefix_body_change_with_insert_errors.yml
— prefix partition body differs (T5);
- partition_interior_insert_not_before_catchall_errors.yml
— insert NOT directly before catch-all (T5 position arm);
- partition_catchall_insert_extras_name_overlap_errors.yml
— extras' name reuses a catalog name (T7; vitess accepts
duplicate partition names in CREATE TABLE so this guard
is real).
Validation order (existing check fires before auto-detect):
- partition_catchall_insert_non_monotonic_errors.yml —
`validateDesiredRangeMonotonic` (round 23) catches the
bad boundary first.
Removed `partition_maxvalue_catchall_insert_errors.yml` (it
pinned the now-replaced "split / merge / reorder" behaviour
for the canonical interior-insert case — the new positive
fixture `partition_catchall_insert_one.yml` covers the same
shape with the new auto-emit assertion).
…detect The previous commit added the auto-detect for the RANGE catch- all interior insert shape but didn't update the user-facing / internal docs that still framed this case as an unsupported "split / merge / reorder" error. Bring everything in line: PARTITIONING.md: - "Suffix add — RANGE / LIST" caveat: rewrite the catch-all paragraph from "fails with split / merge / reorder; drop the catch-all by hand" to "RANGE auto-handled, see Catch-all interior insert (RANGE) below; LIST DEFAULT not auto-handled". - New "Catch-all interior insert (RANGE)" sub-section under "Generated diffs" describing the trigger envelope (scalar RANGE, catch-all body byte-equal, prefix byte-equal, extras' names new), the SQL shape, the row-redistribution semantics, the MySQL Error 1481 it sidesteps, and the LIST-DEFAULT not-yet-covered limitation. - "Diffs that still error → Split / merge / reorder": removed "interior inserts in front of a catch-all" from the bullet list (it's now auto-handled) and added more precise sub- bullets for the cases that *still* fall through (no catalog catch-all, mid-prefix insert, LIST DEFAULT, compound diffs on top of an insert). diff/partitions.go: - `diffPartitions` doc comment's v1-scope listing: added the catch-all interior insert as a generated case alongside suffix-add / subset-drop / per-partition definition rewrite. - Inline "drops AND adds both non-empty" outcome listing in `diffPartitions`: split the previous "any other shape → error" bullet into two — first the catch-all auto-detect (handled by `detectCatchAllInteriorInsert` further down), then the genuine "any other shape → error" residual. No behaviour change — these are documentation updates only. `make lint` clean, full `go test -p 1 ./...` still green.
Codecov Report❌ Patch coverage is
❌ Your patch status has failed because the patch coverage (29.16%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #56 +/- ##
==========================================
- Coverage 71.83% 71.16% -0.67%
==========================================
Files 28 28
Lines 3014 3062 +48
==========================================
+ Hits 2165 2179 +14
- Misses 615 647 +32
- Partials 234 236 +2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds an auto-detect + rewrite for a common RANGE-partitioning diff shape: inserting new partitions immediately before an existing MAXVALUE catch-all, emitting a single REORGANIZE PARTITION <catch-all> INTO (extras…, <catch-all>) instead of falling through to the generic split/merge/reorder error.
Changes:
- Add RANGE catch-all interior-insert detection and REORGANIZE formatting in
diffPartitions. - Extend partitioning documentation to describe the new supported shape and remaining limitations.
- Replace the old “errors” fixture with new apply/plan YAML fixtures covering positive/negative cases.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| diff/partitions.go | Implements catch-all interior insert detection + SQL generation for scalar RANGE. |
| PARTITIONING.md | Documents the new auto-handled RANGE catch-all interior insert behavior and limits. |
| testdata/plan/partition_maxvalue_catchall_insert_errors.yml | Removes the fixture that pinned the now-obsolete error behavior for the canonical case. |
| testdata/plan/partition_interior_insert_not_before_catchall_errors.yml | Pins that mid-prefix inserts still fall through to split/merge/reorder. |
| testdata/plan/partition_interior_insert_no_catchall_errors.yml | Pins that interior inserts without a catalog MAXVALUE catch-all still error. |
| testdata/plan/partition_catchall_renamed_errors.yml | Pins that catch-all rename+insert does not auto-detect. |
| testdata/plan/partition_catchall_prefix_body_change_with_insert_errors.yml | Pins that compound diffs (prefix body change + insert) do not auto-detect. |
| testdata/plan/partition_catchall_insert_non_monotonic_errors.yml | Pins validation order: monotonicity error should occur before auto-detect. |
| testdata/plan/partition_catchall_insert_extras_name_overlap_errors.yml | Pins that extra-name overlap with catalog prevents auto-detect and errors instead. |
| testdata/plan/partition_catchall_drop_with_replacement_errors.yml | Pins that dropping the catch-all (no MAXVALUE tail) does not auto-detect. |
| testdata/plan/partition_catchall_body_change_with_insert_errors.yml | Pins that catch-all body changes disqualify auto-detect. |
| testdata/apply/partition_catchall_insert_only_catchall.yml | Apply fixture for n=1 (catalog has only catch-all) edge case. |
| testdata/apply/partition_catchall_insert_one.yml | Apply fixture for canonical single extra partition insertion. |
| testdata/apply/partition_catchall_insert_multiple.yml | Apply fixture for multiple extras packed into one REORGANIZE. |
| testdata/apply/partition_catchall_insert_case_only_rename.yml | Apply fixture ensuring case-only catch-all name diffs still auto-detect. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Both Copilot comments flagged the same gap: T7 (extras' name uniqueness) only checked extras vs catalog names, not extras vs each other. So a desired CREATE TABLE that copy-paste- typoed two extras with the same name would slip past the auto-detect and produce a `REORGANIZE pmax INTO (p_new …, p_new …, pmax …)` statement MySQL rejects with a duplicate- partition-name error at apply time. Fix: extend the catalog-names map into a single `seenNames` set seeded with catalog names, then walk extras and reject if any extra's lowercased name is already present. Catches the catalog-overlap arm AND the extras-internal-duplicate arm in one pass with the same MySQL identifier case-insensitivity. Test-first: new `testdata/plan/partition_catchall_insert_extras_internal_dup_errors.yml` pins the duplicate-extras shape (`p_new` listed twice in desired) — falls through to the existing split / merge / reorder error instead of auto-emitting the bad REORGANIZE. Confirmed failing on the unfixed code via probe (myschema emitted the duplicate-name REORGANIZE) before the seenNames extension landed. Doc comments on `detectCatchAllInteriorInsert` and the trigger-envelope summary above its callsite updated to spell out both arms of the uniqueness check explicitly.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Three doc / wording nits: #1 / #2 `diff/partitions.go` had three places spelling the emitted SQL as `REORGANIZE pmax INTO ...` (omitting the `PARTITION` keyword). MySQL's actual grammar — and the string we emit — is `REORGANIZE PARTITION pmax INTO ...`. Updated all three sites (overview comment, inline outcome listing, extras-internal-dup explainer) to match. Found one more occurrence the reviewer didn't flag and fixed it too while I was at it. #3 PARTITIONING.md "Catch-all interior insert (RANGE)" trigger envelope listed the extras-uniqueness rule as "names don't reuse any name already present in catalog". The round-1 fix to PR #56 extended that to also reject duplicates within the extras slice itself, but the doc bullet wasn't updated. Reworded to "unique — both vs catalog names AND within the extras slice itself" so the doc matches the implementation.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Summary
Closes one of the most common "split / merge / reorder" cases that PARTITIONING.md previously sent to manual ALTERs: catalog ends in
VALUES LESS THAN MAXVALUEand desired slots one or more new partitions strictly between the catalog prefix and the catch-all (catch-all stays in place with the same body, prefix unchanged). Until now this hit the genericsplit / merge / reorderplan-time error, even though the shape is unambiguous — the new partitions slice the catch-all's range without changing total coverage. myschema now emits a singleALTER TABLE … REORGANIZE PARTITION pmax INTO (extras…, pmax).A naive
ADD PARTITIONwould be rejected by MySQL with Error 1481 ("MAXVALUE can only be used in last partition definition") and the manual workaround was aREORGANIZE pmax INTO (new…, pmax)typed out by hand. After this PR the operator just edits the Schemafile andmyschema applydoes the rewrite.Trigger envelope (intentionally narrow)
Auto-emit fires only when ALL of:
RANGE COLUMNSexcluded — the monotonicity check punts on tuple boundaries, and vitess can't even round-tripVALUES LESS THAN (MAXVALUE, MAXVALUE)today, so the catalog re-parse fails before this branch is reached);VALUES LESS THAN MAXVALUE;VALUES LESS THAN MAXVALUEANDpartitionDefEqualto catalog's catch-all (case-insensitive name + byte-equal body);[0..n-2]ispartitionDefEqualto desired's prefix[0..n-2]for each pair (no compound diffs — body changes on prefix partitions disqualify);desired[n-1..m-2]) names don't reuse any name already present in catalog (rules out reorder / dup-name desired SQL — vitess accepts duplicate partition names in CREATE TABLE so this guard is real).Anything outside the envelope falls through to the existing split / merge / reorder error so the operator runs the right ALTER by hand. The new partitions' boundary ordering (strictly increasing, MAXVALUE only at the tail) is enforced upstream by
validateDesiredRangeMonotonic(round 23 on PR #53), so non-monotonic inserts surface there with the boundary-ordering error, not as a misfire of this auto-emit.Implementation
diff/partitions.goaddsdetectCatchAllInteriorInsert(returns the catalog catch-all + the extras when the trigger fires,falseotherwise) andformatCatchAllInteriorInsertReorganize(renders the single REORGANIZE statement withmodel.Identquoting on the OLD-name list and the catalog's catch-all body in INTO so case-only renames don't leak).partitionByNameOrderPreserving(split / merge / reorder error) path, so it's purely additive — every other code path throughdiffPartitionsis byte-for-byte unchanged.VALUES IN (DEFAULT)equivalent isn't covered here; that would need a parallel detector keyed onInTypeplus a different boundary-coverage proof (out of scope).Tests
12 new YAML fixtures pin the spec coverage matrix. All run as part of
make test:Positive (auto-emits, apply + verify_no_drift):
partition_catchall_insert_one.yml— canonical 1-extra case (with pre-existing rows in pmax that get redistributed correctly).partition_catchall_insert_multiple.yml— multiple extras packed into a single REORGANIZE.partition_catchall_insert_only_catchall.yml— catalog has only the catch-all (n=1 boundary of the prefix-equal check).partition_catchall_insert_case_only_rename.yml— case-only catch-all name diff (pmaxvsPMAX) still triggers viapartitionDefEqual's EqualFold; OLD-name list uses catalog's casing.Negative — auto-detect must NOT fire, existing error preserved (plan):
partition_interior_insert_no_catchall_errors.yml— catalog has no MAXVALUE.partition_catchall_drop_with_replacement_errors.yml— desired drops the catch-all entirely.partition_catchall_renamed_errors.yml— catch-all name differs beyond case.partition_catchall_body_change_with_insert_errors.yml— catch-all body differs (strict spec, no compound diffs).partition_catchall_prefix_body_change_with_insert_errors.yml— prefix partition body differs.partition_interior_insert_not_before_catchall_errors.yml— insert NOT directly before catch-all.partition_catchall_insert_extras_name_overlap_errors.yml— extras' name reuses a catalog name (vitess accepts duplicate partition names in CREATE TABLE so this guard is real).Validation order (existing check fires before auto-detect):
partition_catchall_insert_non_monotonic_errors.yml—validateDesiredRangeMonotoniccatches the bad boundary first; the auto-detect never runs.partition_maxvalue_catchall_insert_errors.yml(which pinned the now-replaced "split / merge / reorder" behaviour for the canonical case) is removed —partition_catchall_insert_one.ymlcovers the same shape with the new auto-emit assertion.All 78 existing partition fixtures + 7 partition Go-level tests still pass — every other partition path is unaffected.
Docs
PARTITIONING.md:diff/partitions.godoc comments updated to match.CAVEATS.md/AGENTS.md/TODO.mdalready point atPARTITIONING.md— no edits needed.Test plan
make lintmake test(8.0)make test-scenario