Skip to content

diff: auto-detect RANGE catch-all interior insert (REORGANIZE pmax)#56

Merged
winebarrel merged 4 commits into
mainfrom
auto-detect-catchall-insert
May 4, 2026
Merged

diff: auto-detect RANGE catch-all interior insert (REORGANIZE pmax)#56
winebarrel merged 4 commits into
mainfrom
auto-detect-catchall-insert

Conversation

@winebarrel

Copy link
Copy Markdown
Owner

Summary

Closes one of the most common "split / merge / reorder" cases that PARTITIONING.md previously sent to manual ALTERs: catalog ends in VALUES LESS THAN MAXVALUE and desired slots one or more new partitions strictly between the catalog prefix and the catch-all (catch-all stays in place with the same body, prefix unchanged). Until now this hit the generic split / merge / reorder plan-time error, even though the shape is unambiguous — the new partitions slice the catch-all's range without changing total coverage. myschema now emits a single ALTER TABLE … REORGANIZE PARTITION pmax INTO (extras…, pmax).

A naive ADD PARTITION would be rejected by MySQL with Error 1481 ("MAXVALUE can only be used in last partition definition") and the manual workaround was a REORGANIZE pmax INTO (new…, pmax) typed out by hand. After this PR the operator just edits the Schemafile and myschema apply does the rewrite.

Trigger envelope (intentionally narrow)

Auto-emit fires only when ALL of:

  • scalar RANGE (RANGE COLUMNS excluded — the monotonicity check punts on tuple boundaries, and vitess can't even round-trip VALUES LESS THAN (MAXVALUE, MAXVALUE) today, so the catalog re-parse fails before this branch is reached);
  • catalog has at least one partition AND the last one is VALUES LESS THAN MAXVALUE;
  • desired's last partition is also VALUES LESS THAN MAXVALUE AND partitionDefEqual to catalog's catch-all (case-insensitive name + byte-equal body);
  • desired has strictly more partitions than catalog (m > n — at least one extra to insert);
  • catalog's prefix [0..n-2] is partitionDefEqual to desired's prefix [0..n-2] for each pair (no compound diffs — body changes on prefix partitions disqualify);
  • extras' (desired[n-1..m-2]) names don't reuse any name already present in catalog (rules out reorder / dup-name desired SQL — vitess accepts duplicate partition names in CREATE TABLE so this guard is real).

Anything outside the envelope falls through to the existing split / merge / reorder error so the operator runs the right ALTER by hand. The new partitions' boundary ordering (strictly increasing, MAXVALUE only at the tail) is enforced upstream by validateDesiredRangeMonotonic (round 23 on PR #53), so non-monotonic inserts surface there with the boundary-ordering error, not as a misfire of this auto-emit.

Implementation

  • diff/partitions.go adds detectCatchAllInteriorInsert (returns the catalog catch-all + the extras when the trigger fires, false otherwise) and formatCatchAllInteriorInsertReorganize (renders the single REORGANIZE statement with model.Ident quoting on the OLD-name list and the catalog's catch-all body in INTO so case-only renames don't leak).
  • The new branch sits between the existing matched-name-list (per-partition definition rewrite) path and the existing partitionByNameOrderPreserving (split / merge / reorder error) path, so it's purely additive — every other code path through diffPartitions is byte-for-byte unchanged.
  • The LIST VALUES IN (DEFAULT) equivalent isn't covered here; that would need a parallel detector keyed on InType plus a different boundary-coverage proof (out of scope).

Tests

12 new YAML fixtures pin the spec coverage matrix. All run as part of make test:

Positive (auto-emits, apply + verify_no_drift):

  • partition_catchall_insert_one.yml — canonical 1-extra case (with pre-existing rows in pmax that get redistributed correctly).
  • partition_catchall_insert_multiple.yml — multiple extras packed into a single REORGANIZE.
  • partition_catchall_insert_only_catchall.yml — catalog has only the catch-all (n=1 boundary of the prefix-equal check).
  • partition_catchall_insert_case_only_rename.yml — case-only catch-all name diff (pmax vs PMAX) still triggers via partitionDefEqual's EqualFold; OLD-name list uses catalog's casing.

Negative — auto-detect must NOT fire, existing error preserved (plan):

  • partition_interior_insert_no_catchall_errors.yml — catalog has no MAXVALUE.
  • partition_catchall_drop_with_replacement_errors.yml — desired drops the catch-all entirely.
  • partition_catchall_renamed_errors.yml — catch-all name differs beyond case.
  • partition_catchall_body_change_with_insert_errors.yml — catch-all body differs (strict spec, no compound diffs).
  • partition_catchall_prefix_body_change_with_insert_errors.yml — prefix partition body differs.
  • partition_interior_insert_not_before_catchall_errors.yml — insert NOT directly before catch-all.
  • partition_catchall_insert_extras_name_overlap_errors.yml — extras' name reuses a catalog name (vitess accepts duplicate partition names in CREATE TABLE so this guard is real).

Validation order (existing check fires before auto-detect):

  • partition_catchall_insert_non_monotonic_errors.ymlvalidateDesiredRangeMonotonic catches the bad boundary first; the auto-detect never runs.

partition_maxvalue_catchall_insert_errors.yml (which pinned the now-replaced "split / merge / reorder" behaviour for the canonical case) is removed — partition_catchall_insert_one.yml covers the same shape with the new auto-emit assertion.

All 78 existing partition fixtures + 7 partition Go-level tests still pass — every other partition path is unaffected.

Docs

  • PARTITIONING.md:
    • "Suffix add — RANGE / LIST" caveat: rewrite the catch-all paragraph from "fails with split / merge / reorder; drop the catch-all by hand" to "RANGE auto-handled (see Catch-all interior insert below); LIST DEFAULT not auto-handled".
    • New "Catch-all interior insert (RANGE)" sub-section under "Generated diffs" describing the trigger envelope, the SQL shape, the row-redistribution semantics, the MySQL Error 1481 it sidesteps, and the LIST-DEFAULT not-yet-covered limitation.
    • "Diffs that still error → Split / merge / reorder": removed "interior inserts in front of a catch-all" (now auto-handled) and added precise sub-bullets for the cases that still fall through (no catalog catch-all, mid-prefix insert, LIST DEFAULT, compound diffs on top of an insert).
  • diff/partitions.go doc comments updated to match.
  • CAVEATS.md / AGENTS.md / TODO.md already point at PARTITIONING.md — no edits needed.

Test plan

  • make lint
  • make test (8.0)
  • make test-scenario

winebarrel added 2 commits May 4, 2026 15:08
Catalog ends in `VALUES LESS THAN MAXVALUE` and desired slots
one or more new partitions strictly between the catalog
prefix and the catch-all (catch-all stays in place with the
same body, prefix unchanged). Until now this hit the generic
"split / merge / reorder" plan-time error and forced the
operator to run REORGANIZE PARTITION by hand. The shape is
unambiguous though — the new partitions slice the catch-all's
range without changing total coverage — so myschema can
auto-emit the right REORGANIZE.

`detectCatchAllInteriorInsert` recognises the trigger envelope
and `formatCatchAllInteriorInsertReorganize` builds the
single-statement REORGANIZE that MySQL accepts (Error 1481
on naive ADD PARTITION and Error 1520 on narrowing both
stay clear because total coverage is preserved).

Trigger is intentionally narrow:
  - scalar RANGE only (RANGE COLUMNS punted: vitess can't
    round-trip `VALUES LESS THAN (MAXVALUE, MAXVALUE)`, and
    the monotonicity check punts on tuple boundaries anyway);
  - catch-all body byte-equal between catalog and desired
    (case-only name folding is allowed via partitionDefEqual);
  - catalog prefix `[0..n-2]` byte-equal to desired prefix
    (no compound diffs — body changes on prefix partitions
    disqualify);
  - extras' names don't collide with catalog (rules out
    reorder / dup-name desired SQL).
Anything outside the envelope falls through to the existing
split / merge / reorder error so the operator runs the right
ALTER by hand.

Spec coverage (12 fixtures total):

Positive (auto-emits, apply + verify_no_drift):
  - partition_catchall_insert_one.yml — canonical 1-extra
    case (incl. pre-existing rows in pmax that redistribute);
  - partition_catchall_insert_multiple.yml — multiple extras
    in one REORGANIZE;
  - partition_catchall_insert_only_catchall.yml — catalog
    has only the catch-all (n=1 boundary of prefix-equal
    check);
  - partition_catchall_insert_case_only_rename.yml —
    case-only catch-all name diff (`pmax` vs `PMAX`) still
    triggers via `partitionDefEqual`'s EqualFold.

Negative — auto-detect must NOT fire, existing error
preserved (plan):
  - partition_interior_insert_no_catchall_errors.yml —
    catalog has no MAXVALUE (T3);
  - partition_catchall_drop_with_replacement_errors.yml —
    desired drops catch-all entirely (T4 drop arm);
  - partition_catchall_renamed_errors.yml — catch-all name
    differs beyond case (T4 rename arm);
  - partition_catchall_body_change_with_insert_errors.yml —
    catch-all body differs (T4 body arm — strict spec, no
    compound diffs);
  - partition_catchall_prefix_body_change_with_insert_errors.yml
    — prefix partition body differs (T5);
  - partition_interior_insert_not_before_catchall_errors.yml
    — insert NOT directly before catch-all (T5 position arm);
  - partition_catchall_insert_extras_name_overlap_errors.yml
    — extras' name reuses a catalog name (T7; vitess accepts
    duplicate partition names in CREATE TABLE so this guard
    is real).

Validation order (existing check fires before auto-detect):
  - partition_catchall_insert_non_monotonic_errors.yml —
    `validateDesiredRangeMonotonic` (round 23) catches the
    bad boundary first.

Removed `partition_maxvalue_catchall_insert_errors.yml` (it
pinned the now-replaced "split / merge / reorder" behaviour
for the canonical interior-insert case — the new positive
fixture `partition_catchall_insert_one.yml` covers the same
shape with the new auto-emit assertion).
…detect

The previous commit added the auto-detect for the RANGE catch-
all interior insert shape but didn't update the user-facing /
internal docs that still framed this case as an unsupported
"split / merge / reorder" error. Bring everything in line:

PARTITIONING.md:

- "Suffix add — RANGE / LIST" caveat: rewrite the catch-all
  paragraph from "fails with split / merge / reorder; drop the
  catch-all by hand" to "RANGE auto-handled, see Catch-all
  interior insert (RANGE) below; LIST DEFAULT not auto-handled".

- New "Catch-all interior insert (RANGE)" sub-section under
  "Generated diffs" describing the trigger envelope (scalar
  RANGE, catch-all body byte-equal, prefix byte-equal, extras'
  names new), the SQL shape, the row-redistribution semantics,
  the MySQL Error 1481 it sidesteps, and the LIST-DEFAULT
  not-yet-covered limitation.

- "Diffs that still error → Split / merge / reorder": removed
  "interior inserts in front of a catch-all" from the bullet
  list (it's now auto-handled) and added more precise sub-
  bullets for the cases that *still* fall through (no catalog
  catch-all, mid-prefix insert, LIST DEFAULT, compound diffs
  on top of an insert).

diff/partitions.go:

- `diffPartitions` doc comment's v1-scope listing: added the
  catch-all interior insert as a generated case alongside
  suffix-add / subset-drop / per-partition definition rewrite.

- Inline "drops AND adds both non-empty" outcome listing in
  `diffPartitions`: split the previous "any other shape →
  error" bullet into two — first the catch-all auto-detect
  (handled by `detectCatchAllInteriorInsert` further down),
  then the genuine "any other shape → error" residual.

No behaviour change — these are documentation updates only.
`make lint` clean, full `go test -p 1 ./...` still green.
Copilot AI review requested due to automatic review settings May 4, 2026 06:15
@codecov

codecov Bot commented May 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 29.16667% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.16%. Comparing base (7a182a0) to head (c3c1894).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
diff/partitions.go 29.16% 32 Missing and 2 partials ⚠️

❌ Your patch status has failed because the patch coverage (29.16%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #56      +/-   ##
==========================================
- Coverage   71.83%   71.16%   -0.67%     
==========================================
  Files          28       28              
  Lines        3014     3062      +48     
==========================================
+ Hits         2165     2179      +14     
- Misses        615      647      +32     
- Partials      234      236       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an auto-detect + rewrite for a common RANGE-partitioning diff shape: inserting new partitions immediately before an existing MAXVALUE catch-all, emitting a single REORGANIZE PARTITION <catch-all> INTO (extras…, <catch-all>) instead of falling through to the generic split/merge/reorder error.

Changes:

  • Add RANGE catch-all interior-insert detection and REORGANIZE formatting in diffPartitions.
  • Extend partitioning documentation to describe the new supported shape and remaining limitations.
  • Replace the old “errors” fixture with new apply/plan YAML fixtures covering positive/negative cases.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
diff/partitions.go Implements catch-all interior insert detection + SQL generation for scalar RANGE.
PARTITIONING.md Documents the new auto-handled RANGE catch-all interior insert behavior and limits.
testdata/plan/partition_maxvalue_catchall_insert_errors.yml Removes the fixture that pinned the now-obsolete error behavior for the canonical case.
testdata/plan/partition_interior_insert_not_before_catchall_errors.yml Pins that mid-prefix inserts still fall through to split/merge/reorder.
testdata/plan/partition_interior_insert_no_catchall_errors.yml Pins that interior inserts without a catalog MAXVALUE catch-all still error.
testdata/plan/partition_catchall_renamed_errors.yml Pins that catch-all rename+insert does not auto-detect.
testdata/plan/partition_catchall_prefix_body_change_with_insert_errors.yml Pins that compound diffs (prefix body change + insert) do not auto-detect.
testdata/plan/partition_catchall_insert_non_monotonic_errors.yml Pins validation order: monotonicity error should occur before auto-detect.
testdata/plan/partition_catchall_insert_extras_name_overlap_errors.yml Pins that extra-name overlap with catalog prevents auto-detect and errors instead.
testdata/plan/partition_catchall_drop_with_replacement_errors.yml Pins that dropping the catch-all (no MAXVALUE tail) does not auto-detect.
testdata/plan/partition_catchall_body_change_with_insert_errors.yml Pins that catch-all body changes disqualify auto-detect.
testdata/apply/partition_catchall_insert_only_catchall.yml Apply fixture for n=1 (catalog has only catch-all) edge case.
testdata/apply/partition_catchall_insert_one.yml Apply fixture for canonical single extra partition insertion.
testdata/apply/partition_catchall_insert_multiple.yml Apply fixture for multiple extras packed into one REORGANIZE.
testdata/apply/partition_catchall_insert_case_only_rename.yml Apply fixture ensuring case-only catch-all name diffs still auto-detect.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/partitions.go
Comment thread diff/partitions.go Outdated
Both Copilot comments flagged the same gap: T7 (extras' name
uniqueness) only checked extras vs catalog names, not extras
vs each other. So a desired CREATE TABLE that copy-paste-
typoed two extras with the same name would slip past the
auto-detect and produce a `REORGANIZE pmax INTO (p_new …,
p_new …, pmax …)` statement MySQL rejects with a duplicate-
partition-name error at apply time.

Fix: extend the catalog-names map into a single `seenNames`
set seeded with catalog names, then walk extras and reject if
any extra's lowercased name is already present. Catches the
catalog-overlap arm AND the extras-internal-duplicate arm in
one pass with the same MySQL identifier case-insensitivity.

Test-first: new
`testdata/plan/partition_catchall_insert_extras_internal_dup_errors.yml`
pins the duplicate-extras shape (`p_new` listed twice in
desired) — falls through to the existing split / merge /
reorder error instead of auto-emitting the bad REORGANIZE.
Confirmed failing on the unfixed code via probe (myschema
emitted the duplicate-name REORGANIZE) before the seenNames
extension landed.

Doc comments on `detectCatchAllInteriorInsert` and the
trigger-envelope summary above its callsite updated to spell
out both arms of the uniqueness check explicitly.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diff/partitions.go
Comment thread diff/partitions.go Outdated
Comment thread PARTITIONING.md Outdated
Three doc / wording nits:

#1 / #2 `diff/partitions.go` had three places spelling the
   emitted SQL as `REORGANIZE pmax INTO ...` (omitting the
   `PARTITION` keyword). MySQL's actual grammar — and the
   string we emit — is `REORGANIZE PARTITION pmax INTO ...`.
   Updated all three sites (overview comment, inline outcome
   listing, extras-internal-dup explainer) to match. Found
   one more occurrence the reviewer didn't flag and fixed
   it too while I was at it.

#3 PARTITIONING.md "Catch-all interior insert (RANGE)"
   trigger envelope listed the extras-uniqueness rule as
   "names don't reuse any name already present in catalog".
   The round-1 fix to PR #56 extended that to also reject
   duplicates within the extras slice itself, but the doc
   bullet wasn't updated. Reworded to "unique — both vs
   catalog names AND within the extras slice itself" so the
   doc matches the implementation.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@winebarrel winebarrel merged commit a7f3c68 into main May 4, 2026
8 of 9 checks passed
@winebarrel winebarrel deleted the auto-detect-catchall-insert branch May 4, 2026 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants