parser/catalog/diff: partition round-trip + drift-detect (v1)#49
Conversation
v1 partitioning support: dump → plan reports no diff for a partitioned table, plan errors out (rather than emit ALTERs) when the partition layout differs between desired and catalog. ADD/DROP/TRUNCATE/REORGANIZE PARTITION generation is a future PR; users manage partition changes by hand for now and the next plan reconverges. - model.Table.Partition *string carries the canonical, vitess- formatted PARTITION BY clause. nil = not partitioned. Both sides go through parser.NormalizePartitionOption so byte-equal comparison is meaningful. - parser/partition.go normalises a *sqlparser.PartitionOption: trims the leading newline, lower-cases function-name and column-reference identifiers (vitess preserves the user's YEAR / dt casing but MySQL's SHOW CREATE TABLE always emits year / dt in lower case), and strips the per-partition `engine <name>` option that 8.0+ adds even when the table-level engine alone would suffice. - parser/partition.go also exposes ExtractPartitionFromShowCreate, which lifts the `/*!50100 PARTITION BY … */` block out of a SHOW CREATE TABLE result, wraps it onto a trivial CREATE TABLE skeleton, re-parses with vitess, and runs the same normaliser the desired-side path uses. That symmetry is the whole reason the round-trip is bytewise. - catalog/tables.go gets loadPartitions: list partitioned tables via information_schema.PARTITIONS (DISTINCT, one row per table), then SHOW CREATE TABLE each and pull the clause. Two-phase keeps cost proportional to partitioned-table count rather than total-table count. - diff/tables.go's diffTable errors when current.Partition != desired.Partition with a clear "manage by hand" message. Emitted before the column / index passes so the user gets the partition error immediately, not after a long plan. - model.Table.SQL appends Partition on a fresh line so dump output reads cleanly. SUBPARTITION BY is intentionally out of scope for v1: a desired-side SUBPARTITION fails at parser parseCreateTable, and a catalog-side SUBPARTITION is rejected by the ExtractPartitionFromShowCreate guard so myschema won't try to manage it. The user must drop SUBPARTITION manually before bringing the table under management. Tests: - parser unit (partition_test.go): RANGE / LIST / HASH / KEY / RANGE COLUMNS round-trip through both pipelines and produce byte-equal output, absent partition leaves model nil, SHOW CREATE TABLE without partition block returns "", and both directions reject SUBPARTITION. - end-to-end YAML: dump / plan / apply fixtures pin the no-diff round-trip; a separate plan fixture pins the "partition difference errors" path with a clear message. Docs: - CAVEATS.md "Partitioning is round-trip only in v1": full what-works / what-errors writeup + SUBPARTITION exclusion + rationale (RANGE/LIST vs HASH/KEY differ in the ALTER syntax, so diff-gen is its own PR). - AGENTS.md "In scope": catalog reader now mentions partition clauses; "Not yet implemented" entry for partitions rewritten to point at CAVEATS. - TODO.md: entry rewritten to scope the *generation* work, noting SUBPARTITION is not coming. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #49 +/- ##
==========================================
- Coverage 74.47% 73.71% -0.76%
==========================================
Files 28 29 +1
Lines 2527 2610 +83
==========================================
+ Hits 1882 1924 +42
- Misses 470 499 +29
- Partials 175 187 +12 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds v1 partitioning round-trip + drift-detect support so partitioned tables can be dumped and re-planned without diffs, while any partition drift triggers a hard error (partition ALTER generation remains out of scope).
Changes:
- Model and render
PARTITION BY …clauses viamodel.Table.Partitionand(*Table).SQL(). - Parse/normalize partition clauses from desired SQL and from catalog
SHOW CREATE TABLE, and error out on partition drift (including explicit SUBPARTITION rejection). - Add parser unit tests + end-to-end YAML fixtures, and update docs/TODO/AGENTS to reflect v1 scope.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| testdata/plan/partition_no_diff.yml | Pins “no diff” behavior for unchanged partitioned tables. |
| testdata/plan/partition_diff_errors.yml | Pins plan-time error behavior when partitions differ. |
| testdata/dump/partition_range.yml | Pins dump output including canonical partition clause. |
| testdata/apply/partition_no_diff.yml | Pins apply no-op + verify_no_drift for partitioned tables. |
| parser/partition_test.go | Unit tests covering round-trip normalization + SUBPARTITION rejection. |
| parser/partition.go | Implements partition normalization and SHOW CREATE TABLE clause extraction. |
| parser/parser.go | Stores canonical partition clause on parsed tables; rejects SUBPARTITION. |
| model/table.go | Adds Table.Partition and appends it when rendering CREATE TABLE SQL. |
| diff/tables.go | Errors early when desired vs catalog partition clauses differ. |
| catalog/tables.go | Loads partition clauses for partitioned tables via info_schema + SHOW CREATE TABLE. |
| TODO.md | Re-scopes partition work to “diff generation” only (future). |
| CAVEATS.md | Documents v1 partition behavior and explicit non-support boundaries. |
| AGENTS.md | Updates supported/not-yet-implemented lists to include partition round-trip and drift-detect. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Four nits — one is a real bug, three are doc / shadowing fixes:
- ExtractPartitionFromShowCreate sliced from the *first* `/*!`
to the *last* `*/`. SHOW CREATE TABLE on MySQL 8.0+ can emit
earlier versioned-comment blocks (e.g. `/*!80016
ENCRYPTION='N' */`) before the partition one, in which case
the body would start with ENCRYPTION='N' and the function
would return ("", nil) — silently dropping the partition
metadata. Switch to LastIndex so we always grab the trailing
block (the partition clause is always last when present), and
pin the fix with a regression test that mixes ENCRYPTION + a
RANGE partition.
- Inline comment in NormalizePartitionOption mentioned a
fictional CreateSQL method. Tables render via
`(*model.Table).SQL`; updated.
- model.Table.Partition's field comment claimed "both sides
normalise via sqlparser.String(*sqlparser.PartitionOption)",
but the implementation goes through
parser.NormalizePartitionOption (which adds identifier
lower-casing, leading-newline trim, and per-partition engine
strip on top of vitess's formatter). Spell out what
"canonical" actually means.
- Loop-local clause copy in catalog/tables.go::loadPartitions
was named `c`, shadowing the `*Catalog` receiver. Rename to
`clauseCopy`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Doc-only nit. ExtractPartitionFromShowCreate's comment said it extracts "including any SUBPARTITION BY …", but the function deliberately errors when a SUBPARTITION clause is present (SUBPARTITION is out of scope for v1 — CAVEATS.md "Partitioning is round-trip only"). Fix the comment so future readers don't expect SUBPARTITION pass-through. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add a TestParseSQLPartitionRoundTrip case for `LIST COLUMNS(...)`
with `VALUES IN ('A', 'B')`. Pins two things at once:
- LIST COLUMNS(...) is on the supported list but wasn't actually
exercised by the round-trip test.
- The normaliser only lower-cases function-name and column-
reference identifiers — string literals (`'A'`, `'B'`) must
round-trip case-preserved, because flipping `'A'` to `'a'`
would change the value MySQL stores. Without an explicit test
a future change to the rewriter could quietly break that
guarantee.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Summary
v1 partitioning support.
dump → planreports no diff for an unchanged partitioned table; any partition difference between desired and catalog errors out with a clear "manage by hand" message. GeneratingALTER TABLE … {ADD|DROP|TRUNCATE|REORGANIZE} PARTITIONfrom the diff is a future PR.Per scope decision:
SUBPARTITION BYis intentionally out of scope — desired-side fails at parse time, catalog-side is rejected on read.Supported
PARTITION BY {RANGE|LIST|HASH|KEY|RANGE COLUMNS|LIST COLUMNS} (...)— read from catalog viaSHOW CREATE TABLE, parsed from desired SQL, normalised through one vitess pipeline so byte-equal comparison works.dumpemits the canonicalpartition by …clause as part of the CREATE TABLE.planreports no diff when desired matches catalog.planerrors withpartition definition differs between catalog and desired SQL; partition diffs are not yet generated by myschema (v1 supports round-trip only)when they don't.Out of scope (v1)
INTO (PARTITION …)reorganise, HASH/KEY =COALESCE PARTITIONfor shrinks, etc.) and the safe ordering depends on data layout — its own PR.SUBPARTITION BY …. Desired-side parse error; catalog-sideExtractPartitionFromShowCreaterejects the table so myschema won't try to manage it.Implementation
model.Table.Partition *stringcarries the canonical, vitess-formatted clause.parser/partition.go::NormalizePartitionOptionstrips per-partitionengine <name>(8.0+ always emitsENGINE = InnoDBon every partition), lower-cases function-name and column-reference identifiers (vitess preserves user casing, MySQL output is lower case), and trims the leading newline so(*Table).SQL()can concatenate cleanly.parser/partition.go::ExtractPartitionFromShowCreatelifts the/*!50100 PARTITION BY … */block from a SHOW CREATE TABLE result, wraps it onto aCREATE TABLE _t (id INT) <clause>skeleton, re-parses with vitess, and runs the same normaliser the desired-side path uses. Symmetry = bytewise equality.catalog/tables.go::loadPartitionsis two-phase: list partitioned tables viainformation_schema.PARTITIONS(DISTINCT, one row per table), thenSHOW CREATE TABLEeach. Cost scales with partitioned-table count, not total-table count.diff/tables.go::diffTableerrors whencurrent.Partition != desired.Partitionbefore the column / index passes so the user sees the partition error immediately.model.Table.SQL()appends the partition clause on a fresh line.Tests
partition_test.go): RANGE / LIST / HASH / KEY / RANGE COLUMNS round-trip via both pipelines and produce byte-equal output, absent partition leaves model nil,SHOW CREATE TABLEwithout partition block returns"", both extractor and ParseSQL reject SUBPARTITION.testdata/dump/partition_range.yml— dump emits canonical clause.testdata/plan/partition_no_diff.yml— desired matches catalog → no plan output.testdata/plan/partition_diff_errors.yml— adding a partition errors with the documented message.testdata/apply/partition_no_diff.ymlwithverify_no_drift: true— apply is no-op and the next plan is clean.Docs
CAVEATS.md"Partitioning is round-trip only in v1": full what-works / what-errors writeup + SUBPARTITION exclusion + rationale.AGENTS.md"In scope": catalog reader now mentions partition clauses; the "Not yet implemented" entry for partitions points at the CAVEATS section and the diff-generation gap.TODO.md: entry rewritten to scope the generation work, noting SUBPARTITION is not coming.Test plan
make lintmake test(8.0)