Skip to content

parser/catalog/diff: partition round-trip + drift-detect (v1)#49

Merged
winebarrel merged 4 commits into
mainfrom
partitioning-roundtrip
May 3, 2026
Merged

parser/catalog/diff: partition round-trip + drift-detect (v1)#49
winebarrel merged 4 commits into
mainfrom
partitioning-roundtrip

Conversation

@winebarrel

Copy link
Copy Markdown
Owner

Summary

v1 partitioning support. dump → plan reports no diff for an unchanged partitioned table; any partition difference between desired and catalog errors out with a clear "manage by hand" message. Generating ALTER TABLE … {ADD|DROP|TRUNCATE|REORGANIZE} PARTITION from the diff is a future PR.

Per scope decision: SUBPARTITION BY is intentionally out of scope — desired-side fails at parse time, catalog-side is rejected on read.

Supported

  • PARTITION BY {RANGE|LIST|HASH|KEY|RANGE COLUMNS|LIST COLUMNS} (...) — read from catalog via SHOW CREATE TABLE, parsed from desired SQL, normalised through one vitess pipeline so byte-equal comparison works.
  • dump emits the canonical partition by … clause as part of the CREATE TABLE.
  • plan reports no diff when desired matches catalog.
  • plan errors with partition definition differs between catalog and desired SQL; partition diffs are not yet generated by myschema (v1 supports round-trip only) when they don't.

Out of scope (v1)

  • Partition diff generation (ADD/DROP/TRUNCATE/REORGANIZE PARTITION). Each strategy needs its own grammar (RANGE/LIST = explicit INTO (PARTITION …) reorganise, HASH/KEY = COALESCE PARTITION for shrinks, etc.) and the safe ordering depends on data layout — its own PR.
  • SUBPARTITION BY …. Desired-side parse error; catalog-side ExtractPartitionFromShowCreate rejects the table so myschema won't try to manage it.

Implementation

  • model.Table.Partition *string carries the canonical, vitess-formatted clause.
  • parser/partition.go::NormalizePartitionOption strips per-partition engine <name> (8.0+ always emits ENGINE = InnoDB on every partition), lower-cases function-name and column-reference identifiers (vitess preserves user casing, MySQL output is lower case), and trims the leading newline so (*Table).SQL() can concatenate cleanly.
  • parser/partition.go::ExtractPartitionFromShowCreate lifts the /*!50100 PARTITION BY … */ block from a SHOW CREATE TABLE result, wraps it onto a CREATE TABLE _t (id INT) <clause> skeleton, re-parses with vitess, and runs the same normaliser the desired-side path uses. Symmetry = bytewise equality.
  • catalog/tables.go::loadPartitions is two-phase: list partitioned tables via information_schema.PARTITIONS (DISTINCT, one row per table), then SHOW CREATE TABLE each. Cost scales with partitioned-table count, not total-table count.
  • diff/tables.go::diffTable errors when current.Partition != desired.Partition before the column / index passes so the user sees the partition error immediately.
  • model.Table.SQL() appends the partition clause on a fresh line.

Tests

  • parser unit (partition_test.go): RANGE / LIST / HASH / KEY / RANGE COLUMNS round-trip via both pipelines and produce byte-equal output, absent partition leaves model nil, SHOW CREATE TABLE without partition block returns "", both extractor and ParseSQL reject SUBPARTITION.
  • end-to-end YAML against MySQL 8.0:
    • testdata/dump/partition_range.yml — dump emits canonical clause.
    • testdata/plan/partition_no_diff.yml — desired matches catalog → no plan output.
    • testdata/plan/partition_diff_errors.yml — adding a partition errors with the documented message.
    • testdata/apply/partition_no_diff.yml with verify_no_drift: true — apply is no-op and the next plan is clean.

Docs

  • CAVEATS.md "Partitioning is round-trip only in v1": full what-works / what-errors writeup + SUBPARTITION exclusion + rationale.
  • AGENTS.md "In scope": catalog reader now mentions partition clauses; the "Not yet implemented" entry for partitions points at the CAVEATS section and the diff-generation gap.
  • TODO.md: entry rewritten to scope the generation work, noting SUBPARTITION is not coming.

Test plan

  • make lint
  • make test (8.0)

v1 partitioning support: dump → plan reports no diff for a
partitioned table, plan errors out (rather than emit ALTERs)
when the partition layout differs between desired and catalog.
ADD/DROP/TRUNCATE/REORGANIZE PARTITION generation is a future
PR; users manage partition changes by hand for now and the next
plan reconverges.

- model.Table.Partition *string carries the canonical, vitess-
  formatted PARTITION BY clause. nil = not partitioned. Both
  sides go through parser.NormalizePartitionOption so byte-equal
  comparison is meaningful.
- parser/partition.go normalises a *sqlparser.PartitionOption:
  trims the leading newline, lower-cases function-name and
  column-reference identifiers (vitess preserves the user's
  YEAR / dt casing but MySQL's SHOW CREATE TABLE always emits
  year / dt in lower case), and strips the per-partition
  `engine <name>` option that 8.0+ adds even when the
  table-level engine alone would suffice.
- parser/partition.go also exposes ExtractPartitionFromShowCreate,
  which lifts the `/*!50100 PARTITION BY … */` block out of a
  SHOW CREATE TABLE result, wraps it onto a trivial CREATE
  TABLE skeleton, re-parses with vitess, and runs the same
  normaliser the desired-side path uses. That symmetry is the
  whole reason the round-trip is bytewise.
- catalog/tables.go gets loadPartitions: list partitioned
  tables via information_schema.PARTITIONS (DISTINCT, one row
  per table), then SHOW CREATE TABLE each and pull the clause.
  Two-phase keeps cost proportional to partitioned-table count
  rather than total-table count.
- diff/tables.go's diffTable errors when current.Partition !=
  desired.Partition with a clear "manage by hand" message.
  Emitted before the column / index passes so the user gets
  the partition error immediately, not after a long plan.
- model.Table.SQL appends Partition on a fresh line so dump
  output reads cleanly.

SUBPARTITION BY is intentionally out of scope for v1: a
desired-side SUBPARTITION fails at parser parseCreateTable, and
a catalog-side SUBPARTITION is rejected by the
ExtractPartitionFromShowCreate guard so myschema won't try to
manage it. The user must drop SUBPARTITION manually before
bringing the table under management.

Tests:
- parser unit (partition_test.go): RANGE / LIST / HASH / KEY /
  RANGE COLUMNS round-trip through both pipelines and produce
  byte-equal output, absent partition leaves model nil,
  SHOW CREATE TABLE without partition block returns "", and
  both directions reject SUBPARTITION.
- end-to-end YAML: dump / plan / apply fixtures pin the no-diff
  round-trip; a separate plan fixture pins the "partition
  difference errors" path with a clear message.

Docs:
- CAVEATS.md "Partitioning is round-trip only in v1": full
  what-works / what-errors writeup + SUBPARTITION exclusion +
  rationale (RANGE/LIST vs HASH/KEY differ in the ALTER
  syntax, so diff-gen is its own PR).
- AGENTS.md "In scope": catalog reader now mentions partition
  clauses; "Not yet implemented" entry for partitions rewritten
  to point at CAVEATS.
- TODO.md: entry rewritten to scope the *generation* work,
  noting SUBPARTITION is not coming.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 3, 2026 09:43
@codecov

codecov Bot commented May 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 50.60241% with 41 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.71%. Comparing base (e17b476) to head (29e616d).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
catalog/tables.go 29.72% 21 Missing and 5 partials ⚠️
parser/partition.go 72.22% 5 Missing and 5 partials ⚠️
model/table.go 0.00% 2 Missing and 1 partial ⚠️
diff/tables.go 0.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #49      +/-   ##
==========================================
- Coverage   74.47%   73.71%   -0.76%     
==========================================
  Files          28       29       +1     
  Lines        2527     2610      +83     
==========================================
+ Hits         1882     1924      +42     
- Misses        470      499      +29     
- Partials      175      187      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds v1 partitioning round-trip + drift-detect support so partitioned tables can be dumped and re-planned without diffs, while any partition drift triggers a hard error (partition ALTER generation remains out of scope).

Changes:

  • Model and render PARTITION BY … clauses via model.Table.Partition and (*Table).SQL().
  • Parse/normalize partition clauses from desired SQL and from catalog SHOW CREATE TABLE, and error out on partition drift (including explicit SUBPARTITION rejection).
  • Add parser unit tests + end-to-end YAML fixtures, and update docs/TODO/AGENTS to reflect v1 scope.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
testdata/plan/partition_no_diff.yml Pins “no diff” behavior for unchanged partitioned tables.
testdata/plan/partition_diff_errors.yml Pins plan-time error behavior when partitions differ.
testdata/dump/partition_range.yml Pins dump output including canonical partition clause.
testdata/apply/partition_no_diff.yml Pins apply no-op + verify_no_drift for partitioned tables.
parser/partition_test.go Unit tests covering round-trip normalization + SUBPARTITION rejection.
parser/partition.go Implements partition normalization and SHOW CREATE TABLE clause extraction.
parser/parser.go Stores canonical partition clause on parsed tables; rejects SUBPARTITION.
model/table.go Adds Table.Partition and appends it when rendering CREATE TABLE SQL.
diff/tables.go Errors early when desired vs catalog partition clauses differ.
catalog/tables.go Loads partition clauses for partitioned tables via info_schema + SHOW CREATE TABLE.
TODO.md Re-scopes partition work to “diff generation” only (future).
CAVEATS.md Documents v1 partition behavior and explicit non-support boundaries.
AGENTS.md Updates supported/not-yet-implemented lists to include partition round-trip and drift-detect.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread parser/partition.go Outdated
Comment thread model/table.go Outdated
Comment thread parser/partition.go Outdated
Comment thread catalog/tables.go Outdated
Four nits — one is a real bug, three are doc / shadowing fixes:

- ExtractPartitionFromShowCreate sliced from the *first* `/*!`
  to the *last* `*/`. SHOW CREATE TABLE on MySQL 8.0+ can emit
  earlier versioned-comment blocks (e.g. `/*!80016
  ENCRYPTION='N' */`) before the partition one, in which case
  the body would start with ENCRYPTION='N' and the function
  would return ("", nil) — silently dropping the partition
  metadata. Switch to LastIndex so we always grab the trailing
  block (the partition clause is always last when present), and
  pin the fix with a regression test that mixes ENCRYPTION + a
  RANGE partition.
- Inline comment in NormalizePartitionOption mentioned a
  fictional CreateSQL method. Tables render via
  `(*model.Table).SQL`; updated.
- model.Table.Partition's field comment claimed "both sides
  normalise via sqlparser.String(*sqlparser.PartitionOption)",
  but the implementation goes through
  parser.NormalizePartitionOption (which adds identifier
  lower-casing, leading-newline trim, and per-partition engine
  strip on top of vitess's formatter). Spell out what
  "canonical" actually means.
- Loop-local clause copy in catalog/tables.go::loadPartitions
  was named `c`, shadowing the `*Catalog` receiver. Rename to
  `clauseCopy`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread parser/partition.go Outdated
Doc-only nit. ExtractPartitionFromShowCreate's comment said it
extracts "including any SUBPARTITION BY …", but the function
deliberately errors when a SUBPARTITION clause is present
(SUBPARTITION is out of scope for v1 — CAVEATS.md "Partitioning
is round-trip only"). Fix the comment so future readers don't
expect SUBPARTITION pass-through.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread parser/partition_test.go
Add a TestParseSQLPartitionRoundTrip case for `LIST COLUMNS(...)`
with `VALUES IN ('A', 'B')`. Pins two things at once:

- LIST COLUMNS(...) is on the supported list but wasn't actually
  exercised by the round-trip test.
- The normaliser only lower-cases function-name and column-
  reference identifiers — string literals (`'A'`, `'B'`) must
  round-trip case-preserved, because flipping `'A'` to `'a'`
  would change the value MySQL stores. Without an explicit test
  a future change to the rewriter could quietly break that
  guarantee.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@winebarrel winebarrel merged commit a19fe7b into main May 3, 2026
7 of 9 checks passed
@winebarrel winebarrel deleted the partitioning-roundtrip branch May 3, 2026 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants