parser: support column-level INVISIBLE (MySQL 8.0+)#84
Conversation
Closes the Medium-severity TODO item from the PR #81 silent-drop audit. Pre-fix, vitess parsed `col INT INVISIBLE` into `cd.Type.Options.Invisible` (`*bool`) but `parseColumnDef` ignored the field, so a `hidden_col INT INVISIBLE` desired-side column silently lost the attribute on parse. The catalog reader (which surfaces it via `information_schema.COLUMNS.EXTRA`) saw a permanent diff against the parser side's empty state — drift never closed. Wire the attribute through the four layers that need to know: - model/column.go: new `Invisible bool` field on Column. - parser/parser.go: parseColumnDef reads `opts.Invisible` (only the explicit-true case promotes; nil / false stays the visible default and emits nothing). - model/table.go: columnDefSQL emits ` INVISIBLE` between AUTO_INCREMENT and COMMENT to match SHOW CREATE TABLE order. - catalog/tables.go: loadColumns scans EXTRA for the literal "INVISIBLE" token (Contains, so it composes with other extras like AUTO_INCREMENT). - diff/tables.go: columnEqual gains the Invisible mismatch check so a VISIBLE↔INVISIBLE toggle fires MODIFY COLUMN. Tests: - TestParseColumnInvisible (parser) pins the parsed-model shape. - TestColumnDefSQLInvisible (model) pins the emitted token and the visible-default no-emit path. - TestColumnEqual/invisible_differs (diff) pins the mismatch check. - TestColumnInvisibleRoundTrip (catalog) pins the EXTRA → Invisible read for per-package coverage; the YAML round-trip exercises the same path transitively but doesn't register against catalog package coverage. - testdata/apply/invisible_column.yml integration round-trip: visible → INVISIBLE column add, verify_no_drift: true. Toggling back (INVISIBLE → visible) is exercised manually and confirmed to emit `MODIFY COLUMN x int` (no INVISIBLE keyword). Coverage on touched functions: columnDefSQL / columnEqual 100%, loadColumns 91.9% → 94.6%, parseColumnDef 96.8% (all new INVISIBLE branches covered). TODO.md: drop the entry now that the fix is in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #84 +/- ##
==========================================
+ Coverage 86.62% 86.66% +0.03%
==========================================
Files 30 30
Lines 3268 3277 +9
==========================================
+ Hits 2831 2840 +9
Misses 269 269
Partials 168 168 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR adds end-to-end support for MySQL 8.0 column-level INVISIBLE columns so parser-side schemas, catalog reads, emitted DDL, and diffs all agree instead of drifting forever.
Changes:
- Add
Column.Invisibleand wire parser/catalog reads into that model field. - Emit
INVISIBLEin column SQL and compare visibility in column diffs. - Add unit/integration coverage and remove the resolved TODO entry.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| TODO.md | Removes the resolved silent-drift TODO for column-level INVISIBLE. |
| testdata/apply/invisible_column.yml | Adds an end-to-end apply fixture for adding an invisible column. |
| parser/parser.go | Reads Vitess column visibility metadata into the model. |
| parser/parser_test.go | Adds parser coverage for invisible column definitions. |
| model/table.go | Emits INVISIBLE in generated column definitions. |
| model/table_test.go | Adds unit coverage for invisible-column SQL emission. |
| model/column.go | Extends the column model with visibility state. |
| diff/tables.go | Treats column visibility changes as schema diffs. |
| diff/tables_test.go | Adds equality-test coverage for invisible-column mismatches. |
| catalog/tables.go | Loads column visibility from information_schema.COLUMNS.EXTRA. |
| catalog/tables_test.go | Adds catalog round-trip coverage for invisible columns. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Three real issues from the review: - catalog/tables.go: ON UPDATE EXTRA parsing leaked the trailing INVISIBLE token into Column.OnUpdate (`DEFAULT_GENERATED on update CURRENT_TIMESTAMP INVISIBLE` → OnUpdate "CURRENT_TIMESTAMP INVISIBLE"), so dump emitted the keyword twice (`ON UPDATE CURRENT_TIMESTAMP INVISIBLE INVISIBLE`) and parser-side desired never compared equal. Fix: detect INVISIBLE first, strip the token from extraUp before parsing ON UPDATE / AUTO_INCREMENT / generated. - catalog/tables_test.go: TestColumnInvisibleComposesWithOnUpdate pins the ON UPDATE + INVISIBLE composition (would have failed the regression above pre-fix). - testdata/apply/invisible_to_visible.yml: end-to-end pin for the reverse toggle (INVISIBLE → VISIBLE), which only the YAML apply harness can exercise. Asserts the diff emits a bare `MODIFY COLUMN x int` (no INVISIBLE keyword, no explicit VISIBLE) and verify_no_drift confirms convergence. - model/table_test.go: TestColumnDefSQLInvisible/`INVISIBLE sits between AUTO_INCREMENT and COMMENT` adds an index-comparison subtest so a future reorder of columnDefSQL fails CI rather than silently shipping wrong-order DDL — the previous Contains- only check was order-blind. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
PR #84's review caught a real bug (catalog `loadColumns` leaked the trailing `INVISIBLE` token into `Column.OnUpdate` when both attributes sat on the same EXTRA row) that the original test suite missed because every existing test exercised attributes in isolation. The bug class — composed-state parsers where multiple branches read/write the same shared string — has more potential sites in the codebase. Audit those now so the next contributor working in the area can pick them up. Adds a "Medium — test coverage audit" section between the existing "Medium — silent diffs / fidelity gaps" (data-model fidelity) and "Low — tests / docs / release" (process), with six prioritised items: - Composed-state EXTRA / option parsing matrix tests for `catalog/loadColumns` and `loadIndexes`. - `parseCreateTable` column-loop interactions (inline PK / UNIQUE / REFERENCES × column attributes). - Index-comparison emitter ordering pins for `columnDefSQL`, `indexInlineSQL`, `constraintInlineSQL` (PR #84 pinned only INVISIBLE's slot). - Diff cross-cutting / order-sensitivity (column rename + drop with same name, index rename + DROP INDEX same plan, etc.). - Directive-composition pins for inline directives in one CREATE TABLE. - One "kitchen-sink" round-trip YAML fixture exercising every supported attribute together. Each item lists the surface, the missing combination, and a realistic-harm sketch so the work scope is clear without re-deriving context. Combined estimate: 2.5–3 days; recommended order is catalog matrix → kitchen-sink fixture → emitter ordering → diff → directive composition. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs: log the PR #84 postmortem test-coverage audit to TODO
- model/table.go: ColumnDefSQL doc comment listed the emitted attribute order ending at AUTO_INCREMENT / COMMENT and didn't mention INVISIBLE — stale after this PR's emitter change. Update the attribute list and call out that the order is pinned by index-comparison assertions in model/table_test.go. - parser/parser_test.go: TestParseColumnInvisible now exercises all three vitess Invisible *bool states explicitly: - INVISIBLE keyword → *bool=true → Column.Invisible=true - explicit VISIBLE keyword → *bool=false → Column.Invisible=false - no keyword → *bool=nil → Column.Invisible=false Previously only the nil and true branches were covered; the false branch (the `*opts.Invisible` deref's other arm) ran unverified. Splitting into named sub-tests pins each state individually so a future regression in the nil-vs-false handling fails a specific case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- README.md / AGENTS.md: feature lists called out INVISIBLE only for secondary indexes. With column-level INVISIBLE wired in this PR, both surfaces should mention it. README's "Tables, columns …" bullet now includes "(incl. INVISIBLE on MySQL 8.0+)"; AGENTS.md's "In scope (v1)" CREATE TABLE entry adds "column-level INVISIBLE on MySQL 8.0+" to its attribute list. - parser/parser_test.go: TestParseColumnInvisible's doc comment said it pins "parser → model → emitter → catalog → diff" consistency, but the test only inspects the parsed model. Reword to describe the actual coverage boundary (parser-side vitess *bool decoding) and link to the dedicated tests for the other layers (TestColumnDefSQLInvisible, TestColumnInvisibleRoundTrip, TestColumnEqual/invisible_differs) so a reader navigating the feature can find each layer's pin without confusion. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Fifth and final item from the PR #84 postmortem audit. Inline `-- myschema:renamed-from` directives across all four supported kinds (column / index / FK / CHECK) in one CREATE TABLE body must route to their own buckets without competing. - TestExtractInlineRenamesAllKindsInOneTable (parser) puts a rename of each kind in one body and asserts each routes to its matching map (Columns / Indexes / ForeignKeys / Constraints). - TestExtractInlineRenamesColumnAndIndexShareName (parser): a column named `email` AND a UNIQUE KEY auto-named `email` (the default when the user writes `UNIQUE KEY email (email)`) can both carry rename directives without confusing the line classifier. - testdata/apply/inline_rename_all_kinds.yml exercises the full plan → apply → re-plan-empty round-trip. While composing the YAML fixture, hit a real cross-cutting bug: when a CHECK constraint references the renamed column, MySQL rejects the RENAME COLUMN step (`Error 3959 Check constraint X uses column Y, hence column cannot be dropped or renamed`) because the diff layer's pass-order emits column renames *before* CHECK drops. The fixture works around it by referencing a different column in the CHECK; the underlying bug is logged as a new TODO entry under "Medium — silent diffs / fidelity gaps" so a follow-up can route CHECK drops into a dedicated bucket the way FK drops already are. This closes the test-coverage audit started after PR #84. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five of the six items from the PR #84 postmortem audit landed: - composed-state catalog tests → PR #86 - kitchen-sink round-trip → PR #87 - emitter ordering pins → PR #88 - diff cross-cutting → PR #89 - directive composition → PR #90 Drop the closed items. Keep `parseCreateTable` column-loop interactions under "Medium — test coverage gaps" — that one was scoped out of the kitchen-sink because the open shapes (inline `UNIQUE × DEFAULT`, inline `REFERENCES × NOT NULL × ON UPDATE`, inline `PK × column-level CHECK`) need direct parser unit tests, not a round-trip fixture. The kitchen-sink uses `email VARCHAR NOT NULL UNIQUE` (no DEFAULT), an explicit named CONSTRAINT FK (not inline REFERENCES), and a table-level CHECK (not the column-level shape that exercises the vitess promotion path), so the gap is real. The "Medium — silent diffs / fidelity gaps" section keeps the real open items, including the three new ones surfaced by the audit itself (REFERENCES auto-name, vitess-keyword identifier drift, CHECK referencing renamed column). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Closes the Medium-severity TODO item from the PR #81 silent-drop audit. vitess parsed
col INT INVISIBLEintocd.Type.Options.Invisible(*bool) butparseColumnDefignored the field, so ahidden_col INT INVISIBLEdesired-side column silently lost the attribute on parse. The catalog reader (which surfaces it viainformation_schema.COLUMNS.EXTRA) saw a permanent diff against the parser side's empty state — drift never closed.Wire the attribute through the four layers that need to know:
model/column.go— newInvisible boolfield onColumn.parser/parser.go—parseColumnDefreadsopts.Invisible(only the explicit-true case promotes;nil/falsestays the visible default and emits nothing).model/table.go—columnDefSQLemitsINVISIBLEbetweenAUTO_INCREMENTandCOMMENTto matchSHOW CREATE TABLEorder.catalog/tables.go—loadColumnsscansEXTRAfor the literalINVISIBLEtoken (Contains, so it composes with other extras likeAUTO_INCREMENT).diff/tables.go—columnEqualgains theInvisiblemismatch check so a VISIBLE↔INVISIBLE toggle firesMODIFY COLUMN.TODO.md: drop the entry now that the fix is in.Test plan
Unit:
TestParseColumnInvisible(parser) pins the parsed-model shape.TestColumnDefSQLInvisible(model) pins the emitted token and the visible-default no-emit path.TestColumnEqual/invisible_differs(diff) pins the mismatch check.TestColumnInvisibleRoundTrip(catalog) pins theEXTRA → Invisibleread for per-package coverage; the YAML round-trip exercises the same path transitively but doesn't register against catalog package coverage.Integration:
testdata/apply/invisible_column.ymlround-trip: VISIBLE column → ADD COLUMN withINVISIBLE,verify_no_drift: true.MODIFY COLUMN x intagainst a real MySQL 8.0.Coverage on touched functions:
columnDefSQL/columnEqual: 100%loadColumns: 91.9% → 94.6%parseColumnDef: 96.8% (all new INVISIBLE branches covered).make lintclean,make testall six packages green.🤖 Generated with Claude Code