Skip to content

parser: support column-level INVISIBLE (MySQL 8.0+)#84

Merged
winebarrel merged 4 commits into
mainfrom
parser-support-invisible-column
May 5, 2026
Merged

parser: support column-level INVISIBLE (MySQL 8.0+)#84
winebarrel merged 4 commits into
mainfrom
parser-support-invisible-column

Conversation

@winebarrel

Copy link
Copy Markdown
Owner

Summary

Closes the Medium-severity TODO item from the PR #81 silent-drop audit. vitess parsed col INT INVISIBLE into cd.Type.Options.Invisible (*bool) but parseColumnDef ignored the field, so a hidden_col INT INVISIBLE desired-side column silently lost the attribute on parse. The catalog reader (which surfaces it via information_schema.COLUMNS.EXTRA) saw a permanent diff against the parser side's empty state — drift never closed.

Wire the attribute through the four layers that need to know:

  • model/column.go — new Invisible bool field on Column.
  • parser/parser.goparseColumnDef reads opts.Invisible (only the explicit-true case promotes; nil / false stays the visible default and emits nothing).
  • model/table.gocolumnDefSQL emits INVISIBLE between AUTO_INCREMENT and COMMENT to match SHOW CREATE TABLE order.
  • catalog/tables.goloadColumns scans EXTRA for the literal INVISIBLE token (Contains, so it composes with other extras like AUTO_INCREMENT).
  • diff/tables.gocolumnEqual gains the Invisible mismatch check so a VISIBLE↔INVISIBLE toggle fires MODIFY COLUMN.

TODO.md: drop the entry now that the fix is in.

Test plan

Unit:

  • TestParseColumnInvisible (parser) pins the parsed-model shape.
  • TestColumnDefSQLInvisible (model) pins the emitted token and the visible-default no-emit path.
  • TestColumnEqual/invisible_differs (diff) pins the mismatch check.
  • TestColumnInvisibleRoundTrip (catalog) pins the EXTRA → Invisible read for per-package coverage; the YAML round-trip exercises the same path transitively but doesn't register against catalog package coverage.

Integration:

  • testdata/apply/invisible_column.yml round-trip: VISIBLE column → ADD COLUMN with INVISIBLE, verify_no_drift: true.
  • Manually verified the inverse toggle (INVISIBLE → VISIBLE) emits MODIFY COLUMN x int against a real MySQL 8.0.

Coverage on touched functions:

  • columnDefSQL / columnEqual: 100%
  • loadColumns: 91.9% → 94.6%
  • parseColumnDef: 96.8% (all new INVISIBLE branches covered).

make lint clean, make test all six packages green.

🤖 Generated with Claude Code

Closes the Medium-severity TODO item from the PR #81 silent-drop
audit. Pre-fix, vitess parsed `col INT INVISIBLE` into
`cd.Type.Options.Invisible` (`*bool`) but `parseColumnDef` ignored
the field, so a `hidden_col INT INVISIBLE` desired-side column
silently lost the attribute on parse. The catalog reader (which
surfaces it via `information_schema.COLUMNS.EXTRA`) saw a permanent
diff against the parser side's empty state — drift never closed.

Wire the attribute through the four layers that need to know:

- model/column.go: new `Invisible bool` field on Column.
- parser/parser.go: parseColumnDef reads `opts.Invisible` (only
  the explicit-true case promotes; nil / false stays the visible
  default and emits nothing).
- model/table.go: columnDefSQL emits ` INVISIBLE` between
  AUTO_INCREMENT and COMMENT to match SHOW CREATE TABLE order.
- catalog/tables.go: loadColumns scans EXTRA for the literal
  "INVISIBLE" token (Contains, so it composes with other extras
  like AUTO_INCREMENT).
- diff/tables.go: columnEqual gains the Invisible mismatch check
  so a VISIBLE↔INVISIBLE toggle fires MODIFY COLUMN.

Tests:

- TestParseColumnInvisible (parser) pins the parsed-model shape.
- TestColumnDefSQLInvisible (model) pins the emitted token and the
  visible-default no-emit path.
- TestColumnEqual/invisible_differs (diff) pins the mismatch check.
- TestColumnInvisibleRoundTrip (catalog) pins the EXTRA → Invisible
  read for per-package coverage; the YAML round-trip exercises
  the same path transitively but doesn't register against catalog
  package coverage.
- testdata/apply/invisible_column.yml integration round-trip:
  visible → INVISIBLE column add, verify_no_drift: true. Toggling
  back (INVISIBLE → visible) is exercised manually and confirmed
  to emit `MODIFY COLUMN x int` (no INVISIBLE keyword).

Coverage on touched functions: columnDefSQL / columnEqual 100%,
loadColumns 91.9% → 94.6%, parseColumnDef 96.8% (all new
INVISIBLE branches covered).

TODO.md: drop the entry now that the fix is in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 5, 2026 13:43
@codecov

codecov Bot commented May 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.66%. Comparing base (8264344) to head (33b01b5).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #84      +/-   ##
==========================================
+ Coverage   86.62%   86.66%   +0.03%     
==========================================
  Files          30       30              
  Lines        3268     3277       +9     
==========================================
+ Hits         2831     2840       +9     
  Misses        269      269              
  Partials      168      168              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds end-to-end support for MySQL 8.0 column-level INVISIBLE columns so parser-side schemas, catalog reads, emitted DDL, and diffs all agree instead of drifting forever.

Changes:

  • Add Column.Invisible and wire parser/catalog reads into that model field.
  • Emit INVISIBLE in column SQL and compare visibility in column diffs.
  • Add unit/integration coverage and remove the resolved TODO entry.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
TODO.md Removes the resolved silent-drift TODO for column-level INVISIBLE.
testdata/apply/invisible_column.yml Adds an end-to-end apply fixture for adding an invisible column.
parser/parser.go Reads Vitess column visibility metadata into the model.
parser/parser_test.go Adds parser coverage for invisible column definitions.
model/table.go Emits INVISIBLE in generated column definitions.
model/table_test.go Adds unit coverage for invisible-column SQL emission.
model/column.go Extends the column model with visibility state.
diff/tables.go Treats column visibility changes as schema diffs.
diff/tables_test.go Adds equality-test coverage for invisible-column mismatches.
catalog/tables.go Loads column visibility from information_schema.COLUMNS.EXTRA.
catalog/tables_test.go Adds catalog round-trip coverage for invisible columns.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread catalog/tables.go
Comment thread testdata/apply/invisible_column.yml
Comment thread model/table_test.go
Three real issues from the review:

- catalog/tables.go: ON UPDATE EXTRA parsing leaked the trailing
  INVISIBLE token into Column.OnUpdate (`DEFAULT_GENERATED on
  update CURRENT_TIMESTAMP INVISIBLE` → OnUpdate
  "CURRENT_TIMESTAMP INVISIBLE"), so dump emitted the keyword
  twice (`ON UPDATE CURRENT_TIMESTAMP INVISIBLE INVISIBLE`) and
  parser-side desired never compared equal. Fix: detect INVISIBLE
  first, strip the token from extraUp before parsing ON UPDATE /
  AUTO_INCREMENT / generated.
- catalog/tables_test.go: TestColumnInvisibleComposesWithOnUpdate
  pins the ON UPDATE + INVISIBLE composition (would have failed
  the regression above pre-fix).
- testdata/apply/invisible_to_visible.yml: end-to-end pin for the
  reverse toggle (INVISIBLE → VISIBLE), which only the YAML
  apply harness can exercise. Asserts the diff emits a bare
  `MODIFY COLUMN x int` (no INVISIBLE keyword, no explicit
  VISIBLE) and verify_no_drift confirms convergence.
- model/table_test.go: TestColumnDefSQLInvisible/`INVISIBLE sits
  between AUTO_INCREMENT and COMMENT` adds an index-comparison
  subtest so a future reorder of columnDefSQL fails CI rather
  than silently shipping wrong-order DDL — the previous Contains-
  only check was order-blind.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread model/table.go
Comment thread parser/parser.go
winebarrel added a commit that referenced this pull request May 5, 2026
PR #84's review caught a real bug (catalog `loadColumns` leaked the
trailing `INVISIBLE` token into `Column.OnUpdate` when both
attributes sat on the same EXTRA row) that the original test suite
missed because every existing test exercised attributes in
isolation. The bug class — composed-state parsers where multiple
branches read/write the same shared string — has more potential
sites in the codebase. Audit those now so the next contributor
working in the area can pick them up.

Adds a "Medium — test coverage audit" section between the existing
"Medium — silent diffs / fidelity gaps" (data-model fidelity) and
"Low — tests / docs / release" (process), with six prioritised
items:

- Composed-state EXTRA / option parsing matrix tests for
  `catalog/loadColumns` and `loadIndexes`.
- `parseCreateTable` column-loop interactions (inline PK / UNIQUE /
  REFERENCES × column attributes).
- Index-comparison emitter ordering pins for `columnDefSQL`,
  `indexInlineSQL`, `constraintInlineSQL` (PR #84 pinned only
  INVISIBLE's slot).
- Diff cross-cutting / order-sensitivity (column rename + drop with
  same name, index rename + DROP INDEX same plan, etc.).
- Directive-composition pins for inline directives in one CREATE
  TABLE.
- One "kitchen-sink" round-trip YAML fixture exercising every
  supported attribute together.

Each item lists the surface, the missing combination, and a
realistic-harm sketch so the work scope is clear without re-deriving
context. Combined estimate: 2.5–3 days; recommended order is
catalog matrix → kitchen-sink fixture → emitter ordering → diff →
directive composition.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
winebarrel added a commit that referenced this pull request May 5, 2026
docs: log the PR #84 postmortem test-coverage audit to TODO
- model/table.go: ColumnDefSQL doc comment listed the emitted
  attribute order ending at AUTO_INCREMENT / COMMENT and didn't
  mention INVISIBLE — stale after this PR's emitter change.
  Update the attribute list and call out that the order is pinned
  by index-comparison assertions in model/table_test.go.
- parser/parser_test.go: TestParseColumnInvisible now exercises
  all three vitess Invisible *bool states explicitly:
  - INVISIBLE keyword → *bool=true → Column.Invisible=true
  - explicit VISIBLE keyword → *bool=false → Column.Invisible=false
  - no keyword → *bool=nil → Column.Invisible=false
  Previously only the nil and true branches were covered; the
  false branch (the `*opts.Invisible` deref's other arm) ran
  unverified. Splitting into named sub-tests pins each state
  individually so a future regression in the nil-vs-false
  handling fails a specific case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread TODO.md
Comment thread parser/parser_test.go Outdated
- README.md / AGENTS.md: feature lists called out INVISIBLE only
  for secondary indexes. With column-level INVISIBLE wired in this
  PR, both surfaces should mention it. README's "Tables, columns
  …" bullet now includes "(incl. INVISIBLE on MySQL 8.0+)";
  AGENTS.md's "In scope (v1)" CREATE TABLE entry adds "column-level
  INVISIBLE on MySQL 8.0+" to its attribute list.
- parser/parser_test.go: TestParseColumnInvisible's doc comment
  said it pins "parser → model → emitter → catalog → diff"
  consistency, but the test only inspects the parsed model. Reword
  to describe the actual coverage boundary (parser-side vitess
  *bool decoding) and link to the dedicated tests for the other
  layers (TestColumnDefSQLInvisible, TestColumnInvisibleRoundTrip,
  TestColumnEqual/invisible_differs) so a reader navigating the
  feature can find each layer's pin without confusion.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@winebarrel winebarrel merged commit e22d785 into main May 5, 2026
9 checks passed
@winebarrel winebarrel deleted the parser-support-invisible-column branch May 5, 2026 15:11
winebarrel added a commit that referenced this pull request May 6, 2026
Fifth and final item from the PR #84 postmortem audit. Inline
`-- myschema:renamed-from` directives across all four supported
kinds (column / index / FK / CHECK) in one CREATE TABLE body must
route to their own buckets without competing.

- TestExtractInlineRenamesAllKindsInOneTable (parser) puts a
  rename of each kind in one body and asserts each routes to its
  matching map (Columns / Indexes / ForeignKeys / Constraints).
- TestExtractInlineRenamesColumnAndIndexShareName (parser): a
  column named `email` AND a UNIQUE KEY auto-named `email` (the
  default when the user writes `UNIQUE KEY email (email)`) can
  both carry rename directives without confusing the line
  classifier.
- testdata/apply/inline_rename_all_kinds.yml exercises the
  full plan → apply → re-plan-empty round-trip.

While composing the YAML fixture, hit a real cross-cutting bug:
when a CHECK constraint references the renamed column, MySQL
rejects the RENAME COLUMN step (`Error 3959 Check constraint X
uses column Y, hence column cannot be dropped or renamed`)
because the diff layer's pass-order emits column renames *before*
CHECK drops. The fixture works around it by referencing a
different column in the CHECK; the underlying bug is logged as a
new TODO entry under "Medium — silent diffs / fidelity gaps" so
a follow-up can route CHECK drops into a dedicated bucket the
way FK drops already are.

This closes the test-coverage audit started after PR #84.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
winebarrel added a commit that referenced this pull request May 6, 2026
Five of the six items from the PR #84 postmortem audit landed:
- composed-state catalog tests   → PR #86
- kitchen-sink round-trip        → PR #87
- emitter ordering pins          → PR #88
- diff cross-cutting             → PR #89
- directive composition          → PR #90

Drop the closed items. Keep `parseCreateTable` column-loop
interactions under "Medium — test coverage gaps" — that one was
scoped out of the kitchen-sink because the open shapes (inline
`UNIQUE × DEFAULT`, inline `REFERENCES × NOT NULL × ON UPDATE`,
inline `PK × column-level CHECK`) need direct parser unit tests,
not a round-trip fixture. The kitchen-sink uses `email VARCHAR
NOT NULL UNIQUE` (no DEFAULT), an explicit named CONSTRAINT FK
(not inline REFERENCES), and a table-level CHECK (not the
column-level shape that exercises the vitess promotion path), so
the gap is real.

The "Medium — silent diffs / fidelity gaps" section keeps the
real open items, including the three new ones surfaced by the
audit itself (REFERENCES auto-name, vitess-keyword identifier
drift, CHECK referencing renamed column).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants