Skip to content

test/scenario: cover the major happy-path workflows#31

Merged
winebarrel merged 7 commits into
mainfrom
expand-scenario-tests
May 2, 2026
Merged

test/scenario: cover the major happy-path workflows#31
winebarrel merged 7 commits into
mainfrom
expand-scenario-tests

Conversation

@winebarrel

Copy link
Copy Markdown
Owner

Summary

The scenario suite shipped with one smoke test (CREATE TABLE + idempotent re-apply). This PR adds four more scenarios covering the features users exercise day-to-day, plus extends helper.sh with two generic assertion primitives.

Scenarios

File What it covers
evolution.test.sh CREATE → ADD COLUMN → ADD INDEX → ADD FK → MODIFY COLUMN → DROP COLUMN (with column-attached index auto-suppression). Six stages, no drift after each.
rename.test.sh -- myschema:renamed-from end-to-end: table rename → column rename → index rename (KEY + UNIQUE).
dump_roundtrip.test.sh Load schema, myschema dump to a temp file, drop DB, re-apply the dump, plan against the original source SQL → no drift. Pins dump-renderer fidelity.
views.test.sh View lifecycle: ADD → MODIFY body (CREATE OR REPLACE) → DROP under --allow-drop=all.
filter.test.sh --include / --exclude with three tables in DB but only one in desired. Glob patterns; verifies filtered tables don't show as DROPs while unfiltered ones do.

Helper additions

helper.sh gains two generic primitives so scenarios beyond run_step* can assert on raw command output without each rolling its own grep:

  • assert_contains <step> <cmd...> -- <substring>
  • assert_not_contains <step> <cmd...> -- <substring>

Plus a myschema_dump wrapper to match the existing myschema_plan / myschema_apply pair.

Discovered along the way

While writing evolution, ADD COLUMN ... NOT NULL DEFAULT '' round-tripped with drift (catalog reads back '' differently than parser produces it). I worked around it by dropping the explicit DEFAULT in the scenario fixture; the underlying catalog default-normalisation gap is a separate issue worth filing later.

Test plan

  • make test-scenario (mysql 8.0) — all 22 steps PASS
  • make test-scenario MYSQL_PORT=3307 (mysql 9.4) — all 22 steps PASS
  • make lint

🤖 Generated with Claude Code

The scenario suite shipped with one smoke test (CREATE TABLE +
idempotent re-apply). Add four more scenarios so the CLI covers the
features users actually exercise day-to-day, plus extend the
existing helpers with a couple of generic assertion primitives.

Scenarios

  - evolution.test.sh — walks one schema through six stages:
    CREATE → ADD COLUMN → ADD INDEX → ADD CONSTRAINT FK →
    MODIFY COLUMN (widen) → DROP COLUMN (with column-attached index
    auto-suppression). Each step asserts no drift after apply, so a
    regression in any of those ALTER paths surfaces here.

  - rename.test.sh — exercises `-- myschema:renamed-from` end-to-end
    via the CLI: table rename → column rename → index rename
    (KEY + UNIQUE in one step). Each step pins the right
    ALTER … RENAME form lands in plan output and post-apply plan
    is silent.

  - dump_roundtrip.test.sh — load a non-trivial schema (tables,
    FK ON DELETE CASCADE, view), `myschema dump` it to a temp file,
    drop the DB, re-apply the dump, then plan against the *original*
    source SQL — expects no drift. Pins that the dump renderer
    produces SQL the parser can read back.

  - views.test.sh — view lifecycle: ADD → MODIFY body
    (CREATE OR REPLACE) → DROP under --allow-drop=all.

  - filter.test.sh — `--include` / `--exclude` with three tables in
    the DB but only one in desired. Asserts filtered tables don't
    show as drops; unfiltered ones do; glob patterns match.

Helpers

  helper.sh gains two generic assertion primitives so scenarios
  beyond the existing run_step* shape can assert on raw command
  output without each rolling its own grep/diff:

  - assert_contains <step> <cmd...> -- <substring>
  - assert_not_contains <step> <cmd...> -- <substring>

  Plus a `myschema_dump` thin wrapper to match the existing
  myschema_plan / myschema_apply pair.

Verified locally on both compose services (8.0 / 9.4): all 22
scenario steps pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 2, 2026 15:37
@codecov

codecov Bot commented May 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.37%. Comparing base (d13d39f) to head (815194c).
⚠️ Report is 18 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #31      +/-   ##
==========================================
+ Coverage   73.33%   73.37%   +0.04%     
==========================================
  Files          27       27              
  Lines        2141     2141              
==========================================
+ Hits         1570     1571       +1     
+ Misses        411      410       -1     
  Partials      160      160              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Expands the bash scenario test suite to cover common end-to-end workflows (schema evolution, renames, views, include/exclude filtering, and dump→apply round-trips) and adds helper primitives to assert against raw command output.

Changes:

  • Added new scenario scripts for evolution, rename directives, dump fidelity, view lifecycle, and filtering behavior.
  • Added scenario fixtures (SQL) for each new scenario.
  • Extended test/scenario/helper.sh with myschema_dump, assert_contains, and assert_not_contains.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
test/scenario/helper.sh Adds dump wrapper + generic output assertion helpers used by new scenarios.
test/scenario/dump_roundtrip.test.sh New scenario validating dump renderer round-trips without drift.
test/scenario/evolution.test.sh New multi-step schema evolution scenario (ADD/INDEX/FK/MODIFY/DROP).
test/scenario/filter.test.sh New scenario validating --include/--exclude filtering affects diffs as expected.
test/scenario/rename.test.sh New scenario validating -- myschema:renamed-from for table/column/index renames.
test/scenario/views.test.sh New scenario validating view create/replace/drop behavior under allow-drop.
test/scenario/testdata/views/*.sql Fixtures for view lifecycle stages.
test/scenario/testdata/rename/*.sql Fixtures for rename directive stages.
test/scenario/testdata/filter/*.sql Fixtures for include/exclude filtering scenario.
test/scenario/testdata/evolution/*.sql Fixtures for evolution stages (including FK + index suppression on dropped column).
test/scenario/testdata/dump_roundtrip/source.sql Source schema fixture used for dump fidelity validation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/scenario/helper.sh Outdated
Comment thread test/scenario/helper.sh Outdated
Comment thread test/scenario/dump_roundtrip.test.sh
Comment thread test/scenario/dump_roundtrip.test.sh
Comment thread test/scenario/evolution.test.sh Outdated
Five inline comments, all valid.

helper.sh
  - assert_contains / assert_not_contains now capture stderr too
    (`2>&1`) so a command that prints the relevant text to stderr
    doesn't cause a false negative. (Copilot #1, #2.)
  - Both helpers explicitly validate the invocation: missing `--`
    separator, no command before `--`, or no substring after `--`
    each fail with a clear message instead of letting bash run a
    malformed command and produce an opaque "command not found"
    or hang on stdin. (Copilot #1, #2.)
  - Sanity-checked all four validation paths (`missing sep`,
    `missing cmd`, `missing substring`, `stderr capture`) by
    sourcing helper.sh in a one-shot bash session.

dump_roundtrip.test.sh
  - The dump-failure and apply-failure branches now `summary; exit 1`
    after `fail`, instead of falling through to the next step. The
    cascading errors that followed obscured the original failure;
    failing fast keeps the scenario log focused on the root cause.
    (Copilot #3, #4.)

evolution.test.sh
  - Header comment said "ADD COLUMN with DEFAULT" but the fixture
    drops the DEFAULT (workaround for the catalog default-
    normalisation drift documented in TODO.md). Comment now matches
    reality and points at the TODO entry. (Copilot #5.)

`make test-scenario` (mysql 8.0) — all 22 steps still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/scenario/helper.sh Outdated
Comment thread test/scenario/evolution.test.sh Outdated
Comment thread test/scenario/filter.test.sh Outdated
Three inline comments, all valid.

helper.sh
  - myschema_dump no longer merges stderr into stdout. Callers
    typically redirect stdout to a .sql file (dump_roundtrip
    re-applies the dump), and any stderr text would otherwise be
    embedded in the dump and break re-apply. Plan / apply wrappers
    keep `2>&1` because their callers run them through `out=$(...)`
    and want both streams. (Copilot #1.)

evolution.test.sh
  - Step "06 DROP COLUMN + auto-index" only asserted the plan
    contains DROP COLUMN; if the column-attached index suppression
    regressed and DROP INDEX users_display_name_idx leaked through,
    apply would still succeed (because MySQL has already auto-removed
    the index, but `Error 1091` would surface — actually a real
    failure). Add an explicit assert_not_contains BEFORE run_step
    so the suppression behaviour itself is pinned in plan output,
    independent of apply outcome. (Copilot #2.)

filter.test.sh
  - Step 01's comment claimed `--include=users` hides BOTH sessions
    and logs, but the assertion only checked sessions. Split into
    two explicit assert_not_contains so both filter targets are
    covered. (Copilot #3.)

`make test-scenario` (mysql 8.0) — all 24 steps pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/scenario/helper.sh
Comment thread test/scenario/helper.sh Outdated
Copilot review #3 follow-up nit: the helpers only consumed `$1`
after the `--` separator and silently ignored anything after it. So
an unquoted multi-word substring (`-- DROP TABLE foo`) would only
search for "DROP" — passing/failing on the wrong target without any
warning.

Add a `$# -ne 1` check after parsing the separator so a missing
quote fails loudly with a clear message that points at the fix
("too many arguments after '--' (got N); quote the substring").
Sanity-checked by sourcing helper.sh in a one-shot bash session
and confirming the error fires.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/scenario/helper.sh Outdated
Comment thread test/scenario/helper.sh
Comment thread test/scenario/helper.sh Outdated
Comment thread test/scenario/helper.sh Outdated
Comment thread test/scenario/dump_roundtrip.test.sh Outdated
Five Copilot inline comments on PR #31, all valid:

helper.sh
  - assert_contains / assert_not_contains validate $# >= 1 before
    referencing $1 / shifting. Under `set -u` (which the scenario
    runner enables), a no-arg call would otherwise abort with
    "unbound variable" instead of falling into the helper's own
    fail() path. (Copilot #1, #2.)
  - Replace `echo "$out" | grep ...` with
    `printf '%s\n' "$out" | grep -qF -- ...`. echo can mangle
    output that starts with `-n` / contains escape sequences;
    printf is faithful, and the `--` after `-qF` keeps grep from
    treating a leading `-` in the substring as an option.
    (Copilot #3, #4.)

test/scenario/dump_roundtrip.test.sh
  - `mktemp -d` without a template is GNU-specific; macOS / BSD
    requires one. Pass an explicit template under `$TMPDIR` so the
    scenario runs portably. (Copilot #5.)

`make test-scenario` (mysql 8.0) — all 24 steps still pass.
Sanity-checked the `set -u` path by sourcing helper.sh in a fresh
bash session and calling `assert_contains` with no args; it now
returns via fail() instead of aborting.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/scenario/filter.test.sh
Comment thread test/scenario/helper.sh Outdated
Two Copilot inline comments on PR #31:

helper.sh
  - Extract the shared parsing / validation / command-execution
    logic of assert_contains and assert_not_contains into a single
    `_assert_substring <mode>` private helper. The two public
    wrappers shrink to one-liners that only choose the mode, so
    diagnostics and behaviour stay in lockstep going forward.
    (Copilot #2.)

filter.test.sh
  - Step 01 only asserted DROP TABLE for the excluded targets was
    *absent*. An unrelated ALTER on `users` could still slip
    through. Add an explicit "No changes" assertion for the
    `-I users` case so the test pins clean-diff behaviour, not just
    absence-of-drops. (Copilot #1.)

`make test-scenario` (mysql 8.0) — all 25 steps pass (gain one).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@winebarrel winebarrel requested a review from Copilot May 2, 2026 16:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/scenario/helper.sh
Comment thread test/scenario/testdata/rename/04_rename_index.sql Outdated
Comment thread test/scenario/filter.test.sh Outdated
Three Copilot inline comments on PR #31:

helper.sh
  - `_assert_substring` now rejects an empty needle. `grep -qF -- ""`
    always matches, so `assert_contains ... -- ''` would silently
    pass (and `_not_contains` would always fail) regardless of
    output. Almost always a caller bug — fail loudly with
    "substring is empty". (Copilot #1.)

testdata/rename/04_rename_index.sql
  - Header comment claimed the UNIQUE index was "previously-
    renamed", but step 03 only renames a column; both indexes are
    actually renamed in step 04. Reword to spell out both
    renames explicitly. (Copilot #2.)

filter.test.sh
  - Step labels jumped 02 → 03 mid-block. Renumber so the two
    `--exclude=log*` checks are 02a/02b and the two no-filter
    checks are 03a/03b, matching the comment grouping. (Copilot #3.)

`make test-scenario` (mysql 8.0) — all 25 steps still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@winebarrel winebarrel merged commit 9393ca8 into main May 2, 2026
9 checks passed
@winebarrel winebarrel deleted the expand-scenario-tests branch May 2, 2026 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants