Skip to content

v0.26.7 test: isolation foundation (helpers + lint + quarantine)#613

Merged
garrytan merged 5 commits into
masterfrom
garrytan/test-concurrent-sweep
May 4, 2026
Merged

v0.26.7 test: isolation foundation (helpers + lint + quarantine)#613
garrytan merged 5 commits into
masterfrom
garrytan/test-concurrent-sweep

Conversation

@garrytan

@garrytan garrytan commented May 4, 2026

Copy link
Copy Markdown
Owner

Summary

Foundation slice of the v0.26.4 P0 follow-up for intra-file test parallelism. Re-scoped from a single ~92-file PR after Codex review surfaced 10 findings (D9–D17 in plan). Two follow-ups carry the heavy lifting: v0.26.8 (env-mutation sweep, 49 files) and v0.26.9 (PGLite sweep + codemod + measurement, 42 files).

Helpers

  • test/helpers/with-env.tswithEnv(overrides, fn) save+restore via try/finally. Sync + async, handles delete via undefined, nested calls compose. 7 unit cases.
  • test/helpers/reset-pglite.ts JSDoc extended with the canonical 4-line PGLite block (no logic change).

Lint

  • scripts/check-test-isolation.sh — grep-based, enforces 4 rules on non-serial unit test files: R1 process.env mutations, R2 mock.module(), R3 PGLiteEngine outside beforeAll context, R4 missing afterAll(disconnect) pair.
  • Wired into bun run verify and bun run check:all (NOT bun run test, which is the parallel runner with no pre-check chain — caught by Codex F4).
  • 51 baseline violators in scripts/check-test-isolation.allowlist; list MUST shrink as v0.26.8/v0.26.9 sweeps land.
  • 16 fixture-driven test cases for the lint.

Quarantine renames (mock.module = R2 violation)

  • test/core/cycle.test.tstest/core/cycle.serial.test.ts
  • test/embed.test.tstest/embed.serial.test.ts

Docs

  • CLAUDE.md ## Testing section extended with R1–R4 rules table + canonical PGLite block + withEnv pattern + when-to-quarantine guidance.
  • CONTRIBUTING.md "Running tests" rewritten with v0.26.4/v0.26.7 test tiers; new "Writing tests that survive the parallel loop" section.
  • README.md Contributing line updated.

Test Coverage

v0.26.7 — Test isolation infra (test-only refactor)

NEW PRODUCTION FILES                                        COVERAGE
====================================================================
test/helpers/with-env.ts                                    [PASS]
  ├─ set string value                                       tests 1,2,4,6,7
  ├─ delete (undefined override)                            tests 3,4
  ├─ sync callback                                          test 1
  ├─ async callback await                                   test 2
  ├─ return value passthrough                               tests 1,2
  ├─ throw → finally restore                                test 5
  ├─ restore: prior was string                              tests 1,2,5,6,7
  ├─ restore: prior was undefined                           tests 4,6,7
  ├─ multiple keys atomic restore                           test 7
  ├─ nested compose                                         test 6
  └─ empty overrides {}                                     — (trivial gap)

scripts/check-test-isolation.sh                             [PASS]
  ├─ clean file → exit 0                                    case 1
  ├─ R1 process.env.X = ...                                 case 2
  ├─ R1 process.env['X'] = ...                              case 3
  ├─ R1 delete process.env.X                                case 4
  ├─ R1 Object.assign(process.env, ...)                     case 5
  ├─ R1 Reflect.set(process.env, ...)                       case 6
  ├─ R1 negative: comparison ===                            case 7
  ├─ R2 mock.module(...)                                    case 8
  ├─ R3 top-level PGLiteEngine                              case 9
  ├─ R3 negative: within beforeAll                          case 10
  ├─ R4 missing afterAll/disconnect                         case 11
  ├─ skip *.serial.test.ts                                  case 12
  ├─ skip test/e2e/                                         case 13
  ├─ allowlist skip                                         case 14
  ├─ allowlist partial (non-listed flagged)                 case 15
  └─ allowlist comments + blanks                            case 16

RENAMES (mock.module → R2 quarantine, justified)
====================================================================
test/core/cycle.test.ts → cycle.serial.test.ts              [VERIFIED]
test/embed.test.ts → embed.serial.test.ts                   [VERIFIED]

REGRESSIONS                                                 None
GAPS                                                        1 minor (empty overrides)
====================================================================
RESULT: 23/23 helper+lint tests pass; lint OK on 206 files; coverage ≈ 96%

Tests: 3720 → 3738 (+18 new). Wallclock: 74s on a Mac dev box (already under v0.26.9's ≤60s informational target).

Pre-Landing Review

No issues found. Diff is 168 lines net across 7 files (mostly docs and the lint script). No SQL, no LLM trust boundary, no API contracts, no migrations.

Plan Completion

All 17 plan decisions (D1–D17) closed. v0.26.7 acceptance gate met: helpers + lint + quarantines + Codex F3/F4 fixes shipped. v0.26.8 and v0.26.9 carry the env and PGLite sweeps.

TODOS

TODOS.md v0.26.4 follow-up entry updated:

Documentation

Post-ship doc sync committed (168a8ee9):

  • README.md Contributing line updated to point at bun run test (parallel fast loop) and bun run verify (pre-push gate).
  • CONTRIBUTING.md "Running tests" rewritten for v0.26.4/v0.26.7 test tiers. New "Writing tests that survive the parallel loop" section documents R1–R4, the canonical PGLite block, withEnv pattern, and when to quarantine.
  • llms-full.txt regenerated to pick up the README + CONTRIBUTING edits.

CLAUDE.md ## Testing section was already extended in-branch for v0.26.7 (rules table + canonical block + withEnv pattern). CHANGELOG.md v0.26.7 entry voice is already sharp.

Test plan

  • bun run verify exits 0 on clean tree
  • bun run verify exits 1 with planted violation (16 fixture cases prove this)
  • bun test test/helpers/with-env.test.ts test/scripts/check-test-isolation.test.ts → 23/23 pass
  • bun run test → 3738 pass, 0 fail, 74s
  • Quarantined files (cycle.serial.test.ts, embed.serial.test.ts) pass at --max-concurrency=1
  • bun run check:all → all 9 checks pass (privacy, jsonb, progress, no-legacy-getconnection, check-test-isolation, trailing-newline, wasm, exports-count, admin-build)

🤖 Generated with Claude Code


View in Codesmith
Need help on this PR? Tag @codesmith with what you need.

  • Let Codesmith autofix CI failures and bot reviews

garrytan and others added 5 commits May 4, 2026 11:16
withEnv(overrides, fn) saves prior values, runs the callback, restores
via try/finally — including on throw. Handles delete via undefined
override. Nested calls compose. Cross-test safe; explicitly NOT
intra-file concurrent-safe (process.env is process-global).

7 unit cases covering sync, async, delete-key, delete-when-prior-unset,
restore-on-throw, nested compose, multi-key atomic restore.

reset-pglite.ts JSDoc extended with the canonical 4-line PGLite block
(beforeAll create + afterAll disconnect + beforeEach reset). The lint
script in the next commit enforces this exact shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Grep-based lint enforcing 4 rules on non-serial unit test files:
  R1: no process.env mutations (use withEnv() or rename to *.serial.test.ts)
  R2: no mock.module() (rename to *.serial.test.ts)
  R3: new PGLiteEngine( only inside beforeAll() context
  R4: PGLiteEngine creators must pair with afterAll{disconnect}

Wired into 'bun run verify' and 'bun run check:all' (NOT 'bun run test'
which is the parallel runner script with no pre-check chain). Matches
the existing scripts/check-*.sh family shape (jsonb, progress, etc).

51 baseline violators captured in scripts/check-test-isolation.allowlist.
List MUST shrink over time — entries removed by v0.26.8 (env sweep) and
v0.26.9 (PGLite sweep). New files cannot be added.

CLAUDE.md ## Testing section extended with R1-R4 rules table, the
canonical 4-line PGLite block, withEnv pattern, and when-to-quarantine
guidance.

16 fixture-driven test cases for the lint: clean, R1 (5 patterns + 1
negative), R2, R3 (top-level vs in-beforeAll), R4 (missing disconnect),
*.serial.test.ts skip, test/e2e/ skip, allowlist (3 cases).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both files use mock.module(...) at top level — leaks across files in
the same shard process. The check-test-isolation lint (R2) bans this
pattern in non-serial files; quarantine is the escape hatch.

Per v0.26.7 plan D5: prefer quarantine over DI on runCycle/runEmbed.
Production signatures stay frozen; tests run at --max-concurrency=1
in the serial post-pass (the existing pattern shipped in v0.26.4 for
brain-registry and reconcile-links).

Quarantine count: 2 → 4. Cap raised to 10 informational per D15.

Renames:
  test/core/cycle.test.ts → test/core/cycle.serial.test.ts
  test/embed.test.ts      → test/embed.serial.test.ts

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- README.md "Contributing" line: point to bun run test + bun run verify (parallel fast loop)
- CONTRIBUTING.md "Running tests": rewrite for the v0.26.4/v0.26.7 test surface (parallel runner, verify, slow/serial/e2e tiers)
- CONTRIBUTING.md adds "Writing tests that survive the parallel loop" section: R1-R4 lint, canonical PGLite block, withEnv pattern, when to quarantine
- llms-full.txt regenerated to pick up the README + CONTRIBUTING changes

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@garrytan garrytan merged commit 058fe69 into master May 4, 2026
7 checks passed
zhaikong pushed a commit to zhaikong/gbrain that referenced this pull request Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant