Summary

Three production reliability fixes in one wave, rebuilt from closed community PRs #1414, #1416, #1421 (originally proposed by @garrytan-agents) with structural improvements from /plan-eng-review + codex outside-voice review.

The user-visible wins:

1. dream.* config you set via gbrain config set now actually reaches the cycle phase that reads it. Pre-fix, gbrain config set dream.synthesize.session_corpus_dir /path wrote successfully to DB but extract-atoms skipped with "no transcripts to process" because it read via file-plane-only loadConfig(). The systemic fix lives in loadConfigWithEngine()'s sparse-merge block.

2. Your background extract and sync cycles stop silently losing rows on PgBouncer connection drops. Pre-fix, ~30% of batched inserts threw No database connection: connect() has not been called and silently dropped 100 rows each. Now a single 500ms retry catches the recycle. Snapshot-before-clear contract means the retry sends the same data even if the producer wrote more during the delay.

3. gbrain ze-switch won't silently corrupt your brain when env vars override the switch. Pre-fix, GBRAIN_EMBEDDING_MODEL=openai:text-embedding-3-large gbrain ze-switch migrated schema to 2560d columns then embedded with OpenAI's 1536d model — the documented 716K-chunk damage incident. Now it refuses pre-apply (before any mutation) with an ASCII warning box. Same gate fires on --resume so there's no bypass.

4. extract_atoms stops creating duplicate atoms on re-runs across days. Pre-existing v0.41.2.0 bug: atom slugs were atoms/${todayDate()}/${title-slug}, so re-discovering the same content on day N+1 wrote a second atom. Now uses source-hash existence check — re-runs on unchanged content produce zero new atoms.

5. Page-based atom extraction — extract_atoms now also processes brain pages (meeting, source, article, video, book, original types), not just raw transcript files. Single SQL query with NOT EXISTS subquery handles discovery + idempotency in one round-trip.

6. New embedding_env_override doctor check — surfaces env-vs-DB embedding-config drift on every doctor run so the silent-corruption class can't repeat.

Closes: #1414, #1416, #1421 (all closed with detailed comments crediting the original contributor and explaining the structural improvements).

Test Coverage

NEW TEST FILES (61 new cases, all hermetic):
[+] test/extract-batch-retry.test.ts                (16 cases)
[+] test/extract-atoms-page-discovery.test.ts       (17 cases, PGLite)
[+] test/ze-switch-env-override.test.ts             (17 cases, PGLite + withEnv)
[+] test/doctor-embedding-env-override.test.ts      ( 7 cases, PGLite + withEnv)
[+] test/e2e/extract-atoms-discovery-sql.test.ts    ( 4 cases, real Postgres)

EXTENDED TEST FILES:
[+] test/loadConfig-merge.test.ts                   (+8 dream.* cases)
[+] test/cycle/extract-atoms-synthesize-concepts.test.ts (+1 critical regression case)

CRITICAL CONTRACTS PINNED:
  [★★★ TESTED] withRetry classifier covers GBrainError typed shape
  [★★★ TESTED] Snapshot-before-clear: producer mutation during delay doesn't poison retry
  [★★★ TESTED] applyRetrievalUpgrade refused = ZERO setConfig calls (spy assertion)
  [★★★ TESTED] resumeRetrievalUpgrade gate parity (no bypass path)
  [★★★ TESTED] extract_atoms re-run produces zero new atoms (idempotency)
  [★★★ TESTED] _pages:[] preserves v0.41.2.0 transcript-only PhaseResult byte-identical
  [★★★ TESTED] Real-Postgres parity for ANY($::text[]) + JSONB ->> + NOT EXISTS

VERIFICATION RESULTS:
  bun run verify                  → clean (privacy + jsonb + progress + wasm + typecheck)
  bun run test (parallel + serial) → 10,598 pass / 0 fail across 168+42 files
  bun run test:e2e (targeted)     → 18 pass / 0 fail on real Postgres

Pre-Landing Review

Completed via /plan-eng-review + codex outside-voice on 2026-05-25. 11 design decisions made (D1-D11). Codex caught 10 real correctness gaps in the plan (sourceId routing, NULL crash, gate ordering, resume bypass, tagged-union extension, Check.issues schema mismatch, dream.* env claim) — all incorporated as D9 before any code was written. See ~/.claude/plans/system-instruction-you-are-working-peppy-whale.md for the plan + verdict.

Adversarial Review

Cross-model synthesis: Claude structured + Claude adversarial subagent + codex outside-voice review all aligned on the correctness gaps. Codex's most valuable catch was sourceId not being threaded through engine.putPage in extract-atoms — would have shipped atoms-write-to-wrong-source on every federated brain. Fixed before any code landed.

Plan Completion

All 7 implementation tasks (T1-T7) completed. Plan file at ~/.claude/plans/system-instruction-you-are-working-peppy-whale.md. The single deferred item (per-atom idempotency to close the partial-failure-leaves-missing class) is captured in TODOS.md as v0.42+.

TODOS

Added two v0.42+ follow-up TODOs to TODOS.md:

• Per-atom idempotency via deterministic atom slug (closes the partial-failure documented limitation)
• Atom-slug consolidation migration (cleans up duplicate atoms from prior v0.41.2.0 runs on long-tenured brains)

Test plan

• bun run verify clean
• bun run test — 10,598 pass / 0 fail
• bun run test:serial — clean
• bun run test:e2e against real Postgres — 18 pass / 0 fail for new test surface
• Manual smoke: ze-switch refuses with env override, ze-switch with --ignore-env-override proceeds, doctor surfaces the disagreement

🤖 Generated with Claude Code