Verified end-to-end on production

Pulled this branch into a live install (v0.41.26.0 base, direct Postgres, 749 pages) and ran gbrain dream. Comparing against the same brain on origin/master run earlier today:

extract.timeline_fs finalize batch

Before:  Timeline: created  0 entries from 706 pages   (13 timeline rows lost)
After:   Timeline: created 37 entries from 706 pages

extract.links_fs connection-blip handling

Audit log entries from ~/.gbrain/audit/batch-retry-2026-W22.jsonl:

Before (24h pre-patch):  192 exhausted retries, worst site extract.links_fs = 174
After (patched run):     1 event total
  ts=15:00:31  site=extract.links_fs  attempt=1  outcome=success  delay=1000ms
               err="No database connection: connect() has not been cal[led]"

The singleton went null mid-extract.links_fs; withRetry fired onRetry, my patch called this.reconnect(), the next attempt found a healthy pool, and the batch completed. One audit row, outcome success, vs. the pre-patch shape of attempt 1/2/3 all exhausted + "batch error (N rows lost)" line.

Doctor

batch_retry_health and cycle_freshness still FAIL in this snapshot — they're 24h-windowed metrics counting the pre-patch exhausted retries that haven't decayed yet. The audit-log delta is the leading indicator and it's clean.

Out-of-scope failures observed in the same run

sync and synthesize phases hit the same "No database connection" at non-batchRetry call sites — exactly what the PR description called out as separate work. Not regressions from this patch; the singleton was equally null there before. Will follow up in a second PR once this direction is approved.

Hit this exact No database connection: connect() has not been called class in production on a long-running gbrain autopilot daemon (v0.42.1.0, instance-pool engine against a shared local Postgres). Wanted to flag a piece that sits underneath this PR, because on current local/integration the batchRetry reconnect callback (reconnect: () => this.reconnect()) is already wired in — yet the crash still happens. The remaining defect is in reconnect() itself.

Symptom: worker runs a full cycle's early phases fine, then late phases throw No database connection, executeJob crashes unhandled, worker exits code=1, respawns, repeats every ~5 min. conversation_facts_backfill is the phase that crashes the worker, but it's a red herring — it's default-OFF, so its only DB touch is loadCfg's engine.getConfig() x7, i.e. it's just the first hard call after the engine already died mid-cycle. The Promotion error: No database connection lines from the worker's promoteDelayed() loop confirm _sql was already null process-wide.

Root cause: PostgresEngine.reconnect() disconnects the live pool before attempting connect():

try { await this.disconnect(); } catch {}   // _sql = null HERE
await this.connect(this._savedConfig);        // if THIS throws, _sql stays null forever

So when a transient blip fires the retry→reconnect path and Postgres is still unreachable for those few seconds, connect() throws and _sql is left null permanently. Every subsequent call uses the accessor this._sql || db.getConnection(); with _sql null it falls to the module singleton, which the autopilot process never connected → throw. Non-bulk methods (getConfig, per-phase reads) have no retry wrapper, so the first one after the failed reconnect throws unhandled and kills the worker.

This is why wiring reconnect into batchRetry (this PR) is necessary but not sufficient: the reconnect it calls can itself brick the engine.

Fix that worked for us — build-then-swap (don't tear down the old pool until the new one is proven live; restore on failure so the next query self-heals):

async reconnect(): Promise<void> {
  if (!this._savedConfig || this._reconnecting) return;
  this._reconnecting = true;
  const oldSql = this._sql;
  const oldManager = this.connectionManager;
  try {
    this._sql = null;                       // force connect() to build fresh
    await this.connect(this._savedConfig);  // validates via SELECT 1 before returning
    if (oldSql) { try { await oldSql.end({ timeout: 5 }); } catch {} }
  } catch (err) {
    if (this._sql && this._sql !== oldSql) { try { await this._sql.end({ timeout: 5 }); } catch {} }
    this._sql = oldSql;
    this.connectionManager = oldManager;
    throw err;                              // let batchRetry's backoff handle it
  } finally {
    this._reconnecting = false;
  }
}

Green on test/core/retry-reconnect.test.ts + retry.test.ts (42 pass), test/e2e/db-singleton-shared-recovery.test.ts (3 pass), and the disconnect-idempotency e2e (7 pass). Looks complementary to the module-singleton refcount work in #1754/#1758 too — that layer and this instance-pool layer are orthogonal; both seem needed. Happy to fold this into whichever PR lands as the canonical fix.

@rayers thank you, this is excellent root-cause work, and it matches what I see on master: the instance-pool branch of reconnect() runs disconnect() (which nulls _sql) before connect(), and the catch only logs and rethrows, so a connect() failure during the blip leaves _sql null with no self-heal. Build-then-swap is the right shape: hold the old pool until the new one validates, restore it on failure so the next query recovers. I'd land that as its own PR since it is cleanly scoped and orthogonal to the #1754 refcount work.

Closing this PR as superseded. The change it proposed (await onRetry plus wiring reconnect into the batch retry path) has since landed on master independently: retry.ts now awaits onRetry and exposes a dedicated reconnect opt, and batchRetry passes reconnect: (ctx) => this.reconnect(ctx). So this diff is redundant against current master.

The two real remaining layers, both confirmed, are now split out:

1. reconnect() itself is not crash-safe on the instance-pool path (your build-then-swap), yours to land when ready.
2. The non-batch call sites (getConfig, per-phase reads) had no retry wrapper, so the first hard DB call after a mid-cycle disconnect threw unhandled and crashed the worker. I took that one as fix(retry): reconnect on null instance pool in non-batch config reads (#1593 follow-up) #1891.

Thanks again for the deep dig.

Landed the build-then-swap reconnect crash-safety as #1906 (Layer 1 from your split). Rebased onto current master, keeps the reap-detection + pool-recovery audit, and updates the recovery-contract guard. Complementary to your #1891 (non-batch call sites). Thanks for the clean split here.