v0.42.22.0 — supervisor progress watchdog + worker DB self-defense (closes #1801)

The bug (production incident)

Behind a Supabase transaction pooler, a child jobs work process's DB pool died (write CONNECTION_ENDED on every background tick) and never recovered — but the process stayed alive the whole time. Jobs piled to 57 waiting / 0 active for ~15h; every long autopilot-cycle dead-lettered with max stalled count exceeded. No crash, no alert, no self-heal. The supervisor logged no_recent_completions once a minute for 914 minutes and took zero action. Manual fix: kill the supervisor tree so the respawn rebuilds a fresh pool.

Distinct from #1720/#1745 (crash-loop) and #1678 (RSS watchdog SIGKILL). Here the worker never exits, so no respawn path is ever entered. Every process-liveness watchdog is defeated — the process passes ps/kill -0. The gap: everything watched liveness, nothing watched forward progress.

The fix (two independent layers + surfacing)

Layer 1 — worker DB self-defense under supervision (worker.ts): the worker's SELECT 1 liveness probe was disabled whenever a supervisor was watching. It now runs under supervision too, so a worker whose own pool is dead self-exits db_dead (~3 min) and the supervisor respawns it with a fresh pool. Stall detection stays supervised-off (the supervisor owns that now).

Layer 2 — supervisor progress watchdog (supervisor.ts + child-worker-supervisor.ts): when a queue has claimable work, zero live-lock active jobs, and stale completions across K health checks while the child is alive, the supervisor restarts it. Catches every wedge cause, not just dead pools. Name+queue-scoped (no false-positive on unhandled job types), active_healthy-gated (a worker that died mid-job leaving an expired-lock row does NOT mask the wedge), startup-grace + loop-budget bounded, restart accounted as an intentional self-heal so it never trips max_crashes.

Surfacing (doctor.ts + queue.ts + jobs.ts): gbrain jobs stats prints a WEDGED QUEUE line; gbrain doctor reports a per-queue wedged_queue health error. Also fixed a latent bug: the remote queue_health doctor check queried a non-existent state column (errored every run, silently returned "No queue activity") — corrected to status.

Pre-existing bug fixed in passing: killChild gated on .killed ("signal sent") instead of liveness, so the follow-up SIGKILL after an ignored SIGTERM was a silent no-op — in the new escalation AND the existing shutdown() drain. Now gates on exitCode/signalCode === null.

What's in this PR (4 commits)

1. fix(minions): supervisor progress watchdog + worker DB self-defense (#1801) — the feature.
2. Merge origin/master — reconciles with v0.42.16.0 feat(doctor): brain health as a solved problem — cause-ranked doctor + OOM-loop line + auto-drain + pool-reap (#1685) #1802 (v0.42.16.0 doctor self-heal: worker_oom_loop, pool_reap_health, extract-atoms-drain). Both check sets coexist; they cover different failure modes.
3. fix(minions): wedge_restart_loop one-shot + … — adversarial-review findings.
4. chore: bump version and changelog (v0.42.22.0).

Tests

New: supervisor-wedge (decision logic via seam + PGLite SQL semantics for every reviewed edge case), worker-supervised-db-probe, doctor-wedged-queue, queue-getstats-wedge, plus child-worker-supervisor additions (behavioral restart coverage + structural regression guards for the killChild fix). Full suite + bun run verify (30 checks) green.

Review

Plan went through /plan-eng-review + a /codex outside-voice pass (16 findings, all folded in — including the killChild .killed no-op, the expired-lock-suppresses-wedge bug, and the SIGTERM-counts-as-crash bug). A pre-landing adversarial pass on the diff caught the wedge_restart_loop per-tick spam (fixed in commit 3).

🤖 Generated with Claude Code

Documentation

• docs/guides/queue-operations-runbook.md: new "The worker is alive but wedged (dead pool)" section — documents the Supervisor never restarts an alive-but-wedged worker: dead worker DB pool + no-op no_recent_completions warn = silent 15h processing halt #1801 case, the automatic self-heal (worker self-exit on dead pool + supervisor progress-watchdog restart), the --wedge-restart-* tuning flags, and the new signals (gbrain jobs stats WEDGED line, gbrain doctor wedged_queue check) + manual fix.
• docs/architecture/KEY_FILES.md: current-state updates to the worker.ts (DB probe runs under supervision), supervisor.ts (progress watchdog + queryWedgeSignals), and child-worker-supervisor.ts (killChild liveness fix, restartCurrentChild, wedge_restart cause) entries.
• CHANGELOG.md: v0.42.22.0 entry (ELI10 lead + how-to + before/after table + review findings).
• CLAUDE.md: not touched — the repo keeps key-files detail in docs/architecture/KEY_FILES.md (anti-bloat guard bans version markers in CLAUDE.md). llms.txt/llms-full.txt regenerated, no diff.

Coverage: all shipped surface (supervisor flags, wedged_queue check, jobs stats WEDGED line, env knobs) has reference + how-to coverage. No diagram drift.