…max-waiting CLI

Addresses three production-hardening findings from the CEO + Eng + Codex
adversarial review of PR #379:

D2/H2: maxWaiting was TOCTOU-racy — two concurrent submitters could both
see waitingCount < max and both insert. Wrap the count+select+insert in
pg_advisory_xact_lock keyed on (name, queue). Serializes concurrent
decisions for the SAME key while leaving different keys fully parallel.
Lock auto-releases on txn commit/rollback — no cleanup path to leak.
Also fix the missing queue-scope bug: count and select now filter on
(name, queue) not name alone, so cross-queue same-name jobs don't
suppress each other.

D3/H3: resolveWorkerConcurrency silently accepted NaN / 0 / negative from
parseInt. `inFlight.size < NaN` is always false → worker claims nothing →
silent wedge from a single-typo env var. Clamp to ≥1 with a loud stderr
warning naming the bad value.

D5/H5: `gbrain jobs submit` never parsed `--max-waiting N` despite the
MinionJobInput field. Wire the flag with clamp [1, 100], mirror
`--max-stalled`. Extract `parseMaxWaitingFlag` for unit testing.

Q1: Silent coalesce was invisible by design. New
src/core/minions/backpressure-audit.ts mirrors shell-audit.ts's ISO-week
JSONL pattern: `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Coalesce
events write one JSONL line with (queue, name, waiting_count, max_waiting,
returned_job_id, ts). Best-effort — disk-full never blocks submission.

A2: `gbrain jobs smoke --wedge-rescue` new opt-in regression case.
Forges a wedged-worker row state, invokes handleStalled + handleTimeouts
+ handleWallClockTimeouts in order, asserts only wall-clock evicts.
Mirrors the v0.14.3 `--sigkill-rescue` shape.

Tests: 23 new unit cases in test/minions.test.ts covering wall-clock
timeout (3 cases + non-interference with handleTimeouts), maxWaiting
(coalesce, clamp 0, floor, concurrent-submitter race via Promise.all,
cross-queue isolation, unset fallthrough), concurrency clamp (7 cases
incl. NaN/0/negative), parseMaxWaitingFlag (5 cases), backpressure
audit file write.

Part of v0.19.1 plan at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…book

A5 / D4: New `queue_health` check in `gbrain doctor`. Postgres-only (PGLite
has no multi-process worker surface). Two subchecks, both cheap (single
SELECT each, status-index-covered):

- stalled-forever: any active job with started_at > 1h. Surfaces the
  worst offenders (top 5 by started_at ASC) with `gbrain jobs get/cancel`
  fix hints. The incident that motivated v0.19.1 ran 90+ min before the
  operator noticed.
- waiting-depth: per-name waiting count exceeds threshold. Default 10,
  overridable via GBRAIN_QUEUE_WAITING_THRESHOLD env (D9). Signals a
  submitter probably needs maxWaiting set.

Worker-heartbeat subcheck from the original plan dropped (D4/H4): no
minion_workers table exists, and lock_until-on-active-jobs is a lossy
proxy that can't distinguish idle-worker from dead-worker. Tracked as
follow-up B7.

A4: --no-worker peer-liveness probe in autopilot. When --no-worker is
set, every cycle runs a cheap SELECT checking for any active job whose
lock_until was refreshed in the last 2 minutes. After 3 consecutive
idle ticks, logs a loud WARNING naming the silent-wedge vector and
referencing B7 as the ground-truth follow-up. Re-arms on next live
signal so the warning doesn't spam every cycle.

A6: New docs/guides/queue-operations-runbook.md (one viewport, ~60
lines). "My queue looks wedged — what do I run?" in order of
escalation. What each doctor subcheck means. Self-check for the
--no-worker / no-worker-running footgun.

CLAUDE.md: key-files updates for handleWallClockTimeouts (v0.19.0 Layer
3 kill shot), maxWaiting advisory-lock rewrite (v0.19.1 D2), queue_health
doctor check (v0.19.1 D4), and backpressure-audit.ts.

Tests: all 143 minions + 13 doctor unit tests pass. No new test cases
required in Lane B; the doctor queue_health exercise is in the E2E
verification step (needs real PG to produce meaningful stalled-forever
rows). The --no-worker probe is exercised by the smoke case's wedge
setup in Lane A.

README: unchanged. Existing `gbrain jobs submit` examples don't show
--max-stalled, so no --max-waiting precedent to extend per A6 conditional.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

VERSION: 0.19.0 → 0.19.1 (patch; bug-fix-dominant, no schema change,
no new user-facing vocabulary).

CHANGELOG: new v0.19.1 entry at the top with the full release-summary
template per CLAUDE.md — bold two-line headline, lead paragraph, "numbers
that matter" before/after table measured against the real incident,
"what this means for OpenClaw users" closer, required "To take
advantage of v0.19.1" block naming the worker-restart requirement,
itemized changes by area, and "For contributors" section closing the
loop on the stale autopilot-idempotency narrative the CEO review was
based on.

Mechanism reframing per D1/H1: the 18-job pile-up was NOT caused by
missing idempotency (autopilot already passes
`idempotency_key: autopilot-cycle:${slot}` at autopilot.ts:241). The
18 jobs were 18 DIFFERENT slots stacking up behind the wedged one.
`maxWaiting` still caps the pile; the incident just wasn't about
idempotency. Adversarial review caught this before ship.

SPEC.md: deleted from repo root. It was Wintermute's planning artifact
for the original PR, not a shipped spec. Design docs belong under
docs/designs/ per repo convention; leaving one at repo root set a
precedent this repo doesn't want (A7/D11). CHANGELOG + the plan file
at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md are
the durable artifacts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Master shipped its own v0.19.1 (smoke-test skillpack, PR #369) and then
v0.20.0 (BrainBench extraction to sibling repo, PR #195) while this
branch was in review. Bumping our release from v0.19.1 to v0.20.1
follows the CLAUDE.md rule: "VERSION must be higher than master's."

Resolved conflicts:

- VERSION: 0.19.1 (ours) + 0.20.0 (master) → 0.20.1
- package.json: same bump applied to the version field
- CHANGELOG.md: our queue-resilience entry renamed from v0.19.1 to
  v0.20.1 (6 inline refs updated across the body: numbers-that-matter
  table, "To take advantage" block, pre-v0.20.1 code reference,
  adversarial-review mention, and the v0.19.2 → v0.20.2 deferral
  reference for composite indexes). Entry stays at the top of the
  file, followed by master's v0.20.0 (BrainBench) and v0.19.1
  (smoke-test skillpack). Sequence is now 0.20.1 → 0.20.0 → 0.19.1 →
  0.19.0 → 0.18.2 → ...

Rebuilt binary reports gbrain 0.20.1.

Pre-merge verification carried forward:
- 143 minions + 13 doctor unit tests pass
- typecheck clean
- 189 E2E tests pass against real Postgres 16 + pgvector
- All 3 smoke cases pass (basic, --sigkill-rescue, --wedge-rescue)
- queue_health doctor check fires correctly on a forged stalled-forever job

No source changes — conflict resolution was version-label surgery only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Master shipped v0.20.2 (jobs supervisor, PR #364) while this branch was
in review. Bumping our release from v0.20.1 to v0.20.3 follows the
CLAUDE.md "VERSION must be higher than master's" rule.

Resolved conflicts:

- VERSION: 0.20.1 (ours) + 0.20.2 (master) → 0.20.3
- package.json: version bumped to match
- CHANGELOG.md: our queue-resilience entry renamed from v0.20.1 to
  v0.20.3 throughout (5 inline refs updated: numbers-that-matter
  table, "To take advantage" block, pre-v0.20.3 code reference,
  adversarial-review mention). The "composite indexes deferred to
  v0.20.2" follow-up reference updated to v0.20.4 because master
  already took v0.20.2 for the supervisor feature.

Auto-merged cleanly:

- src/commands/doctor.ts: master added supervisor health check
  (filesystem-only, PID liveness + audit tail) alongside our new
  queue_health check. No conflict — different sections.
- src/commands/jobs.ts: master added `jobs supervisor` subcommand with
  start / status / stop variants. No conflict with our --max-waiting
  CLI wiring or --wedge-rescue smoke case.

New files pulled in from master (supervisor feature):
- src/core/minions/supervisor.ts (MinionSupervisor class)
- src/core/minions/handlers/supervisor-audit.ts (JSONL audit)
- test/supervisor.test.ts (13 cases)
- test/fixtures/supervisor-runner.ts (integration test helper)

Post-merge verification:
- `bun run typecheck` — clean
- `bun test test/minions.test.ts test/doctor.test.ts` — 156 pass
- `bun run build` → gbrain 0.20.3
- CHANGELOG version sequence: 0.20.3 → 0.20.2 → 0.20.0 → 0.19.1 → ...

No source changes — conflict resolution was version-label surgery only.
The queue-resilience code, tests, and docs landed from master cleanly
alongside the new supervisor feature.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Two CI failure clusters, both pre-existing but surfaced by the v0.20.3
merge:

1) test/minions-shell.test.ts — 12 failing cases. The shell handler
   throws UnrecoverableError when GBRAIN_ALLOW_SHELL_JOBS !== '1' (the
   production RCE guard at shell.ts:210). The unit tests exercise
   handler mechanics, not the guard, but never set the env var — so
   every invocation exits through the guard path instead of the code
   being tested. Fix: set GBRAIN_ALLOW_SHELL_JOBS=1 in beforeAll,
   restore in afterAll. The env-guard IS still tested separately via
   the test/minions.test.ts case added in v0.20.3 Lane A which toggles
   the var itself.

2) llms-full.txt — stale against CLAUDE.md. Key-files entries for
   queue.ts, doctor.ts, and the new backpressure-audit.ts updated in
   v0.20.3 Lane B triggered the build-llms drift guard. Regenerated
   via `bun run build:llms`; no behavior change, just the inlined-docs
   bundle catching up to source.

Full test run: 2367 pass, 0 fail across 137 files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>