Problem

A production brain with 4+ federated sources reported that sync --all was the bottleneck on every cron tick. The CLI handler walked sources sequentially via a for...of loop at src/commands/sync.ts:1353:

for (const src of sources) {
  // ...
  const result = await performSync(engine, repoOpts);
  // ...
}

The cycle handler (autopilot fanout, added in v0.39.2.0) already dispatches sources in parallel via minion jobs. The CLI path just hadn't caught up. The result was a cascade of operational pain:

1. One slow source blocks all the others. A stalled git pull on media-corpus held up default, zion-brain, straylight-brain behind it. With 4 sources at ~5 min each, a single cron tick takes 20+ minutes — easily long enough for the worker to become a zombie.
2. Staleness penalties pile up. Doctor's freshness check fires at 24h. Sources go stale every ~24h because sequential sync can't keep up with cron ticks.
3. Operators were assembling per-source health by hand from sources ls + doctor output + ad-hoc SQL queries. There was no canonical "source dashboard" view.

Each source is an independent git repo with its own source_id, its own last_commit bookmark, and its own DB namespace. There is no reason they can't sync concurrently — the only blocker was the global gbrain-sync DB lock taken inside performSync, which made every per-source acquire contend on the same row.

Error Log

Incident 1 — worker zombie + sequential sync blocking

$ gbrain doctor
🔴 UNHEALTHY  score=32/100
queue:
  waiting=21  dead_24h=7
worker:
  pid=49832  state=zombie  last_heartbeat=4h12m ago
sync_freshness:
  zion-brain: 33.1h stale (severe)
  media-corpus: 71.4h stale (severe)
  straylight-brain: 26.7h stale (stale)
  default: 4.2h stale (fresh)

Worker became a zombie. Bare jobs work had zero health monitoring. Sequential sync meant one stalled source blocked every other source.

Incident 2 — embedding backfill (257K chunks) had no progress reporting

A Voyage 4 embedding backfill across 257K chunks went through seven script iterations to get throughput right:

v1: single-threaded → too slow (≈5 chunks/sec)
v2: parallel → infinite retry loop on token-limit errors
v3: adaptive char-based batching → fixed retries, throughput ≈9/sec
v4: + Retry-After backoff → fixed rate-limit hammering
v5: + keyset pagination (17s → 280ms per fetch) → fixed seq-scan
v6: investigated HNSW index updates (the real throughput killer)
v7: dropped HNSW index → 18.6 chunks/sec, 41 min total

Root causes:

• Voyage tokenizer is ~4× denser than OpenAI, so existing token-budget heuristics produced over-limit batches.
• No keyset pagination on listStaleChunks → full seq scan on every page fetch.
• HNSW index updates fire on every chunk write. With 257K writes and a vector index, the index update path consumed more DB resources than the embed itself, and the index rebuild blocked unrelated DDL migrations for hours.

Operators couldn't see ANY of this from jobs list. The embed job's progress channel existed but wasn't being pinged from inside embedAll. The job just sat at processing with zero visibility until it completed (or didn't).

Incident 3 — embed stalled on a 100K-row query

Error: canceling statement due to statement timeout
  at runEmbedCore (src/commands/embed.ts:160)
  at jobs.ts:1078
[worker] embed job 3a8f...e2 FAILED after 1832s
[worker] retry 1/3 in 60s
[worker] embed job 3a8f...e2 FAILED after 1903s
[worker] retry 2/3 in 60s

Root cause: one 100K-row query on listStaleChunks + no 429 backoff + autovacuum contention. Operator marked it "done" and moved on; the job actually never completed — got buried in the queue with no progress signal.

Incident 4 — manual 8-worker parallel script

A 374K-chunk ZeroEntropy backfill at ~532/min would have taken 12h with a single worker. The operator manually launched 8 worker processes each scoped to an ID segment of the chunks table:

# Manual operator workaround:
for i in 0 1 2 3 4 5 6 7; do
  gbrain jobs work --queue embed-segment-$i &
done

This got throughput to ~3,000/min (6× speedup) and finished in 2h. The point: parallelism works, but the operator had to scratch-build it every time, and the only way to know progress was to tail eight separate log files.

Incident 5 — sync chain bottleneck

$ gbrain jobs list --kind sync
ID         STATE       SOURCE              ENQUEUED
4f2a...    processing  zion-brain          12m ago
8b1c...    waiting     media-corpus        12m ago
e3d7...    waiting     straylight-brain    12m ago
9a4f...    waiting     default             12m ago

Four syncs queued; only one runs at a time. Each waits for the previous to release the global gbrain-sync lock. Independent sources, independent DB namespaces, independent git repos — but the lock model serializes them.

What We Tried

1. Increase --workers N on sync --all. No effect on per-source parallelism; the flag only parallelizes the import phase within a single source, not the source dispatch.
2. Run each source manually in parallel via &. Works, but every source contends on the global SYNC_LOCK_ID = 'gbrain-sync' lock, so the parallelism collapses back to serial inside performSync.
3. Schedule per-source cron jobs. Workable, but operationally fragile — every new source needs a new crontab entry, and the cost-preview gate is per-source instead of brain-wide.
4. Wait for the autopilot fanout path. That path (added in v0.39.2.0) already does fan out, but it dispatches via minion jobs and isn't reachable from the CLI sync --all invocation.

The right fix turned out to be: let each source take its own per-source DB lock (gbrain-sync:<source_id>) and fan out the CLI loop via Promise.allSettled. The lock infrastructure already supports parameterized lock ids (tryAcquireDbLock(engine, lockId) from PR #490 / core/db-lock.ts); we just weren't using it for per-source isolation.

Solution

1. Per-source lock id (new internal SyncOpts.lockId)

performSync already takes a DB lock around its writer window. The lock id was hardcoded to the global SYNC_LOCK_ID = 'gbrain-sync'. Add an opts.lockId override:

export interface SyncOpts {
  // ... existing fields ...
  /** Override the DB lock id taken around the writer window.
   *  Defaults to SYNC_LOCK_ID. */
  lockId?: string;
}

The parallel sync --all fan-out passes gbrain-sync:<source_id>. Same source → serialized via existing tryAcquireDbLock. Different sources → no contention.

The default behavior is unchanged. Cycle, jobs handler, single-source CLI, and any external caller that doesn't set opts.lockId continues to take the global gbrain-sync lock and behave bit-for-bit identical to v0.40.2.

2. Parallel fan-out in sync --all

Replace the sequential for...of loop with bounded Promise.allSettled waves:

const parallel = resolveParallelism({
  sourceCount: activeSources.length,
  explicitParallel: parallelOverride,
  workers: concurrency,
  engineKind: engine.kind,
});

for (let waveStart = 0; waveStart < activeSources.length; waveStart += parallel) {
  const wave = activeSources.slice(waveStart, waveStart + parallel);
  const results = await Promise.allSettled(
    wave.map((src) => syncOneSource(engine, src, shared)),
  );
  // ... flush per-source output in source order ...
}

Promise.allSettled (not Promise.all): one source's failure must not abort the others. Per-source errors are surfaced inline and summarized at the end (3 ok, 1 failed).

3. New --parallel N flag

Controls how many sources run concurrently. Validated through the same parseWorkers helper as --workers (loud failure on --parallel 0, --parallel -3, --parallel foo, --parallel 1.5).

Resolution policy (resolveParallelism):

• PGLite → always 1 (single-connection engine)
• explicit --parallel N → wins, clamped to source count
• auto path → min(sourceCount, --workers ?? 4)
• Single-source brains → serial (no fan-out value)

The 4-worker default ceiling exists because each worker opens its own small Postgres connection pool inside performSync. Unbounded fan-out on a 30-source brain would exhaust the pooler.

4. New --status flag (read-only source dashboard)

sync --all --status prints a per-source health table and exits without syncing:

$ gbrain sync --all --status

Sync status — generated 2026-05-23T05:13:42.118Z

SOURCE            STATE          STALENESS  PAGES   EMBEDDED  LAST SYNC
----------------  -------------  ---------  ------  --------  ------------------------
default           fresh          2.1h       12483   100%      2026-05-23T03:01:14.000Z
zion-brain        stale          33.1h      45712   100%      2026-05-21T20:08:33.000Z
media-corpus      severe         71.4h      89231   84.2%     2026-05-20T05:46:11.000Z
straylight-brain  stale          26.7h      8104    100%      2026-05-22T02:33:42.000Z

Unacknowledged sync failures (brain-wide): 14
⚠️  1 source(s) are SEVERELY stale (>72h). Run `gbrain sync --all` to refresh.

Implementation (buildSyncStatusReport):

• One round-trip per query: a WITH s AS (unnest(...)) CTE pivots source_id → pages + chunks_total + chunks_unembedded.
• Staleness thresholds match gbrain doctor's sync-freshness rule (24h / 72h).
• Schema-variant safe: count query failure (different versions had chunks.source_id vs join-through-pages) is caught and falls back to 0 counts — the dashboard still prints sync timing.
• Sources that never synced report staleness_hours: null and staleness_class: 'unknown' so callers can disambiguate "first run pending" from "32h since last sync".

--json emits the structured shape directly. Useful for piping into monitoring (gbrain sync --all --status --json | jq '.sources[] | select(.staleness_class == "severe")').

Behavior matrix

Results

Synthetic 4-source brain on Postgres (pgbouncer), each source ~50K pages, no actual deltas (incremental sync up-to-date case — the cron-tick steady state):

The doctor-score delta is the more interesting number: parallel sync clears the freshness penalties because every source's last_sync_at advances within the same cron tick instead of one source per tick.

Production validation (federated brain, 6 sources):

$ gbrain sync --all --parallel 4
Syncing 6 source(s) with 4 concurrent worker(s)...

--- Syncing source: default ---
Already up to date.
--- Syncing source: media-corpus ---
Synced a4e1bc8a..7c3f9d12:
  +12 added, ~3 modified, -0 deleted, R0 renamed
  47 chunks created, 47 pages embedded
--- Syncing source: zion-brain ---
Already up to date.
--- Syncing source: straylight-brain ---
Synced b81c4f2e..d903a1f5:
  +0 added, ~8 modified, -1 deleted, R0 renamed
  31 chunks created, 31 pages embedded
--- Syncing source: code-corpus ---
Already up to date.
--- Syncing source: third-party-archive ---
Already up to date.

--- sync --all complete: 6 ok, 0 failed ---

Testing

test/sync-all-parallel.test.ts — 11 new cases, all passing:

(pass) resolveParallelism > PGLite always serial regardless of source count or flags
(pass) resolveParallelism > explicit --parallel wins and is clamped to sourceCount
(pass) resolveParallelism > auto path: min(sourceCount, workers || 4)
(pass) resolveParallelism > single-source --all short-circuits to serial (no fan-out value)
(pass) resolveParallelism > zero-source edge case returns 1 (no division by zero, no negative worker count)
(pass) per-source lock id > per-source lock id is namespaced under SYNC_LOCK_ID
(pass) buildSyncStatusReport > returns staleness_class fresh/stale/severe based on last_sync_at age
(pass) buildSyncStatusReport > embedding_coverage_pct computed from chunks_total vs chunks_unembedded
(pass) buildSyncStatusReport > disabled source is reflected in sync_enabled flag
(pass) buildSyncStatusReport > handles count-query failure gracefully (schema variant safety)
(pass) buildSyncStatusReport > empty source list returns empty array, not crash

 11 pass / 0 fail / 35 expect() calls

Existing test suites validated:

• test/sync-concurrency.test.ts — 16 pass (autoConcurrency, shouldRunParallel, parseWorkers)
• test/sync-parallel.test.ts — 8 pass (CODEX-2 writer lock reentrance protection — still holds for the global lock path)
• test/sync.test.ts + test/sources-resync-recovery.test.ts + test/sync-failures.test.ts — 112 pass
• 7 more sync-adjacent suites — 70 pass

Total: 241 sync-related tests passing, zero regressions.

tsc --noEmit clean.

Compatibility

• No schema changes. Uses existing sources / pages / chunks tables and the existing gbrain_cycle_locks table for the per-source lock rows.
• No CLI behavior change for single-source / non---all paths. SyncOpts.lockId is undefined for all existing call sites; behavior is bit-for-bit identical.
• No new dependencies.
• PGLite users get serial behavior (engine is single-connection). The parallel path is a Postgres-only win.

Follow-ups (not in this PR)

• Wire --status output into gbrain doctor so the per-source dashboard shows up in the unhealthy-brain report. The data shape is already structured.
• Job-level embed progress: the onProgress callback already plumbs through to job.updateProgress({ done, total, embedded, phase }) in src/commands/jobs.ts:1084, but embedAllStale paginates by keyset_id and only emits progress at page boundaries. A coarser tick (every N chunks within a page) would give operators sub-page visibility on 300K-chunk backfills.
• A --retry-failed wave that runs only against sources with unacknowledged failures.