Summary

Bulk deletes no longer jam sync for two days. A single commit deleting 73K files used to take ~5 hours (146K individual DB round-trips) and time out the sync cron every run. This release batches both halves — Phase 1 batched slug resolution + Phase 2 batched DELETE — so the same scenario lands in ~2 minutes. The per-batch shape inherits the v0.41.19 Supavisor withRetry cathedral so transient connection blips don't lose work.

Also closes a latent silent-no-op bug class on gbrain sync without --source: the pre-fix per-file resolver could pull a slug from the wrong source and then issue a delete against source_id='default', which would silently no-op. Files stayed in the brain after the file was gone from disk. The new path scopes lookup AND delete to the same source.

Scope honesty: this fixes the delete-specific hotspot only. The full RFC #1472 cathedral (per-source --timeout, --break-lock-if-stale, --independent mode) already landed in v0.41.15.0 (#1506). Renames, imports, extraction, embedding, and lock contention can still cause sync slowness in their own ways.

Provenance

Productionized from community PR #1538 (@garrytan-agents) per CLAUDE.md's community-PR-wave workflow. The original PR couldn't run CI with secrets because GitHub doesn't pass them to fork PRs from non-collaborators. Moving to a base-repo branch + 12 review-driven improvements layered on top via /plan-eng-review + codex outside voice:

Engine surface

// BrainEngine — non-optional, no fallback branching
deletePages(slugs: string[], opts?: { sourceId?: string; signal?: AbortSignal }): Promise<string[]>

PostgresEngine wraps each batch in withRetry(BULK_RETRY_OPTS) with auditSite: 'deletePages'. PGLite runs the bare DELETE (single-writer, no Supavisor concern). BATCH_AUDIT_SITES enum extended; test/core/retry.test.ts regression pin updated.

Sync rewrite

src/commands/sync.ts delete loop replaced with the two-phase batched path:

• Phase 1: SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2 ORDER BY slug ASC (BATCH_SIZE=100). ORDER BY pins deterministic Map collapse on duplicate source_paths.
• Phase 2: engine.deletePages(batch, deleteOpts) (BATCH_SIZE=100). Returns actually-deleted slugs; caller's pagesAffected appends INPUT batch for parity with pre-fix (downstream extract/embed sweeps no-op on missing pages).

New sync.deletes.resolve progress event fires for Phase 1; existing sync.deletes covers Phase 2. Per-batch --timeout abort checks preserve the v0.41.13.0 D-V4-2 contract.

Tests

• test/sync-batch-deletes.test.ts (NEW, 12 PGLite cases): engine-level contract (D10), missing slugs, abort prefix, source scoping (D6 + default), 250-slug internal batching, D7 deterministic ORDER BY, source_path-NULL fallback, D9 query-count regression, D1 structural source-text regression.
• test/e2e/sync.test.ts (extended, 3 real-Postgres cases): FK CASCADE through content_chunks (100-row batch), end-to-end 250-file delete batches into ≤3 SELECT + ≤3 deletePages calls, abort mid-sync returns partial('timeout') with empty pagesAffected.
• test/core/retry.test.ts regression-pin extended with 'deletePages'.

Docs

• docs/ENGINES.md documents deletePages alongside deletePage.
• docs/progress-events.md documents sync.deletes.resolve.

Closes / supersedes

Supersedes #1538 (same fix, base-repo branch so CI has secret access, plus the 12 improvements above).

Test plan

• bun run typecheck — pass
• bun run verify — 28/28 checks pass
• bun test test/sync-batch-deletes.test.ts — 12/12 (PGLite)
• bun test test/core/retry.test.ts — 37/37 (regression-pin updated)
• bun test test/build-llms.test.ts — 7/7 (llms.txt regenerated for CHANGELOG)
• bun run test:serial — 50/50 serial files
• bun run test — 3105 pass / 1 fail (BATCH_AUDIT_SITES regression-pin, fixed in same commit)
• E2E real-Postgres: bun test test/e2e/sync.test.ts against pgvector container — 17/17

Notes

• VERSION + package.json bumped to 0.41.25.0 (master at 0.41.22.1; v0.41.23.0 and v0.41.24.0 claimed by sibling PRs).
• llms-full.txt regenerated for the CHANGELOG entry per the CLAUDE.md mandatory-regen-after-CHANGELOG-edit rule.
• Two follow-up TODOs filed in TODOS.md: composite (source_id, source_path) partial index for the optimal Phase-1 hot path on 4-source federated brains (P3, latent).

🤖 Generated with Claude Code