v0.40.5.0 Federated Sync v2 — parallel source sync + push triggers + per-source health#1322
Merged
Conversation
Resolve VERSION → 0.41.0.0 (master shipped v0.40.x cathedral). Resolve package.json verify script (keep both check:source-config-leak + check:no-pii-agent-voice). Resolve CHANGELOG (renumber wave entry from 0.40.0.0 → 0.41.0.0; preserve all master entries). Resolve migrate.ts (renumber sources_github_repo_index migration v87 → v89; master added v87+v88). Rename skills/migrations/v0.40.0.md → v0.41.0.md. Refresh bun.lock + llms-full.txt. Bump beforeAll timeouts to 30s on 9 new PGLite-using test files (89 migrations now exceed the 5s default cold-start budget). Auto-merged cleanly: autopilot.ts (D17 freshness gate coexists with master's v0.39.2 per-source fanout; different job names + idempotency- key prefixes), doctor.ts, jobs.ts, serve-http.ts, sync.ts.
…per-source health
Bump VERSION + package.json + CHANGELOG header + migration walkthrough filename
to v0.40.5.0 (claiming the next free slot in the v0.40.x patch series after
master's v0.40.1.0).
What ships (6 components, all behind sync.federated_v2 feature flag default-on):
1. Per-source sync lock — syncLockId(sourceId), phantom-redirect parity
2. Parallel sync --all — pMapAllSettled fan-out, --max-sources N cap
3. embed-backfill minion handler — D2 per-source lock + D6 $10/job budget + D15.1
fire-and-forget submission + D19 source-level cooldown + 24h $25 rolling cap
4. sync trigger CLI + POST /webhooks/github — HMAC-verified (60 req/min/IP),
X-GitHub-Event=push + ref filter against tracked_branch
5. sources status + federation_health doctor — batched GROUP BY pipeline
(4 queries instead of 6×N per-source roundtrips)
6. sources federate/unfederate hook — auto-submit embed-backfill on flip
Correctness fixes (unconditional):
- D21: sync.ts:959 facts backstop now passes sourceId to engine.getPage
- D15.4: redactSourceConfig + CI guard prevent webhook_secret leak
- D15.5: safeHexEqual extracted to src/core/timing-safe.ts
Schema:
- Migration v89 (sources_github_repo_index): partial expression index on
config->>'github_repo' for fast webhook source-lookup
Tests:
- 14 new test files, 112 cases. 4 IRON-RULE regressions pinned (SYNC_LOCK_ID
back-compat, phantom per-source lock, embed-backfill kill+resume,
webhook HMAC prefix-strip). All 9449 unit tests pass.
Caught at test-write time: the webhook handler had a Buffer.from('sha256=...',
'hex') truncation bug — without the prefix-strip, every signature would have
"matched" empty buffers. Pinned by a test/sources-webhook.test.ts IRON-RULE.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolve VERSION → 0.40.5.0 (master shipped v0.40.2.0 trajectory routing). Resolve package.json verify script — no-op (master's verify line unchanged). Resolve CHANGELOG (preserve all entries; my v0.40.5.0 stays on top). Resolve src/core/migrate.ts: master claimed v89 (facts_event_type_column); renumber my sources_github_repo_index v89 → v90. Update CHANGELOG + skills/migrations/v0.40.5.md migration version refs from v89 to v90. Refresh llms.txt + llms-full.txt against the merged tree. All wave tests still green (23/23 across db-lock-per-source + embed-backfill-submit + doctor-federation-health smoke).
The v0.40.5.0 wave added scripts/check-source-config-leak.sh with a too-broad pattern (JSON\.stringify\(.*config) that flagged any variable named 'config' — catching the GLOBAL gbrain config.json serializers in src/commands/init.ts (status envelopes) and src/core/config.ts (the config-file write site). On the CI runner without rg installed, the grep -rE fallback fired correctly and produced 4 false positives that broke the `verify` script. Tightened the patterns to specifically match `(source|src|row|s).config` property access — the actual risk shape (a sources-table row being serialized whole). The global gbrain config has a different shape and threat model (file-mode 0o600 at the write site), so it's safe to exempt at the regex level rather than per-file whitelist. Also fixed a latent bug: the rg branch used `--include='*.ts'` (grep's flag, not rg's). rg silently rejected it and CANDIDATES came back empty, so the local-dev runs (which have rg) would never have caught a real leak. Now branches on tool availability: `-g '*.ts'` for rg, `--include` for grep -rE. Both branches verified against a synthetic leak fixture. Also added init.ts + config.ts to the whitelist as a belt-and-suspenders since they handle gbrain-global config (not source rows) and could otherwise reflect-back via regex iteration. CI: `bun run verify` exit 0 locally with both the original false-positive fixture (clean repo) and a synthetic leak fixture (correctly caught, exit 1).
Master shipped v0.40.3.0 (contextual retrieval + cache invalidation gate) claiming migrations v90 (contextual_retrieval_columns) + v91 (pages_generation_trigger_and_bookmark) + a new doctor check (contextual_retrieval_coverage) + a new sources subcommand (set-cr-mode). Resolved conflicts: - VERSION: kept 0.40.5.0 (mine higher than master's 0.40.3.0) - package.json: kept 0.40.5.0 - CHANGELOG.md: preserved both entries (mine on top + master's v0.40.3.0) - src/core/migrate.ts: renumbered sources_github_repo_index from v90 → v92 (v90 + v91 now taken by master's contextual retrieval work) - src/commands/doctor.ts: kept BOTH check pushes — contextual_retrieval_coverage (master, #11) + federation_health (mine, #12) - src/commands/sources.ts: kept BOTH subcommands — status/webhook/tracked-branch (mine) + set-cr-mode (master) Bumped migration version refs from v90 → v92 in CHANGELOG, migration walkthrough (skills/migrations/v0.40.5.md), and pglite-schema.ts comment. Refreshed llms.txt + llms-full.txt against the merged tree. Verification: - bun run typecheck: clean - bun run verify: clean (all 19 checks pass including the leak-guard fix from the previous push) - Wave smoke tests (db-lock-per-source + embed-backfill-submit + doctor-federation-health): 23/23 pass; migration v92 lands as expected
Master shipped v0.40.4.0 (selective graph signals + per-stage attribution + audit-writer unification). No migration collision this time — my v92 remains the highest. Resolved conflicts: - VERSION: kept 0.40.5.0 (still higher than master's 0.40.4.0) - package.json: kept 0.40.5.0 - CHANGELOG.md: preserved both entries (mine on top + master's v0.40.4.0) - src/commands/doctor.ts: auto-merged cleanly Refreshed llms.txt + llms-full.txt against the merged tree. Verification: - bun run typecheck: clean - bun run verify: clean (all 19 checks pass) - Wave smoke tests (db-lock-per-source + embed-backfill-submit + doctor-federation-health): 23/23 pass; migration v92 lands cleanly
mgunnin
added a commit
to mgunnin/gbrain
that referenced
this pull request
May 28, 2026
* upstream/master: (22 commits) v0.41.4.0 wave: local providers + cross-platform stdin + gateway-routed dream judge (6 community PRs) (garrytan#1377) v0.41.3.0 fix(security/mcp): OAuth CORS lockdown + pre-register without DCR + validator surface (garrytan#1403) v0.41.2.0 feat: lens packs + epistemology unification — atoms + concepts as first-class units, calibration profile widening, gstack-learnings bridge (garrytan#1364) v0.41.1.0 feat: eval-loop wave — gbrain bench publish + gbrain eval gate close the LOOP (garrytan#1352) v0.41.0.0 feat(minions): fleet you supervise (4 field bugs + cathedral) (garrytan#1367) v0.40.10.0 feat: content sanity defense — junk-pattern throw + oversize-skip-embed (garrytan#1351) v0.40.9.0 feat(chunker): .sql indexing via tree-sitter + code-def on SQL DDL (garrytan#1173) (garrytan#1350) v0.40.8.1 docs: README rewrite + personal-brain + company-brain tutorials (garrytan#1345) v0.40.8.0 test: e2e + unit gap coverage + master flake root-cause fixes (garrytan#1313) v0.40.6.1 docs(todos): file v0.41 wave commitments + 7 verified-missing items (garrytan#1333) v0.40.7.0 Schema Cathedral v3 — agent-on-ramp + production rebuild of PR garrytan#1321 (garrytan#1327) v0.40.6.0 feat(sync): parallel sync --all + per-source lock invariant + sources status dashboard (productionized from PR garrytan#1314) (garrytan#1324) v0.40.5.0 Federated Sync v2 — parallel source sync + push triggers + per-source health (garrytan#1322) v0.40.4.0 feat(search): selective graph signals + per-stage attribution + audit-writer unification (garrytan#1300) v0.40.3.0 feat: contextual retrieval + cache invalidation gate + 4 deferred-item closures (garrytan#1323) v0.40.2.0 feat: trajectory routing for temporal + knowledge_update (gbrain think + LongMemEval) (garrytan#1296) v0.40.1.0 Track D — eval infrastructure (catch retrieval regressions, prove answer-quality wins) (garrytan#1298) v0.40.0.0 feat: agent-voice (Mars + Venus) + copy-into-host-repo skillpack paradigm (garrytan#1128) v0.39.3.0: productionize the v0.38 ingestion cathedral (smoke-test fix wave from PR garrytan#1299) (garrytan#1308) v0.39.2.0 feat(autopilot): per-source fan-out + cycle lock primitive + phase taxonomy (garrytan#1295) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Six-component cathedral implementing the Federated Sync v2 spec in a single wave. Behind
sync.federated_v2feature flag (default on; flip false + restart autopilot to revert to v0.40.x sequential behavior). Per-source lock and migration v89 stay on unconditionally as correctness fixes.Performance
gbrain sync --allviapMapAllSettledfan-out with--max-sources Ncap; 4-source brain measured ~4x faster than sequentialsyncLockId(sourceId)) so cross-source syncs no longer serialize on a global writer locksources status+federation_healthdoctor check share a batched GROUP BY pipeline (4 queries instead of 6×N per-source roundtrips)Decoupled embedding
embed-backfillminion handler — per-source DB lock at handler entry (D2), $10/job BudgetTracker cap (D6), fire-and-forget child submission (D15.1, codex catch —parent_job_idwould deadlock the parent), source-level 10min cooldown + 24h $25 rolling spend cap (D19)synchandler withauto_embed_backfill: truedefault (D22, supersedes the "new source-sync handler" plan after codex's "simpler" path)embedStaleForSourceextracted tosrc/core/embed-stale.ts(D15.2 — codex catch: no publicembedBatchprimitive existed)sync --allauto-enqueues embed-backfill per source on completion (D18)Push-triggered sync
gbrain sync trigger --source <id> [--priority high|normal|low]CLIPOST /webhooks/githubendpoint — HMAC-verified (60 req/min/IP rate limit, pre-DB short-circuit on missing signature), filters onX-GitHub-Event=push+ ref matchingtracked_branch(D3, D5)gbrain sources webhook set/show/rotate/clear <id>lifecycle commands with one-time-reveal posture (D8)gbrain sources tracked-branch <id> [--set <branch>] [--detect]for ref-filter config (D20)Autopilot
resolvePrioritywarn-once on invalidconfig.priorityvalues (D9)dispatchPerSourcefanout (different job names + idempotency keys; per-source lock serializes any overlap)Correctness fixes (unconditional, not behind feature flag)
sync.ts:959facts backstop now threadssourceIdtoengine.getPage— fixes pre-existing source-attribution bug on slug collisions in federated brainsredactSourceConfig+scripts/check-source-config-leak.shCI guard prevent webhook_secret leak through any sources.config serializersafeHexEqualextracted fromserve-http.tsclosure tosrc/core/timing-safe.ts(shared between admin login + webhook HMAC verify)Infrastructure
sources_github_repo_index): partial expression index onsources.config->>'github_repo'for fast webhook source-lookup (both engines + bootstrap probes)sync.federated_v2feature flag (D23) for clean rollbackWhat we caught at test-write time: the webhook handler had a
Buffer.from('sha256=<hex>', 'hex')truncation bug —Buffer.fromsilently truncates at non-hex chars, so comparing prefixed sha256= strings to each other would always compare two empty buffers and return true for every signature. Fix: strip thesha256=prefix before the constant-time compare. Pinned bytest/sources-webhook.test.tsIRON-RULE.Test Coverage
14 new test files (10 unit + 4 E2E-style PGLite hermetic) covering the 6 components. 112 new cases, all passing. 4 IRON-RULE regressions pinned:
SYNC_LOCK_ID === syncLockId('default')back-compat aliassyncLockId(sourceId)not bare constantembedding IS NULLpredicatesha256=prefix before constant-time compareTests: full unit suite passes (9449 pass, 0 fail across 8 shards + serial pass).
Pre-Landing Review
Reviewed via
/plan-eng-review(12 in-skill findings, all resolved) and codex outside-voice (14 findings, 13 substantive — all absorbed via D15-D23 plan rewrite). Cross-model agreement on the load-bearing fixes (per-source phantom lock, drop newsource-synchandler in favor of extendingsync, source-level submission gating with cooldown + rolling spend cap).Plan Completion
All 18 implementation tasks completed. Plan file:
~/.claude/plans/system-instruction-you-are-working-wise-hellman.md. Verdict in plan tail: ENG CLEARED.Documentation
CLAUDE.md, README, and TODOS.md updates deferred to a separate
/document-releaserun after this PR lands. The migration walkthrough atskills/migrations/v0.40.5.mdcovers the user-facing setup steps.Test plan
bun run typecheckcleanbun testall shards 0 fail (9449 cases pass)scripts/check-source-config-leak.shpasses (leak guard)🤖 Generated with Claude Code