Summary
A source's last_commit bookmark can point at a commit that no longer exists in the source repo (orphaned by a history rewrite / force-push / branch cleanup). When this happens, gbrain sync silently degrades to a full repo walk on every run, forever, never advancing last_sync_at. On a large brain with a cross-region DB this never completes (see #1958), so the source goes permanently stale with zero error surfaced.
Repro
- Source
default has sources.last_commit = <X>.
- The source repo's history is rewritten so
<X> is no longer reachable (e.g. a master→main consolidation that drops <X>; force-push; squash; branch delete).
- Run
gbrain sync --source default.
Observed: sync reports Large sync (NNN,NNN files) and walks the entire tree every run. git diff <X> HEAD fails internally because <X> is not a valid object. last_commit / last_sync_at never advance. No error, no warning — just eternal "in progress" that the cron timeout keeps killing.
Expected: sync detects the unreachable bookmark and either (a) reseeds to the nearest reachable ancestor by date and diffs forward, or (b) does a one-time full reconcile and then stamps the bookmark to HEAD — and in both cases emits a clear [sync] bookmark <X> unreachable (history rewritten) — re-pinning log line.
Root cause
The incremental path resolves lastCommit = readSyncAnchor(...) and then git diff lastCommit..HEAD. There is handling for a checkpoint target that's no longer reachable (merge-base --is-ancestor → discard + re-pin to HEAD), but no equivalent guard for the persisted last_commit bookmark itself. An unreachable bookmark falls through to the full-walk fallback every time instead of being detected and re-pinned.
Suggested fix
In the incremental sync entrypoint, before diffing:
if (lastCommit && !isAncestor(repoPath, lastCommit, headCommit)) {
slog(`[sync] last_commit ${lastCommit.slice(0,8)} unreachable (history rewritten) — re-pinning to HEAD and reconciling via content_hash`);
lastCommit = null; // forces a content-hash reconcile, not an eternal blind re-walk
// (content_hash already short-circuits unchanged files at ~10ms each)
}
Mirror the existing checkpoint-target reachability check (merge-base --is-ancestor) for the bookmark. Optionally also reseed to the nearest reachable commit ≤ last_sync_at to minimize the reconcile set.
Real-world impact (this brain)
Related
Summary
A source's
last_commitbookmark can point at a commit that no longer exists in the source repo (orphaned by a history rewrite / force-push / branch cleanup). When this happens,gbrain syncsilently degrades to a full repo walk on every run, forever, never advancinglast_sync_at. On a large brain with a cross-region DB this never completes (see #1958), so the source goes permanently stale with zero error surfaced.Repro
defaulthassources.last_commit = <X>.<X>is no longer reachable (e.g. amaster→mainconsolidation that drops<X>; force-push; squash; branch delete).gbrain sync --source default.Observed: sync reports
Large sync (NNN,NNN files)and walks the entire tree every run.git diff <X> HEADfails internally because<X>is not a valid object.last_commit/last_sync_atnever advance. No error, no warning — just eternal "in progress" that the crontimeoutkeeps killing.Expected: sync detects the unreachable bookmark and either (a) reseeds to the nearest reachable ancestor by date and diffs forward, or (b) does a one-time full reconcile and then stamps the bookmark to HEAD — and in both cases emits a clear
[sync] bookmark <X> unreachable (history rewritten) — re-pinninglog line.Root cause
The incremental path resolves
lastCommit = readSyncAnchor(...)and thengit diff lastCommit..HEAD. There is handling for a checkpoint target that's no longer reachable (merge-base --is-ancestor→ discard + re-pin to HEAD), but no equivalent guard for the persistedlast_commitbookmark itself. An unreachable bookmark falls through to the full-walk fallback every time instead of being detected and re-pinned.Suggested fix
In the incremental sync entrypoint, before diffing:
Mirror the existing checkpoint-target reachability check (
merge-base --is-ancestor) for the bookmark. Optionally also reseed to the nearest reachable commit ≤last_sync_atto minimize the reconcile set.Real-world impact (this brain)
last_commitpointed atcd8732419775d93f982248820cd42b8c46892df6, orphaned by amaster→maincleanup on 2026-05-31.last_sync_atfroze at 2026-06-04 03:39.last_committo a reachable commit at the same date → sync diffed only ~1,510 changed files and completed, stampinglast_sync_at.Related
title(content-sanity.ts:379) — silent indexing outage #1939 — sibling silent-staleness mode (poison file freezes the bookmark).