Skip to content

v0.42.34.0 feat(search): typed-edge relational retrieval — relationship questions get relationship answers#1959

Merged
garrytan merged 11 commits into
masterfrom
garrytan/native-gbrain-implementation
Jun 8, 2026
Merged

v0.42.34.0 feat(search): typed-edge relational retrieval — relationship questions get relationship answers#1959
garrytan merged 11 commits into
masterfrom
garrytan/native-gbrain-implementation

Conversation

@garrytan

@garrytan garrytan commented Jun 8, 2026

Copy link
Copy Markdown
Owner

Typed-edge relational retrieval

When a question's answer is a relationship (an edge between entities) rather than a passage — "who invested in widget-co", "who introduced me to alice-example", "what connects fund-a and fund-b" — gbrain now resolves the named entity and walks its typed-edge graph (invested_in, works_at, founded, attended, advises, …) to surface the answer, even when no single page mentions both sides.

Previously the graph was only a re-ranker (graph-signals.ts boosted hubs among already-retrieved candidates). A relationship that lived purely in the edges — an investor whose page never names the company — was invisible to retrieval. It now enters as a fourth RRF arm alongside keyword + vector.

Proof

On a benchmark whose answers are lexically unrecoverable (bodies never name the related entity; only the typed edge connects them), graph-relationship recall@10 goes from <25% → >75% with the arm on. A non-relational query returns byte-identical results arm-on vs arm-off (the no-op / no-regression gate). gbrain eval retrieval-quality --ab-relational reproduces it.

How it works

  • Deterministic: parses the original query (never an LLM-expanded variant); traversal is a recursive CTE with a fixed tiebreak.
  • Safe: within-source only (never crosses a mounted-brain boundary), mentions edges excluded by default, depth + per-node fan-out bounded, deleted_at filtered everywhere, confidence-gated seed resolution (never traverses from a guess), fail-open with an audit row.
  • Default on for balanced/tokenmax, off for conservative; pure no-op for non-relational queries and edge-less brains.
  • Schema-pack-extensible relation vocabulary (subset-validated against the link types ingest produces).

Also fixed (engine-wide, structural)

  • RRF/dedup key now carries source_id — same-slug pages across mounted brains no longer collapse into one result.
  • Query cache scoped per canonical source-set — a federated search can't be served a single-source cached row.

Commits (bisect-friendly)

  1. deterministic relational-query parser
  2. relationalFanout typed-edge fan-out (both engines + parity pin)
  3. relational recall arm + federation key hardening
  4. relational benchmark + recall@k harness metrics
  5. relational A/B proof + arm fires on all retrieval paths
  6. version + CHANGELOG (this)

Tests

New: relational-intent, relational-fanout (PGLite), relational-recall, relational-ab (A/B + no-op + drift), rrf-source-key, cache-scope-key, plus a DATABASE_URL-gated relationalFanout parity block. Full unit suite green except pre-existing environment-only failures (provider-key tests that assume no API key; this shell has ZEROENTROPY_API_KEY set — those files are untouched by this PR and pass in clean CI).

Cache note

KNOBS_HASH_VERSION 9→10 (a relational-on result must not serve a relational-off lookup). One-time cold-miss on first query after upgrade; self-heals.

Deferred (TODOs)

Interactive seed disambiguation; cross-source edge traversal (security-reviewed follow-up); a first-class whole-brain entity-graph index (only if CTE latency proves insufficient on dense graphs); tier-2 resolution-margin gate.

Review

Plan went through CEO review + two engineering-review passes + a Codex outside-voice pass (two BLOCKING design issues — RRF source-key collapse and federation scope — caught and resolved before implementation). The eval also caught a real integration bug: the arm initially only fired on the main RRF path, so it was dead whenever no embedding provider was configured; it now fuses on all three retrieval paths.

🤖 Generated with Claude Code

garrytan and others added 11 commits June 7, 2026 18:08
Pure, ReDoS-bounded parser that detects relationship queries ("who invested
in X", "who at X works on Y", "who introduced me to X", "what connects A and
B") and maps them to typed edges. Schema-pack-extensible vocab with subset
validation against the link types ingest produces, so query-side and
ingest-side relation vocabularies can't drift. No-match / pronoun-seed /
adjacency guards keep it precision-first (a candidate only; the arm fires
only when a real seed also resolves).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generalizes traversePaths to a SEED ARRAY, aggregating to ranked NODES
(shortest hop, edge-richness count, via-link-types, shortest connecting
path, canonical chunk id) instead of edges. Within-source traversal (never
crosses a source boundary even across a federated scope), link_source=
'mentions' excluded by default, deleted_at filtered at seed + every neighbor
+ every node, bounded depth (<=3) + candidate cap. Adds RelationalFanoutRow
/ RelationalFanoutOpts + the relational SearchResult/SearchOpts fields to
types.

Lands in lockstep in postgres + pglite engines, pinned by a DATABASE_URL-
gated parity block in engine-parity.test.ts; a PGLite unit test exercises
the SQL (typed-edge filter, mentions exclusion, deleted_at, canonical chunk,
multi-seed connects, determinism) in default CI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires edge-derived candidates into bare hybridSearch as a FOURTH RRF arm
(relational-recall.ts): parse the original query -> scope-aware,
confidence-gated seed resolution (drops fallback_slugify; never traverses
from a guess) -> relationalFanout -> batch-hydrate, reinforcing each page's
REAL canonical chunk (page-level key for chunkless entity pages) ->
--explain attribution + fail-open audit row. Text-only (no-op in image
mode); pure no-op for non-relational queries; rides every downstream stage
(cosine, post-fusion boosts, dedup, reranker, autocut, token budget).

Mode wiring: relationalRetrieval + relational_retrieval_depth knobs
(conservative off; balanced/tokenmax on; depth 2), per-call thread-through
in both bare + cached paths, KNOBS_HASH_VERSION 9->10 (rel=/reld=), config
keys, modes-dashboard descriptions, and a `relational` param on the query op.

Federation hardening (structural, engine-wide): the RRF/dedup key now
carries source_id via a shared rrfKey() (fixes a latent cross-source
collapse where same-slug pages in different sources merged), and the query
cache scopes by a canonical source-set key (cacheScopeKey) so a federated
read can't be served a single-source row.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NamedThingBench harness gains recall@k / recall@10 (the relational headline
metric) on QuestionResult + FamilyReport, plus typed seed/linkTypes/kind on
NamedThingQuestion so the graph-relationship family is machine-checkable.

Adds the relational benchmark corpus (test/fixtures/retrieval-quality/
relational/): a small entity graph whose answers are LEXICALLY UNRECOVERABLE
— every page body is generic and never names the entity it relates to, so
only the typed edge connects query to answer. corpus.ts is the canonical
source for both the seed loader and the 38-question gold set; relational.jsonl
is generated from it (a drift test pins them equal).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the integration bug the eval caught: the relational arm was only
injected on the main RRF path, so it silently did nothing whenever vector
was unavailable — no embedding provider configured (the default in many
deployments) OR embed failure. The arm is now built once and fused via RRF
on ALL THREE hybridSearch return paths (no-embedding-provider, embed-failed
keyword fallback, main path). Without this it would have been dead in
exactly the setups that most need it.

Adds `gbrain eval retrieval-quality --ab-relational`: runs the gold set
twice (arm off vs on) in a fixed mode and reports the graph-relationship
recall@10 lift + Hit@3 + latency add. The CI A/B test pins the headline
result — recall@10 jumps from <25% (lexically unrecoverable) to >75% with
the arm on — and a non-relational query returns identical results arm-on vs
off (the no-op / no-regression gate).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Relational retrieval feature: typed-edge recall arm + federation key
hardening. Also updates the KNOBS_HASH_VERSION 9→10 assertions across the
remaining search test files (the bump invalidates relational-off cache rows).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLAUDE.md Search Mode: add relationalRetrieval to the knob table, the
knobs_hash v=9→10 note, and a relational-retrieval summary. RETRIEVAL.md:
add the relational recall arm to the pipeline diagram. Regenerate llms
bundles (build:llms).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ain-implementation

# Conflicts:
#	CHANGELOG.md
#	VERSION
#	package.json
…ain-implementation

# Conflicts:
#	CHANGELOG.md
#	VERSION
#	package.json
CI shard runs with the ZeroEntropy gateway default (1280-d), but the
relational test fixtures hardcoded 1536-d embeddings, so chunk inserts were
rejected with "expected 1280 dimensions, not 1536" (CheckExpectedDim) — the
`test (6)` shard + `test-status` failures. The column width tracks the
configured gateway default and can shift with shard order, so fixtures now
probe `content_chunks.embedding`'s actual `atttypmod` after initSchema and
size embeddings to it (the pglite-engine.test.ts pattern), via a shared
`probeEmbeddingDim` helper. Verified passing at a forced 1280-d column.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@garrytan garrytan merged commit 099d9a8 into master Jun 8, 2026
21 checks passed
mgunnin added a commit to mgunnin/gbrain that referenced this pull request Jun 9, 2026
* upstream/master:
  v0.42.37.0 fix(security,ingest): source-isolation grant enforcement + non-string frontmatter guard + papercuts (garrytan#1999)
  v0.42.36.0 fix(sync): resumable, durable, single-flight sync — converges under pool exhaustion + repeated kills (garrytan#1794) (garrytan#1980)
  v0.42.35.0 fix(sync): recover from unreachable last_commit instead of full-walking forever (garrytan#1970) (garrytan#1975)
  v0.42.34.0 feat(search): typed-edge relational retrieval — relationship questions get relationship answers (garrytan#1959)
  docs(designs): add COMMUNITY_IDEAS ledger from open-PR backlog triage (garrytan#1969)
  v0.42.33.0 fix(sources): confine sync re-clone to gbrain-owned clones; never delete a user working tree (garrytan#1881) (garrytan#1960)

# Conflicts:
#	src/core/operations.ts
DomiYoung pushed a commit to DomiYoung/gbrain that referenced this pull request Jun 10, 2026
…ytan#1972)

pool.end() against PgBouncer transaction-mode never drained, so disconnect
blocked until the CLI's 10s force-exit fired and process.exit()'d mid-write,
truncating stdout (e.g. garrytan#1959's relational query returned empty). Add a
gbrain-owned endPoolBounded(pool): Promise.race of pool.end({timeout}) against
a hard timer, so teardown is bounded regardless of what postgres.js does and is
testable. connection-manager ends its direct + read pools concurrently so the
per-pool bounds don't stack. PGLite disconnect is unaffected.
garrytan added a commit that referenced this pull request Jun 10, 2026
…operative-abort (#1972) (#2015)

* fix(jobs): reap stale dead-holder cycle/sync locks (#1972)

A crashed sync (OOM, recycle, SIGKILL) stranded its gbrain_cycle_locks row
until something contended for it — reclaim was on-contention only. Add a
host-scoped background reaper: reapDeadHolderLocks deletes locks whose holder
PID is provably dead on this host, scoped to the gbrain-sync:*/gbrain-cycle*
namespaces only (never elections/supervisor/reindex), with a snapshot-matched
delete (date_trunc on acquired_at) that is TOCTOU-safe against PID reuse.
Reuses isHolderDeadLocally (same-host + ESRCH + 60s grace). doctor --fix now
auto-reaps for no-autopilot brains. DRY: selectLockRows + shared mapper now
back inspectLock + listStaleLocks (killed the triplication).

* fix(db): bound pool disconnect so teardown can't eat CLI output (#1972)

pool.end() against PgBouncer transaction-mode never drained, so disconnect
blocked until the CLI's 10s force-exit fired and process.exit()'d mid-write,
truncating stdout (e.g. #1959's relational query returned empty). Add a
gbrain-owned endPoolBounded(pool): Promise.race of pool.end({timeout}) against
a hard timer, so teardown is bounded regardless of what postgres.js does and is
testable. connection-manager ends its direct + read pools concurrently so the
per-pool bounds don't stack. PGLite disconnect is unaffected.

* fix(cycle): complete cooperative-abort coverage + wire lock reaper (#1972)

v0.42.29 made only the embed phase honor the abort signal; a 24h pull still
showed force-evicts from a long non-embed phase ignoring it. Thread the signal
into every cycle-reachable long loop: extract (extractForSlugs + the full-walk
extractLinksFromDir/extractTimelineFromDir), extract_facts (per-page loop +
embed signal + the phantom-redirect 30s lock-retry), and consolidate's bucket
loop. Add a terminal abort check so an aborted cycle never stamps
last_full_cycle_at as a completed run (Codex #9). lint now yields + checks
abort every 200 pages (it's synchronous; the yield is what lets the signal
land). New phase-duration force-evict attribution log names any phase that
crosses the 30s deadline. Wire reapDeadHolderLocks at cycle start.

* chore: bump version and changelog (v0.42.38.0)

#1972 — stale-lock reaper, bounded pool disconnect, and complete
cooperative-abort coverage across cycle phases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(key-files): current-state for the #1972 reaper, bounded disconnect, abort coverage

document-release: update db-lock.ts (reapDeadHolderLocks + selectLockRows DRY),
db.ts (endPoolBounded), and abort-check.ts (coverage now spans extract/
extract_facts/consolidate/lint + terminal guard) entries to current truth.

* test(isolation): fix shard-order flakes exposed by #1972's new test files

Adding 3 new test files reshuffled the hash-based shards, exposing two
pre-existing test-isolation bugs:

- cycle-consolidate.test.ts assumed the global legacy-embedding preload's
  1536-d gateway config still held at initSchema, but a co-sharded test that
  calls resetGateway() in teardown nulls it, so initSchema fell back to the
  1280-d default and built a halfvec(1280) facts column its 1536-d fixtures
  can't fill. Re-pin the legacy OpenAI/1536 config in beforeAll (the pattern
  legacy-embedding-preload.ts documents for 1536-d fixture tests).
- db-lock-heartbeat-takeover.test.ts (merged from master's #1794) mutated
  process.env.GBRAIN_LOCK_STEAL_GRACE_SECONDS raw, tripping check:test-isolation
  rule R1. Convert to withEnv().

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant