v0.42.34.0 feat(search): typed-edge relational retrieval — relationship questions get relationship answers#1959
Merged
Conversation
Pure, ReDoS-bounded parser that detects relationship queries ("who invested
in X", "who at X works on Y", "who introduced me to X", "what connects A and
B") and maps them to typed edges. Schema-pack-extensible vocab with subset
validation against the link types ingest produces, so query-side and
ingest-side relation vocabularies can't drift. No-match / pronoun-seed /
adjacency guards keep it precision-first (a candidate only; the arm fires
only when a real seed also resolves).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generalizes traversePaths to a SEED ARRAY, aggregating to ranked NODES (shortest hop, edge-richness count, via-link-types, shortest connecting path, canonical chunk id) instead of edges. Within-source traversal (never crosses a source boundary even across a federated scope), link_source= 'mentions' excluded by default, deleted_at filtered at seed + every neighbor + every node, bounded depth (<=3) + candidate cap. Adds RelationalFanoutRow / RelationalFanoutOpts + the relational SearchResult/SearchOpts fields to types. Lands in lockstep in postgres + pglite engines, pinned by a DATABASE_URL- gated parity block in engine-parity.test.ts; a PGLite unit test exercises the SQL (typed-edge filter, mentions exclusion, deleted_at, canonical chunk, multi-seed connects, determinism) in default CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires edge-derived candidates into bare hybridSearch as a FOURTH RRF arm (relational-recall.ts): parse the original query -> scope-aware, confidence-gated seed resolution (drops fallback_slugify; never traverses from a guess) -> relationalFanout -> batch-hydrate, reinforcing each page's REAL canonical chunk (page-level key for chunkless entity pages) -> --explain attribution + fail-open audit row. Text-only (no-op in image mode); pure no-op for non-relational queries; rides every downstream stage (cosine, post-fusion boosts, dedup, reranker, autocut, token budget). Mode wiring: relationalRetrieval + relational_retrieval_depth knobs (conservative off; balanced/tokenmax on; depth 2), per-call thread-through in both bare + cached paths, KNOBS_HASH_VERSION 9->10 (rel=/reld=), config keys, modes-dashboard descriptions, and a `relational` param on the query op. Federation hardening (structural, engine-wide): the RRF/dedup key now carries source_id via a shared rrfKey() (fixes a latent cross-source collapse where same-slug pages in different sources merged), and the query cache scopes by a canonical source-set key (cacheScopeKey) so a federated read can't be served a single-source row. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NamedThingBench harness gains recall@k / recall@10 (the relational headline metric) on QuestionResult + FamilyReport, plus typed seed/linkTypes/kind on NamedThingQuestion so the graph-relationship family is machine-checkable. Adds the relational benchmark corpus (test/fixtures/retrieval-quality/ relational/): a small entity graph whose answers are LEXICALLY UNRECOVERABLE — every page body is generic and never names the entity it relates to, so only the typed edge connects query to answer. corpus.ts is the canonical source for both the seed loader and the 38-question gold set; relational.jsonl is generated from it (a drift test pins them equal). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the integration bug the eval caught: the relational arm was only injected on the main RRF path, so it silently did nothing whenever vector was unavailable — no embedding provider configured (the default in many deployments) OR embed failure. The arm is now built once and fused via RRF on ALL THREE hybridSearch return paths (no-embedding-provider, embed-failed keyword fallback, main path). Without this it would have been dead in exactly the setups that most need it. Adds `gbrain eval retrieval-quality --ab-relational`: runs the gold set twice (arm off vs on) in a fixed mode and reports the graph-relationship recall@10 lift + Hit@3 + latency add. The CI A/B test pins the headline result — recall@10 jumps from <25% (lexically unrecoverable) to >75% with the arm on — and a non-relational query returns identical results arm-on vs off (the no-op / no-regression gate). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ain-implementation
Relational retrieval feature: typed-edge recall arm + federation key hardening. Also updates the KNOBS_HASH_VERSION 9→10 assertions across the remaining search test files (the bump invalidates relational-off cache rows). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLAUDE.md Search Mode: add relationalRetrieval to the knob table, the knobs_hash v=9→10 note, and a relational-retrieval summary. RETRIEVAL.md: add the relational recall arm to the pipeline diagram. Regenerate llms bundles (build:llms). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ain-implementation # Conflicts: # CHANGELOG.md # VERSION # package.json
…ain-implementation # Conflicts: # CHANGELOG.md # VERSION # package.json
CI shard runs with the ZeroEntropy gateway default (1280-d), but the relational test fixtures hardcoded 1536-d embeddings, so chunk inserts were rejected with "expected 1280 dimensions, not 1536" (CheckExpectedDim) — the `test (6)` shard + `test-status` failures. The column width tracks the configured gateway default and can shift with shard order, so fixtures now probe `content_chunks.embedding`'s actual `atttypmod` after initSchema and size embeddings to it (the pglite-engine.test.ts pattern), via a shared `probeEmbeddingDim` helper. Verified passing at a forced 1280-d column. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mgunnin
added a commit
to mgunnin/gbrain
that referenced
this pull request
Jun 9, 2026
* upstream/master: v0.42.37.0 fix(security,ingest): source-isolation grant enforcement + non-string frontmatter guard + papercuts (garrytan#1999) v0.42.36.0 fix(sync): resumable, durable, single-flight sync — converges under pool exhaustion + repeated kills (garrytan#1794) (garrytan#1980) v0.42.35.0 fix(sync): recover from unreachable last_commit instead of full-walking forever (garrytan#1970) (garrytan#1975) v0.42.34.0 feat(search): typed-edge relational retrieval — relationship questions get relationship answers (garrytan#1959) docs(designs): add COMMUNITY_IDEAS ledger from open-PR backlog triage (garrytan#1969) v0.42.33.0 fix(sources): confine sync re-clone to gbrain-owned clones; never delete a user working tree (garrytan#1881) (garrytan#1960) # Conflicts: # src/core/operations.ts
DomiYoung
pushed a commit
to DomiYoung/gbrain
that referenced
this pull request
Jun 10, 2026
…ytan#1972) pool.end() against PgBouncer transaction-mode never drained, so disconnect blocked until the CLI's 10s force-exit fired and process.exit()'d mid-write, truncating stdout (e.g. garrytan#1959's relational query returned empty). Add a gbrain-owned endPoolBounded(pool): Promise.race of pool.end({timeout}) against a hard timer, so teardown is bounded regardless of what postgres.js does and is testable. connection-manager ends its direct + read pools concurrently so the per-pool bounds don't stack. PGLite disconnect is unaffected.
garrytan
added a commit
that referenced
this pull request
Jun 10, 2026
…operative-abort (#1972) (#2015) * fix(jobs): reap stale dead-holder cycle/sync locks (#1972) A crashed sync (OOM, recycle, SIGKILL) stranded its gbrain_cycle_locks row until something contended for it — reclaim was on-contention only. Add a host-scoped background reaper: reapDeadHolderLocks deletes locks whose holder PID is provably dead on this host, scoped to the gbrain-sync:*/gbrain-cycle* namespaces only (never elections/supervisor/reindex), with a snapshot-matched delete (date_trunc on acquired_at) that is TOCTOU-safe against PID reuse. Reuses isHolderDeadLocally (same-host + ESRCH + 60s grace). doctor --fix now auto-reaps for no-autopilot brains. DRY: selectLockRows + shared mapper now back inspectLock + listStaleLocks (killed the triplication). * fix(db): bound pool disconnect so teardown can't eat CLI output (#1972) pool.end() against PgBouncer transaction-mode never drained, so disconnect blocked until the CLI's 10s force-exit fired and process.exit()'d mid-write, truncating stdout (e.g. #1959's relational query returned empty). Add a gbrain-owned endPoolBounded(pool): Promise.race of pool.end({timeout}) against a hard timer, so teardown is bounded regardless of what postgres.js does and is testable. connection-manager ends its direct + read pools concurrently so the per-pool bounds don't stack. PGLite disconnect is unaffected. * fix(cycle): complete cooperative-abort coverage + wire lock reaper (#1972) v0.42.29 made only the embed phase honor the abort signal; a 24h pull still showed force-evicts from a long non-embed phase ignoring it. Thread the signal into every cycle-reachable long loop: extract (extractForSlugs + the full-walk extractLinksFromDir/extractTimelineFromDir), extract_facts (per-page loop + embed signal + the phantom-redirect 30s lock-retry), and consolidate's bucket loop. Add a terminal abort check so an aborted cycle never stamps last_full_cycle_at as a completed run (Codex #9). lint now yields + checks abort every 200 pages (it's synchronous; the yield is what lets the signal land). New phase-duration force-evict attribution log names any phase that crosses the 30s deadline. Wire reapDeadHolderLocks at cycle start. * chore: bump version and changelog (v0.42.38.0) #1972 — stale-lock reaper, bounded pool disconnect, and complete cooperative-abort coverage across cycle phases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): current-state for the #1972 reaper, bounded disconnect, abort coverage document-release: update db-lock.ts (reapDeadHolderLocks + selectLockRows DRY), db.ts (endPoolBounded), and abort-check.ts (coverage now spans extract/ extract_facts/consolidate/lint + terminal guard) entries to current truth. * test(isolation): fix shard-order flakes exposed by #1972's new test files Adding 3 new test files reshuffled the hash-based shards, exposing two pre-existing test-isolation bugs: - cycle-consolidate.test.ts assumed the global legacy-embedding preload's 1536-d gateway config still held at initSchema, but a co-sharded test that calls resetGateway() in teardown nulls it, so initSchema fell back to the 1280-d default and built a halfvec(1280) facts column its 1536-d fixtures can't fill. Re-pin the legacy OpenAI/1536 config in beforeAll (the pattern legacy-embedding-preload.ts documents for 1536-d fixture tests). - db-lock-heartbeat-takeover.test.ts (merged from master's #1794) mutated process.env.GBRAIN_LOCK_STEAL_GRACE_SECONDS raw, tripping check:test-isolation rule R1. Convert to withEnv(). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Typed-edge relational retrieval
When a question's answer is a relationship (an edge between entities) rather than a passage — "who invested in widget-co", "who introduced me to alice-example", "what connects fund-a and fund-b" — gbrain now resolves the named entity and walks its typed-edge graph (
invested_in,works_at,founded,attended,advises, …) to surface the answer, even when no single page mentions both sides.Previously the graph was only a re-ranker (
graph-signals.tsboosted hubs among already-retrieved candidates). A relationship that lived purely in the edges — an investor whose page never names the company — was invisible to retrieval. It now enters as a fourth RRF arm alongside keyword + vector.Proof
On a benchmark whose answers are lexically unrecoverable (bodies never name the related entity; only the typed edge connects them),
graph-relationshiprecall@10 goes from <25% → >75% with the arm on. A non-relational query returns byte-identical results arm-on vs arm-off (the no-op / no-regression gate).gbrain eval retrieval-quality --ab-relationalreproduces it.How it works
mentionsedges excluded by default, depth + per-node fan-out bounded,deleted_atfiltered everywhere, confidence-gated seed resolution (never traverses from a guess), fail-open with an audit row.Also fixed (engine-wide, structural)
source_id— same-slug pages across mounted brains no longer collapse into one result.Commits (bisect-friendly)
relationalFanouttyped-edge fan-out (both engines + parity pin)Tests
New:
relational-intent,relational-fanout(PGLite),relational-recall,relational-ab(A/B + no-op + drift),rrf-source-key,cache-scope-key, plus aDATABASE_URL-gatedrelationalFanoutparity block. Full unit suite green except pre-existing environment-only failures (provider-key tests that assume no API key; this shell hasZEROENTROPY_API_KEYset — those files are untouched by this PR and pass in clean CI).Cache note
KNOBS_HASH_VERSION9→10 (a relational-on result must not serve a relational-off lookup). One-time cold-miss on first query after upgrade; self-heals.Deferred (TODOs)
Interactive seed disambiguation; cross-source edge traversal (security-reviewed follow-up); a first-class whole-brain entity-graph index (only if CTE latency proves insufficient on dense graphs); tier-2 resolution-margin gate.
Review
Plan went through CEO review + two engineering-review passes + a Codex outside-voice pass (two BLOCKING design issues — RRF source-key collapse and federation scope — caught and resolved before implementation). The eval also caught a real integration bug: the arm initially only fired on the main RRF path, so it was dead whenever no embedding provider was configured; it now fuses on all three retrieval paths.
🤖 Generated with Claude Code