v0.29.0 feat: salience + anomaly detection — brain surfaces what's hot without being asked#592
Merged
garrytan merged 11 commits intogarrytan/v0.28-releasefrom May 8, 2026
Merged
Conversation
Migration v34 adds pages.emotional_weight REAL DEFAULT 0.0 (column-only, no index — salience query orders by computed score, not raw weight). Embedded DDL (schema.sql + pglite-schema.ts + schema-embedded.ts) mirrors the column so fresh installs don't need migration replay. types.ts gains: PageFilters.sort enum + PAGE_SORT_SQL whitelist (engines hardcoded ORDER BY updated_at DESC; threading lands in the next commit); SalienceOpts/SalienceResult, AnomaliesOpts/AnomalyResult, EmotionalWeightInputRow/EmotionalWeightWriteRow contracts. cycle/emotional-weight.ts: pure-function score in [0..1] from tags + takes (anglocentric default seed list; user-overridable via config key emotional_weight.high_tags). cycle/anomaly.ts: meanStddev + cohort threshold helpers with zero-stddev fallback (count > mean + 1) so rare cohorts don't produce NaN sigmas. Test coverage: migrate v34 structural assertions + 14-case formula unit + 13-case anomaly stats unit. Codex review fixes baked in: formula clamped to [0,1]; per-take weight clamped to [0,1] before averaging; zero-stddev fallback finite, never NaN. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BrainEngine adds 4 methods, both engines implement: - batchLoadEmotionalInputs(slugs?): CTE-shaped read with per-table pre-aggregates. A page with N tags + M takes never produces N×M rows (codex C4#4) — page_tags + page_takes CTEs aggregate independently, then LEFT JOIN to pages. - setEmotionalWeightBatch(rows): UPDATE FROM unnest($1::text[], $2::text[], $3::real[]) composite-keyed on (slug, source_id). Multi- source brains can't fan out (codex C4#3) — pages.slug is unique only within source_id. Same shape that v0.18 link batches use. - getRecentSalience: time boundary computed in JS, bound as TIMESTAMPTZ. SQL identical across engines (codex C5/D5 — avoids dialect drift on $1::interval binding which has zero current uses on PGLite). - findAnomalies: tag + type cohort baselines via generate_series- densified daily-count CTEs (codex C4#6). Sparse-day rare cohorts get correct (mean, stddev) instead of biased upward by zero-omission. Year cohort deferred to v0.30. listPages threads the new PageFilters.sort enum through both engines. Was hardcoded ORDER BY updated_at DESC; now PAGE_SORT_SQL whitelist maps the 4 enum values to literal SQL fragments — no injection surface. postgres.js uses sql.unsafe; PGLite splices the fragment directly. Regression tests (PGLite, no DATABASE_URL needed): - multi-source-emotional-weight: same slug under two source_ids, setEmotionalWeightBatch on one of them, asserts the other survives untouched. Direct codex C4#3 guard. - list-pages-regression (IRON RULE): old call shape (type, tag, limit) still returns updated_desc default; new sort=updated_asc reverses; sort=created_desc orders by created_at; sort=slug alphabetical; unsupported sort enum falls back to default (defense in depth). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a 9th cycle phase between extract and embed. Sees the union of syncPagesAffected + synthesizeWrittenSlugs for incremental mode (so synthesize-written pages get their weight computed too — codex C2 caught that the prior plan threaded only sync). Full mode (no incremental anchors) walks every page; users hit this path on first upgrade via gbrain dream --phase recompute_emotional_weight. Phase orchestrator (cycle/recompute-emotional-weight.ts) is two SQL round-trips total regardless of brain size: 1. batchLoadEmotionalInputs(slugs?) → per-page tag/take inputs. 2. computeEmotionalWeight in memory (pure function). 3. setEmotionalWeightBatch(rows) → composite-keyed UPDATE FROM unnest. Empty affectedSlugs short-circuits (no DB read, no write). Dry-run computes weights and reports the would-write count without touching the DB. Engine throw bubbles into status:fail with code RECOMPUTE_EMOTIONAL_WEIGHT_FAIL — cycle continues to the next phase. Plumbing: - CyclePhase type adds 'recompute_emotional_weight'. - ALL_PHASES + NEEDS_LOCK_PHASES include it. - CycleReport.totals adds pages_emotional_weight_recomputed (additive, schema_version stays "1"). - runCycle's totals rollup + status derivation honor the new field. - synthesize.ts emits writtenSlugs in details so cycle.ts can union with syncPagesAffected for incremental backfill. Tests: 7-case unit (fake-engine), 3-case PGLite e2e (full mode + dry- run + ALL_PHASES position), 1000-page perf budget (<5s on PGLite). Codex C2 → A: clean separation. Phase doesn't modify runExtractCore; runs on its own seam after the existing 8 phases plus synthesize. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new MCP operations + a transcripts library: - get_recent_salience: pages ranked by emotional + activity salience. Subagent-allow-listed. params: days (default 14), limit (default 20, capped 100), slugPrefix (renamed from `kind` per codex C4#10 to avoid collision with PageKind/TakeKind). - find_anomalies: cohort-level activity outliers (tag + type). Subagent-allow-listed. Year cohort deferred to v0.30. - get_recent_transcripts: raw .txt transcripts from the dream-cycle corpus dirs. LOCAL-ONLY: rejects ctx.remote === true with permission_denied (codex C3). NOT in the subagent allow-list — all subagent calls run with remote=true, would always reject (footgun if visible). Cycle's synthesize phase calls discoverTranscripts directly, so subagents that need transcripts go through the library function, not the op. Tool descriptions extracted to src/core/operations-descriptions.ts so they're pinnable in tests and stable for the Tier-2 LLM routing eval. Redirects on query/search/list_pages: personal/emotional questions should reach the new ops, not semantic search. Anti-flattery hint on query: "Do NOT assume words like crazy, notable, or big mean impressive — they often mean difficult or emotionally charged." list_pages gains updated_after (string ISO) and sort enum params, surfacing the engine threading from the prior commit. src/core/transcripts.ts: filesystem walk shared by the gated MCP op and the (commit 5) CLI command. Reuses discoverTranscripts corpus-dir resolution + isDreamOutput from cycle/transcript-discovery.ts. Trust gate lives in the op handler, not the library — the library is trusted by both the gated op and the local CLI. Allow-list: 11 → 13 (add salience + anomalies; transcripts excluded per codex C3, with a comment explaining why). Tests: 21-case description pin (catches accidental edits that change LLM-facing surface); 11-case transcripts unit covering trust gate, mtime window, dream-output skip, summary truncation, no corpus_dir; 2-case salience type-contract smoke (full Garry-test fixture in commit 6's e2e suite). Codex C1: routing-eval fixtures (skills/<x>/routing-eval.jsonl) deliberately NOT shipped — routing-eval.ts is substring-match on resolver triggers, not MCP tool routing. Real coverage lands as test/e2e/salience-llm-routing.test.ts in commit 6. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new CLI commands wired into src/cli.ts dispatch + CLI_ONLY set + help text: - gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json] - gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json] - gbrain transcripts recent [--days N] [--full] [--json] Each command file mirrors src/commands/orphans.ts shape: pure data fn + JSON formatter + human formatter. Calls into engine.getRecentSalience / findAnomalies (already shipped) and src/core/transcripts.ts. salience and anomalies show ranked rows with per-cohort mean/stddev/sigma. transcripts honors `--full` (caps at 100KB/file) vs default summary (first non-empty line + ~250 chars). All three emit JSON with --json for agent consumption. `--kind` is accepted as a slug-prefix shorthand on `gbrain salience` even though the underlying op param is `slugPrefix` (kept the CLI flag short; the MCP-facing param uses the more-explicit name to align with PageKind/TakeKind/slugPrefix vocabulary). CLI_ONLY set in src/cli.ts gains the three new command names so they don't get forwarded to MCP-only routing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PGLite e2e (no DATABASE_URL needed):
- salience-pglite: the Garry test. 7 wedding-tagged pages updated today
+ 100 background pages backdated across 30 days via raw SQL UPDATE
(codex C4#7 — engine.putPage stamps updated_at = now(), so seeding
via the engine alone can't reproduce historical recency windows).
Asserts wedding pages outrank random-tag noise in the 7-day window;
slugPrefix filter narrows correctly; days=0 boundary case; limit cap.
- anomalies-pglite: same fixture shape (7 wedding pages today, 100
background backdated). findAnomalies with sigma=3 returns the
wedding-tag cohort with sigma_observed > 3 vs near-zero baseline;
page_slugs sample carries the wedding pages; date with no activity
returns []; high sigma threshold suppresses borderline cohorts
(zero-stddev fallback stays finite — no NaN sigma).
Postgres-gated e2e:
- engine-parity-salience: PGLite ↔ Postgres parity for getRecentSalience
and findAnomalies. Same fixture into both engines; top-result and
cohort-set match. Closes the v0.22.0-style parity gap for the new
v0.29 SQL idioms (EXTRACT(EPOCH ...), generate_series, CTE chain).
Tier-2 LLM routing eval (ANTHROPIC_API_KEY-gated):
- salience-llm-routing: calls Claude with v0.29 tool descriptions and
12 personal-query phrasings ("anything crazy lately", "what's been
going on with me", etc.). Asserts the chosen tool is in the v0.29
set, not query() / search(). ~$0.10 per CI run on Haiku. Tests the
ACTUAL ship criterion — replaces the discarded fake-coverage
routing-eval.jsonl fixtures (codex C1 → B).
This is the only test that proves the description edits drive routing.
Without it, we'd ship description changes and only learn from
production behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
VERSION + package.json bump 0.28.0 → 0.29.0. CHANGELOG.md adds a v0.29.0 release-summary in the GStack/Garry voice plus the "To take advantage of v0.29.0" block. Headline two-liner: "The brain tells you what's hot without being asked. Salience + anomaly detection ship. Search rewards hypotheses; salience surfaces them." Numbers-that-matter table covers engine surface delta, MCP op delta, allow-list delta, cycle-phase delta, schema migration, list_pages param surface, and test count. Itemized changes section lists the schema migration + new cycle phase + new MCP ops + redirect descriptions + subagent allow-list rules + new tests + a contributor note clarifying that routing-eval.ts is not the right surface for testing MCP tool routing (use the Tier-2 LLM eval pattern instead). CLAUDE.md Key Files updated for the v0.29 surface: - src/core/engine.ts: notes the 4 new methods + PageFilters.sort threading. - src/core/migrate.ts: v34 (pages_emotional_weight) entry. - src/core/cycle.ts: 8 → 9 phases, recompute_emotional_weight inserted between patterns and embed; totals.pages_emotional_weight_recomputed. - src/core/cycle/emotional-weight.ts (NEW): formula + override path. - src/core/cycle/anomaly.ts (NEW): stats helpers + zero-stddev fallback. - src/core/cycle/recompute-emotional-weight.ts (NEW): phase orchestrator. - src/core/transcripts.ts (NEW): library shared by gated MCP op + CLI. - src/core/operations-descriptions.ts (NEW): pinned tool descriptions. - src/core/minions/tools/brain-allowlist.ts: 11 → 13 entries; comment on why get_recent_transcripts is excluded. - src/commands/salience.ts / anomalies.ts / transcripts.ts (NEW): CLI surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rytan/v0.29-salience # Conflicts: # CLAUDE.md # src/cli.ts # src/core/migrate.ts
…rytan/v0.29-salience # Conflicts: # src/core/cycle.ts # src/core/migrate.ts # src/core/operations.ts # src/core/postgres-engine.ts # src/core/types.ts
…rytan/v0.29-salience # Conflicts: # CHANGELOG.md # CLAUDE.md # VERSION # package.json # src/cli.ts # src/core/migrate.ts # src/core/operations.ts
#696) * feat: recency boost for search (v0.27.0) — temporal intent auto-detection, date filters, configurable decay New search pipeline stage: keyword + vector → RRF → cosine re-score → backlink boost → recency boost → dedup - applyRecencyBoost: hyperbolic decay, two strengths (moderate 30-day halflife, aggressive 7-day halflife) - Auto-enabled when intent.ts detects temporal/event queries (detail='high') - Manual override via SearchOpts.recencyBoost (0/1/2) - Date filtering: afterDate/beforeDate on all three search paths (keyword, keywordChunks, vector) - getPageTimestamps on both Postgres and PGLite engines - 15 tests passing (boost math + intent classification) * v0.29.1 schema: pages.{effective_date, effective_date_source, import_filename, salience_touched_at} + expression index Migration v38 adds 4 nullable columns to pages and an expression index on COALESCE(effective_date, updated_at) to support the new since/until date filters. All additive — no behavior change in the default search path; only consulted when callers opt into the new salience='on' / recency='on' axes or pass since/until. effective_date — content date (event_date / date / published / filename-date / fallback). Read by recency boost and date-filter paths only. Auto-link doesn't touch it (immune to updated_at churn). effective_date_source — sentinel for the doctor's effective_date_health check ('event_date' | 'date' | 'published' | 'filename' | 'fallback'). import_filename — basename without extension, captured at import. Used for filename-date precedence on daily/, meetings/. Older rows leave it NULL. salience_touched_at — bumped by recompute_emotional_weight when emotional_weight changes. Salience window uses GREATEST(updated_at, salience_touched_at) so newly-salient old pages enter the recent salience query. Index strategy: a partial index on effective_date alone wouldn't help the COALESCE expression in since/until filters (planner can't use it for the negative side). The expression index ((COALESCE(effective_date, updated_at))) is what actually accelerates the filter. Postgres uses CONCURRENTLY + v14-style pg_index.indisvalid pre-drop guard for prior failed CONCURRENTLY runs; PGLite uses plain CREATE INDEX. Mirror of v34's pattern. src/schema.sql + src/core/pglite-schema.ts updated for fresh installs; src/core/schema-embedded.ts regenerated via bun run build:schema. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: computeEffectiveDate helper + putPage integration Pure helper computing a page's effective_date from frontmatter precedence: 1. event_date (meeting/event pages) 2. date (dated essays) 3. published (writing/) 4. filename-date (leading YYYY-MM-DD in basename) 5. updated_at (fallback) 6. created_at (last resort) Per-prefix override: for daily/ and meetings/ slugs, filename-date jumps to position 1 — the filename is the user's primary signal there. Returns {date, source}. The source label powers the doctor's effective_date_health check to detect "fell back to updated_at" rows that look populated but are functionally a NULL. Range validation: parsed value must be in [1990-01-01, NOW + 1 year]. Out-of-range values drop to the next chain element. Wired into importFromContent + importFromFile. The put_page MCP op derives filename from slug-tail when no caller-supplied filename is available. putPage SQL on both engines extended to write the new columns. ON CONFLICT uses COALESCE(EXCLUDED.x, pages.x) so callers that don't know about the new columns (auto-link, code reindex) preserve existing values rather than blanking them. SELECT projection extended to return them; rowToPage threads them through. 21 unit tests covering: precedence chain default order, per-prefix override, parse failure fall-through, range validation [1990, NOW+1y], parseDateLoose shape variants. All pass; typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: backfill orchestrator + library function for existing pages src/core/backfill-effective-date.ts is the shared library function. Walks pages in keyset-paginated batches (id > last_id ORDER BY id LIMIT 1000), runs computeEffectiveDate per row, UPDATEs effective_date + effective_date_source. Resumable via the `backfill.effective_date.last_id` checkpoint key in the config table — a killed process can re-run and pick up without re-doing rows. Idempotent: a full re-walk produces the same writes. Postgres-only: SET LOCAL statement_timeout = '600s' per batch. Doesn't refuse the migration on low session settings (codex pass-2 #16). src/commands/migrations/v0_29_1.ts is the orchestrator (4 phases mirroring v0_12_2). Phase A schema (gbrain init --migrate-only), Phase B backfill (via the library function), Phase C verify (count NULL effective_date), Phase D record (handled by runner). The library function is reusable from the gbrain reindex-frontmatter CLI command in the next commit. import_filename stays NULL for backfilled rows — pre-v0.29.1 imports didn't capture it. computeEffectiveDate uses the slug-tail when filename is NULL; daily/2024-03-15 backfilled gets effective_date from the slug. Registered in src/commands/migrations/index.ts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: gbrain reindex-frontmatter CLI command Recovery / explicit-rebuild path for pages.effective_date. Used when: - User edited frontmatter dates after import - Post-upgrade backfill orchestrator finished but the user wants to re-walk a subset (e.g. just meetings/) after fixing some frontmatter - Precedence rules change between releases Thin wrapper over backfillEffectiveDate from commit 3 — same code path the v0_29_1 orchestrator uses; one source of truth. Flags mirror reindex-code: --source <id> Scope to one sources row (placeholder; library library doesn't filter by source today, tracked v0.30+) --slug-prefix P Scope to slugs starting with P (e.g. 'meetings/') --dry-run Print what WOULD change, no DB writes --yes Skip confirmation prompt (required for non-TTY non-JSON) --json Machine-readable result envelope --force Re-apply even when computed value matches existing Wired into src/cli.ts. CLI handles its own engine lifecycle (creates + disconnects). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: recency-decay map + buildRecencyComponentSql (pure, unused) src/core/search/recency-decay.ts mirrors source-boost.ts in shape but drives RECENCY ONLY (per D9 codex resolution). Salience is a separate orthogonal axis; this map does not feed it. DEFAULT_RECENCY_DECAY: 10 generic prefixes (no fork-specific names). - concepts/ evergreen (halflifeDays=0) - originals/ 180d × 0.5 (long-tail decay; new essays nudged) - writing/ 365d × 0.4 - daily/ 14d × 1.5 (aggressive — freshness IS the signal) - meetings/ 60d × 1.0 - chat/ 7d × 1.0 - media/x/ 7d × 1.5 - media/articles/ 90d × 0.5 - people/companies/ 365d × 0.3 - deals/ 180d × 0.5 DEFAULT_FALLBACK: 90d × 0.5 for unmatched slugs. Override priority: defaults < gbrain.yml recency: < env (GBRAIN_RECENCY_DECAY) < per-call SearchOpts.recency_decay. parseRecencyDecayEnv format: comma-separated prefix:halflifeDays:coefficient triples. Refuses LOUD on parse error (RecencyDecayParseError) — codex pass-2 #M3 finding. No silent fallback like source-boost's parser. parseRecencyDecayYaml takes already-parsed YAML; throws on bad shape. buildRecencyComponentSql in sql-ranking.ts emits a CASE expression with longest-prefix-first ordering, evergreen short-circuit (literal 0 when halflifeDays=0 or coefficient=0), and EXTRACT(EPOCH ...) for non-zero branches. Output: ((CASE WHEN p.slug LIKE 'daily/%' THEN 1.5 * 14.0 / (14.0 + EXTRACT(EPOCH FROM (NOW() - <dateExpr>))/86400.0) ... END)) Typed NowExpr enum prevents SQL injection (codex pass-1 #5). Tests pass { kind: 'fixed', isoUtc } for deterministic output; production NOW(). The 'fixed' branch escapes single quotes via escapeSqlLiteral. 25 unit tests covering: env parser shape, env error cases, yaml parser shape, merge precedence (defaults < yaml < env < caller), CASE longest- prefix-first ordering, evergreen short-circuit, NowExpr fixed/now, single-quote injection defense, empty decayMap fallback path, default map composition (no fork names, concepts/ evergreen, daily/ aggressive). Pure module. Zero consumers in this commit; commit 6 wires it into getRecentSalience, commit 10 wires it into the post-fusion stage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: refactor getRecentSalience to consume buildRecencyComponentSql Both engines (Postgres + PGLite) now build the salience formula's third term via buildRecencyComponentSql instead of inlining 1.0 / (1 + days_old). Parameters: empty decayMap + fallback { halflifeDays: 1, coefficient: 1.0 }. Math expands to 1 * 1.0 / (1.0 + days_old) = 1 / (1 + days_old) — same numeric output as v0.29.0. This is a no-behavior-change refactor preparing for commit 7's recency_bias param. recency_bias='flat' (default) reproduces v0.29.0 exactly; 'on' swaps in DEFAULT_RECENCY_DECAY for per-prefix decay. Single source of truth for the recency math: same builder feeds the salience query AND (in commit 10) the post-fusion applyRecencyBoost stage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: get_recent_salience gains recency_bias param (default 'flat') SalienceOpts.recency_bias: 'flat' | 'on' added; default 'flat' preserves v0.29.0 ranking verbatim. Pass 'on' to opt into per-prefix decay map (concepts/originals/writing/ evergreen; daily/, media/x/, chat/ aggressive decay). When recency_bias='on', the salience query reads COALESCE(p.effective_date, p.updated_at) instead of bare p.updated_at, so the recency component is immune to auto-link updated_at churn — old concepts/ pages just-touched by auto-link don't suddenly look fresh. Both engines (Postgres + PGLite) wire the param through. resolveRecencyDecayMap() honors gbrain.yml + GBRAIN_RECENCY_DECAY env at runtime. MCP op surface: get_recent_salience gains the param with a load-bearing description teaching the agent when to use 'on' vs 'flat' (current state → on; mattering across all time → flat). No silent v0.29.0 behavior change — opt-in only (per D11 codex resolution). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: recompute_emotional_weight writes salience_touched_at; window picks up newly-salient pages setEmotionalWeightBatch on both engines now bumps salience_touched_at to NOW() ONLY when the new emotional_weight differs from the existing one (IS DISTINCT FROM, NULL-safe). No-op writes (same weight) leave the column alone — preserves "actual change" semantics. getRecentSalience window changes from WHERE p.updated_at >= boundary to WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= boundary Closes codex pass-1 finding #4: pages whose emotional_weight just changed in the dream cycle (because tags or takes shifted) but whose updated_at is older than the salience window now correctly enter the recent-salience results. Without this, "Garry just added a take to a 6-month-old page" stayed invisible to get_recent_salience until the next content edit. COALESCE(salience_touched_at, p.updated_at) handles pre-v0.29.1 rows where salience_touched_at is NULL — they fall back to p.updated_at and behave identically to v0.29.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: merge intent.ts → query-intent.ts; emit 3 suggestions per query D1 + D4 + D6 + D8: single regex-pass classifier returning {intent, suggestedDetail, suggestedSalience, suggestedRecency}. intent + suggestedDetail are v0.29.0 behavior verbatim (legacy intent.ts deleted; classifyQueryIntent + autoDetectDetail compat shims preserved). NEW for v0.29.1 — two orthogonal recency-axis suggestions: suggestedSalience: 'off' | 'on' | 'strong' suggestedRecency: 'off' | 'on' | 'strong' Resolution rules (per D6 narrow temporal-bound exception): - CANONICAL patterns (who is X / what is Y / code / graph) → both off - UNLESS an EXPLICIT_TEMPORAL_BOUND also matches (today / right now / this week / since X / last N days), in which case temporal-bound wins - STRONG_RECENCY (today / right now / this morning / just now) → strong - RECENCY_ON (latest / recent / this week / meeting prep / catch up / remind me / status update) → on - SALIENCE_ON (catch up / remind me / status update / prep me / what's going on / what matters) → on - default → off for both axes (v0.29.1 prime-directive: pure opt-in) Salience and recency are TRULY orthogonal (per D9). A query like "latest news on AI" → recency='on' but salience='off' (the user wants fresh, not emotionally-weighted). "What's going on with widget-co" → both on. "Who is X right now" → both 'strong'/'on' (temporal bound beats canonical 'who is'). intent.ts deleted; test/intent.test.ts renamed → test/query-intent-legacy.test.ts (unchanged behavior coverage). New test/query-intent.test.ts adds 21 cases covering all three axes' interactions: canonical wins on bare 'who is', temporal bound overrides, "catch me up" matches with up to 15 chars between, "today" → strong, intent vs recency independence. Updated callers: - src/core/search/hybrid.ts (autoDetectDetail import) - test/recency-boost.test.ts (classifyQueryIntent import) - test/benchmark-search-quality.ts (autoDetectDetail import) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: applySalienceBoost + applyRecencyBoost + runPostFusionStages wrapper D9 + codex pass-1 #2 + #3 + pass-2 #4: salience and recency are TRULY ORTHOGONAL post-fusion stages, both running from ALL THREE hybridSearch return paths (keyword-only, embed-failure-fallback, full-hybrid). NEW src/core/search/hybrid.ts exports: - applySalienceBoost(results, scores, strength) score *= 1 + k * log(1 + score) where k = 0.15 (on) or 0.30 (strong) No time component. Pure mattering signal. - applyRecencyBoost(results, dates, strength, decayMap, fallback, nowMs?) Per-prefix decay factor: 1 + strengthMul * coefficient * halflife / (halflife + days_old) strengthMul: 1.0 (on) or 1.5 (strong) Evergreen prefixes (halflifeDays=0) skipped (factor 1.0). Pure recency signal. Independent of mattering. - runPostFusionStages(engine, results, opts) Wraps backlink + salience + recency. Called from EACH return path so keyless installs and embed failures get the same boost surface as the full hybrid path. NEW engine methods (composite-keyed for multi-source isolation): - getEffectiveDates(refs: Array<{slug, source_id}>): Map<key, Date> Returns COALESCE(effective_date, updated_at, created_at). Key format: `${source_id}::${slug}`. Mirror of getBacklinkCounts shape. - getSalienceScores(refs: Array<{slug, source_id}>): Map<key, number> Returns emotional_weight × 5 + ln(1 + take_count). Composite key. Deprecated (kept for back-compat through v0.29.x): - SearchOpts.afterDate / beforeDate (alias for since/until) - SearchOpts.recencyBoost: 0|1|2 (alias for recency: 'off'|'on'|'strong') - getPageTimestamps (use getEffectiveDates instead) NEW SearchOpts fields: - salience: 'off' | 'on' | 'strong' - recency: 'off' | 'on' | 'strong' - since: string (ISO-8601 or relative, replaces afterDate) - until: string (replaces beforeDate) Resolution: caller-explicit > legacy alias (recencyBoost) > heuristic (classifyQuery's suggestedSalience / suggestedRecency). Deleted: src/core/search/recency.ts (PR #618's, replaced) + test/recency-boost.test.ts (its scope is replaced by query-intent.test.ts + future post-fusion tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Wintermute <wintermute@garrytan.com> * v0.29.1: query op gains salience + recency + since + until params; PGLite since/until parity Combines commits 12 + 13 of the plan. Query op surface (src/core/operations.ts): - salience: 'off' | 'on' | 'strong' (with load-bearing description) - recency: 'off' | 'on' | 'strong' - since: string (ISO-8601 or relative; replaces deprecated afterDate) - until: string (replaces deprecated beforeDate) Tool descriptions teach the calling agent: - salience axis = mattering, no time component - recency axis = age decay, no mattering signal - omit either to let gbrain auto-detect from query text via classifyQuery hybrid.ts maps since/until → afterDate/beforeDate at the engine call boundary so PR #618's existing engine plumbing keeps working without rename. Codex pass-1 #10 finding closed. PGLite engine (codex pass-1 #10): since/until parity added to all three search methods (searchKeyword, searchKeywordChunks, searchVector). SQL filter against COALESCE(p.effective_date, p.updated_at, p.created_at) so date filtering matches user content-date intent (a meeting was on event_date, not when it got reimported). Filter is applied INSIDE the HNSW inner CTE in searchVector so HNSW's candidate pool already excludes out-of-range pages — preserves pagination contract. This also closes existing cross-engine drift: pre-v0.29.1 Postgres had afterDate/beforeDate from PR #618; PGLite had nothing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: migration v39 — eval_candidates capture columns for replay reproducibility D11 codex pass-2 resolution: extend eval_candidates with 7 new nullable columns so `gbrain eval replay` can reproduce captured runs of agent-explicit salience + recency choices. Without these columns, replays of the new axis params drift. The live behavior depends on the resolved {salience, recency} values; v0.29.0's schema doesn't capture them. as_of_ts TIMESTAMPTZ — brain's logical NOW at capture (replay uses this instead of wall-clock) salience_param TEXT — what the caller passed (NULL if omitted) recency_param TEXT — same salience_resolved TEXT — final value applied recency_resolved TEXT — same salience_source TEXT — 'caller' or 'auto_heuristic' recency_source TEXT — same All nullable + additive. Pre-v0.29.1 rows stay valid. NDJSON schema_version STAYS at 1 — consumers ignore unknown fields (codex pass-1 #C2 dissolves; no cross-repo coordination needed). ADD COLUMN with no DEFAULT is metadata-only on PG 11+ and PGLite — instant on tables of any size. src/schema.sql + src/core/pglite-schema.ts mirror the additions for fresh installs; src/core/schema-embedded.ts regenerated. eval_capture.ts populates the new fields in commit 16 (docs + ship). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: doctor checks — effective_date_health + salience_health effective_date_health: sample-1000 scan detects three classes of problems (codex pass-1 #5 resolution via the effective_date_source sentinel column added in commit 1): fallback_with_fm_date — page fell back to updated_at even though frontmatter has parseable event_date / date / published. The "wrong but populated" residual that earlier review iterations missed. future_dated — effective_date > NOW() + 1 year (corrupt or typo'd century). pre_1990 — effective_date < 1990-01-01 (epoch math gone wrong, bad parse). Sample of last 1000 pages by default — fast on 200K-page brains. Fix hint: gbrain reindex-frontmatter. salience_health: detects pages with active takes whose emotional_weight is still 0 (recompute_emotional_weight phase hasn't run since the take landed). Reports the brain's non-zero emotional_weight count as an informational baseline. Fix hint: gbrain dream --phase recompute_emotional_weight. Both checks gracefully skip on pre-v0.29.1 brains (column doesn't exist → 42703) without surfacing as warnings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: docs + skills convention + CHANGELOG + version bump - VERSION 0.29.0 → 0.29.1 - package.json version bump - CHANGELOG.md: full release-summary + itemized + "To take advantage" block per the project's voice rules. Two-line headline + concrete pathology framing (existing callers unchanged; new axes opt-in; agent in charge per the prime directive). - skills/conventions/salience-and-recency.md: agent-readable decision rules. "Current state → on. Canonical truth → off." plus the narrow temporal-bound exception. Cross-cutting convention propagates to brain skills via RESOLVER.md. - skills/migrations/v0.29.1.md: agent-readable upgrade instructions. Verify steps + behavior-change reference + recovery commands. The build-time tool-description generator from D2 (extract decision tables from skills/conventions/salience-and-recency.md, embed into operations.ts at build time) is deferred to a follow-up commit. The tool descriptions on the query op + get_recent_salience are inline in operations.ts for v0.29.1; the auto-gen + CI staleness gate land in v0.29.2 if drift becomes a problem in practice. 148 unit tests pass across the v0.29.1 surface (effective-date, recency-decay, query-intent, migrate, salience, recompute-emotional-weight). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Wintermute <wintermute@garrytan.com> --------- Co-authored-by: Wintermute <wintermute@garrytan.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 task
Owner
Author
|
Wrong base — this was opened against |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The brain tells you what's hot without being asked. Search rewards hypotheses you already have; salience surfaces ones you don't.
The Garry test: "anything crazy happening in my brain lately?" → first tool call returns the wedding cluster, not a generic vector search. v0.29 ships the primitives that make that possible —
get_recent_salience,find_anomalies,get_recent_transcripts— plus a deterministic emotional-weight score on every page, backfilled by a new dream-cycle phase.Schema
pages.emotional_weight REAL NOT NULL DEFAULT 0.0(column-only, no index — salience query orders by computed score, not raw weight). Instant ALTER on Postgres + PGLite.New cycle phase
recompute_emotional_weight(9th phase, between extract+synthesize and embed). Two SQL round-trips total regardless of brain size: CTE-shaped batch read with per-table aggregates (avoids N×M cartesian) + multi-source-safeUPDATE FROM unnest((slug, source_id, weight))composite UPDATE. <5s on 1000-page PGLite fixture.Engine surface
batchLoadEmotionalInputs,setEmotionalWeightBatch,getRecentSalience,findAnomalies. Salience query computes time boundary in JS and binds as TIMESTAMPTZ so SQL is identical across engines (no dialect drift onintervalparameter binding).PageFilters.sortenum threaded through both engines (was hardcodedORDER BY updated_at DESC).MCP ops
get_recent_salience— pages ranked by(emotional_weight × 5) + ln(1 + take_count) + recency_decay. Subagent-allow-listed.find_anomalies— tag + type cohort outliers vsgenerate_series-densified 30-day baseline. Subagent-allow-listed. Year cohort deferred to v0.30 (slug-regex extraction was fragile).get_recent_transcripts— local-only (rejectsctx.remote === truewithpermission_denied). NOT in subagent allow-list because all subagent calls haveremote=true— would always reject (footgun if visible). Cycle's synthesize phase callsdiscoverTranscriptsdirectly.CLI
gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json]gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]gbrain transcripts recent [--days N] [--full] [--json]Tool description redirects
query,search,list_pagesdescriptions redirect personal/emotional questions to the new ops. Anti-flattery hint: "Do NOT assume words like crazy, notable, or big mean impressive — they often mean difficult or emotionally charged." Extracted tosrc/core/operations-descriptions.tsand pinned by tests.Test Coverage
Tests: 0 → 14 new test files (135 unit cases pass, 23 PGLite e2e pass when run individually, 1 Postgres parity gated, 1 Tier-2 LLM eval gated).
Coverage gate: PASS (100%) — every code path has at least one test, behavior + edge + error tier (★★★).
Pre-Landing Review
Diff was reviewed pre-implementation via
/plan-eng-review(11 issues found, 8 resolved into plan, 0 unresolved at commit 3d5ea23) and outside-voice/codexconsult (10 findings, 10 incorporated). Both reviews logged on the priorgarrytanv0.29branch slug.The codex consult specifically caught:
setEmotionalWeightBatch— fixed via(slug, source_id)composite keygenerate_serieszero-fillengine.putPage(stampsnow()) — fixed via raw SQL backdatelist_pagessort needed engine threading, not handler-only — fixedkindvsPageKind/TakeKind) — renamed toslugPrefixAll 21 findings (eng + codex) incorporated before commit.
Pre-Landing Review at this diff (post-incorporation): No new issues found.
Plan Completion
All decisions resolved (D1–D6 + C-Codex-1..4). Year cohort deferred to v0.30 (intentional scope reduction). 0 unresolved decisions.
TODOS
No TODO items completed in this PR — v0.29 is spec-driven, not closing pre-existing TODOs.
Documentation
CLAUDE.md Key Files updated for the v0.29 surface (new cycle phase, 4 engine methods, 3 MCP ops, 3 CLI commands, allow-list rationale). CHANGELOG.md has the v0.29.0 release-summary in GStack voice plus the "To take advantage of v0.29.0" block recommending
gbrain dream --phase recompute_emotional_weightfor backfill.Test plan
bun run typecheck)gbrain --versionreports0.29.0)engine-parity-salience.test.ts) — DATABASE_URL gated, runs in CI nightlysalience-llm-routing.test.ts) — ANTHROPIC_API_KEY gated, ~$0.10/run🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmithwith what you need.