Skip to content

v0.37.1.0 feat: brainstorm + lsd — bisociation idea generator grounded in your own brain#1214

Merged
garrytan merged 3 commits into
masterfrom
garrytan/calgary-v2
May 20, 2026
Merged

v0.37.1.0 feat: brainstorm + lsd — bisociation idea generator grounded in your own brain#1214
garrytan merged 3 commits into
masterfrom
garrytan/calgary-v2

Conversation

@garrytan

Copy link
Copy Markdown
Owner

Summary

Your brain can now collide its own ideas. Two new CLI commands ship a bisociation idea generator grounded in the user's own knowledge:

  • gbrain brainstorm <question> — defensible, cite-heavy. Pulls 4 close-set pages via hybrid search + 6 far-set pages via prefix-stratified domain bank, crosses them, judges with a 5-axis rubric (threshold 4.0/5). Saves by default to wiki/ideas/<date>-brainstorm-<slug>.md. ~$0.10/run.
  • gbrain lsd <question> — Lateral Synaptic Drift. Same engine, dial at 11: 2×12 close/far, inverted judge that REJECTS ideas scoring too high on coherence ("too obvious"), stale-page bias via pages.last_retrieved_at, axiomatic inversions required. Ephemeral by default. ~$0.30/run.
  • gbrain eval brainstorm <fixture.jsonl> — three-axis evaluation gate (distance + usefulness + grounding, conjunctive). Distance alone is gameable; the grounding axis enforces every idea cites a real slug.

Test Coverage

48 unique paths identified during eng review. 38 new unit tests across test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts pin the math: distance normalization (same-vector→0, orthogonal→0.5, opposite→1), LSD frontmatter detection (with quoted-value tolerance), eval verdict logic (conjunctive pass), grounding rate computation, fixture parsing, JSONL skipping malformed rows.

```
[+] src/core/brainstorm/domain-bank.ts [★★★] distance + grounding + verdict pinned
[+] src/core/brainstorm/judges.ts [★★] config validation + weights sum-to-1
[+] src/core/brainstorm/orchestrator.ts [★★] smoke via CLI + doctor (live brain)
[+] src/core/last-retrieved.ts [★★] fire-and-forget contract verified
[+] src/cli.ts + src/commands/{brainstorm,lsd,eval-brainstorm}.ts [★★] --help works without DB
[+] Migration v79 (renumbered from v77 — master claimed v77+v78) [★★★] schema-bootstrap-coverage extended
```

Tests: 8033 pass on master+branch (1 flake in sync.test.ts:detached HEAD — passes in isolation, unrelated to brainstorm changes, classified pre-existing per /ship triage).

Pre-Landing Review

Two rounds of codex outside-voice review during plan-eng-review caught three critical architectural bugs that would have shipped silent:

  1. diversify-as-rerank was broken — hybridSearch already filters to nearest neighbors; reranking that pool gets "least similar among the similar," not actually distant brain regions. Replaced with separate domain-bank.ts module orthogonal to hybridSearch.
  2. updated_at is not a stale signalputPage bumps it on every sync, so it measures sync churn, not forgottenness. Replaced with real pages.last_retrieved_at column.
  3. connection_count was invented — does not exist as a column. Real signal: JOIN to page_links for inbound link aggregate.

Plus factual fix: migration number bumped v68 (claimed) → v77 (claimed by parallel waves) → v79 (this PR).

Plan Completion

All 11 implementation tasks complete (see ~/.claude/plans/system-instruction-you-are-working-staged-coral.md). Plan went through CEO review (SELECTIVE EXPANSION, 6 proposals / 5 accepted), eng review (8 decisions), 2× codex outside voice (25 findings, all addressed).

Five expansions accepted into scope: D3 eval surface, D4 dream-cycle loop closure, D5 stale-page bias for LSD, D6 citation badges + distance score, D8 cost preview.

Deferred to v0.38+ TODO: D7 MCP exposure of brainstorm (trust-gate design preserved in plan).

TODOS

  • Brainstorm/LSD wave delivered as planned. No prior TODOs blocked by this PR.

Documentation

  • CLAUDE.md Key Files extended with brainstorm/lsd module description (~1300 chars covering domain-bank, orchestrator, judges, operations.ts wire-in, migration v79, doctor check).
  • CHANGELOG.md v0.37.1.0 entry with release-summary format (matches v0.36.4.0 + v0.37.0.0 templates) + "To take advantage of v0.37.1.0" upgrade verification block.
  • llms-full.txt regenerated.

Test plan

  • Typecheck clean (bun run typecheck)
  • Brainstorm tests pass (bun test test/brainstorm/ — 38 pass)
  • Schema bootstrap coverage extended for v79 (test/schema-bootstrap-coverage.test.ts)
  • Migration v79 applies cleanly on PGLite (verified via fresh gbrain init --pglite + gbrain doctor)
  • gbrain brainstorm --help and gbrain lsd --help work without a brain
  • gbrain doctor surfaces brainstorm_health check with paste-ready hints
  • CI guards clean (privacy, jsonb-pattern, progress-to-stdout, trailing-newline)
  • Live smoke test on real brain (post-merge — verify cross-domain ideas surface and cite real slugs)

🤖 Generated with Claude Code

garrytan and others added 2 commits May 19, 2026 20:05
Brainstorm + LSD bisociation idea generator. Will rebase + bump after master merge.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…aimed v0.37.0.0)

Conflicts resolved:
- VERSION + package.json: bumped to 0.37.1.0 (minor bump past master's claim)
- CHANGELOG.md: kept both my v0.37.1.0 brainstorm/lsd entry (renamed from v0.37.0) and master's v0.37.0.0 skillpack-registry entry
- src/core/migrate.ts: my pages_last_retrieved_at migration renumbered v77 → v79 (master's v77 mcp_spend_log + v78 embedding_multimodal_column landed first)

Auto-merged with master's cross-modal search wave, skillpack registry, and DATABASE_URL shell-jobs fix. Typecheck clean. Brainstorm tests (38) pass. Schema-bootstrap test (9) pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
garrytan added a commit that referenced this pull request May 20, 2026
PR #1214 (brainstorm + lsd) claimed v0.37.1.0 concurrently with #1215.
Skipping 0.37.2.0 leaves a buffer for #1214's adjacent slot. Trio
(VERSION + package.json + CHANGELOG header + inline "To take advantage
of v0.37.3.0" block) aligned at 0.37.3.0.

No behavior changes — version metadata only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
garrytan added a commit that referenced this pull request May 20, 2026
PRs #1214 and #1215 both claim v0.37.1.0; bumping past to the next free
slot. Migration v79 renamed `takes_unresolvable_quality_v0_37_0_1` →
`takes_unresolvable_quality_v0_37_2_0`. VERSION + package.json +
CHANGELOG + llms bundles + inline doc references all swept.
…entry

CLAUDE.md gains an IRON RULE for CHANGELOG entries: the changelog describes
what the user gets, not how the work happened. No mentions of review
processes, plan files, decision tags, migration version drama, or 'what we
caught and fixed before merging.' If a fact only exists because of the
development workflow, it does not belong in release notes.

Rewrite v0.37.1.0 entry to comply: cut the 'what we caught' section
(architectural review drama), the 'plan + reviews' bullet, and the
migration-renumbering aside. Entry shrinks 67 → 56 lines, every sentence
now answers 'what can I do / how do I use it / what should I watch for.'

Regenerated llms-full.txt to absorb the CLAUDE.md voice update.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@garrytan garrytan merged commit 39e14cd into master May 20, 2026
7 checks passed
garrytan added a commit that referenced this pull request May 20, 2026
Master landed PR #1214 (brainstorm + lsd) as v0.37.1.0. Wave's
v0.37.3.0 claim sits cleanly above it in the CHANGELOG with both
entries preserved. VERSION + package.json + CHANGELOG header all
at 0.37.3.0; master's v0.37.1.0 entry slots in as the second-most-
recent release. llms-full.txt regenerated against the merged docs.

Verify gate + 114 brain-first + adjacent unit/E2E cases green
post-merge under CI-simulated GBRAIN_HOME=/tmp/empty-... env.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
garrytan added a commit that referenced this pull request May 20, 2026
Master shipped v0.37.1.0 (brainstorm/lsd, PR #1214) claiming migration v79
for `pages_last_retrieved_at`. Renumbered this hotfix's migration v79→v80
to land cleanly on top. v0.37.2.0 remains the version (v0.37.1.0 + v0.37.0.0
already taken on master).

Conflict resolution:
- VERSION/package.json: kept 0.37.2.0 (ours, higher semver)
- CHANGELOG.md: both entries preserved, ours on top
- migrate.ts: kept master's v79 unchanged, added ours as v80
- All references in test/migrate.test.ts, CHANGELOG, CLAUDE.md, spec doc,
  engine.ts updated from v79→v80

Verified: typecheck clean, migrate.test.ts passes (130/130, both v79 and
v80 migrations apply in order).
garrytan added a commit that referenced this pull request May 20, 2026
…1211)

* v0.36.1.1 hotfix: takes_resolution_consistency CHECK accepts 'unresolvable'

Unblocks production grading scripts that write the judge's 4th verdict
type. Before this fix, every quality='unresolvable' INSERT/UPDATE hit
a CHECK violation — 0 of 34 writes landed in a recent prod run.

Migration v74 widens BOTH:
  - takes_resolution_consistency (table-level CHECK) — admits the
    ('unresolvable', NULL) pair alongside the existing 4 legal shapes
  - resolved_quality column-level CHECK — drops the auto-generated
    name from v37, re-adds as takes_resolved_quality_values with the
    4-state enum

Backward compatible. Existing rows with quality IN (NULL, 'correct',
'incorrect', 'partial') all satisfy the new CHECKs unchanged.

TakesScorecard gains sibling fields unresolvable_count + unresolvable_rate;
the existing `resolved` field deliberately keeps its 3-state meaning
so historical scorecards compare apples-to-apples (T1c sibling-field
design from the eng review).

Pinned by:
  - test/takes-resolution.test.ts — R1-R5 round-trip
  - test/migrate.test.ts — v74 structural assertions + PGLite E2E
    suite exercising all valid + invalid (quality, outcome) shapes

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e/schema-drift): reset public schema in beforeAll to isolate from caller bootstrap state

Previously the test trusted caller-provided DATABASE_URL to point at a fresh
database. CLAUDE.md's E2E lifecycle prescribes 'gbrain doctor --json' as the
bootstrap step (needed by oauth-related tests for table creation), but doctor
configures the gateway and bakes the configured embedding model into
content_chunks.model DEFAULT during the initial CREATE TABLE.

On re-run, CREATE TABLE IF NOT EXISTS is a no-op and the bootstrapped default
sticks. PGLite (always fresh-in-memory) gets the unconfigured-gateway fallback
'text-embedding-3-large'. The test reported phantom drift:
  pg.default="'zembed-1'::text"  pglite.default="'text-embedding-3-large'::text"

Fix: DROP SCHEMA public CASCADE + CREATE SCHEMA public before pg.initSchema.
Resets every table/index/sequence/constraint added by prior tooling. The PGLite
side is already fresh-per-test by construction.

Verified order-independent:
  - Fresh DB → 6/0 pass
  - After 'gbrain doctor' bootstrap → 6/0 pass
  - Full E2E suite (mechanical + schema-drift) → 84/0 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(safety): gate schema-drift DROP SCHEMA + relax TakesScorecard interface

Two findings from Codex adversarial review on the v0.37.0.1 hotfix:

1. **DROP SCHEMA safety gate (P0).** test/e2e/schema-drift.test.ts had an
   unguarded DROP SCHEMA public CASCADE. A developer running the E2E with
   DATABASE_URL pointing at a real brain or staging DB would lose the entire
   public schema. The fix: triple-check before destruction.
   - Parse the DATABASE_URL hostname + db name
   - Allow reset only when: explicit GBRAIN_TEST_DB=1 OR (localhost host AND
     test-shaped db name like gbrain_test, *_test, test_*, *_e2e)
   - Refuse otherwise with a loud paste-ready warning
   - The test still proceeds (the parity check is the fail-safe — if the
     caller already had a fresh DB, parity passes; if not, parity fails
     LOUDLY instead of nuking their data)

   Verified all three branches: localhost+gbrain_test resets (6/0 pass);
   localhost+production_brain refuses + warns (6/0 pass against pre-existing
   schema); GBRAIN_TEST_DB=1 override on production_brain name allows reset.

2. **TakesScorecard interface compat.** Making `unresolvable_count` +
   `unresolvable_rate` required fields on the public TakesScorecard
   interface broke downstream SDK consumers who construct scorecard
   fixtures (gbrain-evals, custom engines). The hotfix shouldn't impose
   a compile-break on hotfix users.

   Fix: make both fields optional (`?: number` / `?: number | null`).
   `finalizeScorecard` still always populates them, so all internal code
   sees the real values. External fixtures that omit them compile cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(safety,ux): tighten DROP SCHEMA gate + surface unresolvable in scorecard CLI

Second codex adversarial pass on v0.37.0.1 surfaced two residual findings.

**P0 — Safety gate still bypassable.** First-pass safety gate used
`explicitOptIn || (isLocalhost && looksLikeTestDb)` — meaning
`GBRAIN_TEST_DB=1` bypassed BOTH the host check AND the db-name check.
Someone running the E2E with that env set against a production DATABASE_URL
would still nuke their schema. Codex re-flagged it as P0.

Tightened logic: `looksLikeTestDb && (isLocalhost || ciOptIn)`. The db-name
pattern is now the hard floor — `gbrain_test`, `*_test`, `test_*`, `*_e2e`.
GBRAIN_TEST_DB=1 only relaxes the localhost requirement (for CI service-name
hosts). Setting the env on a DATABASE_URL pointing at `production_data` is
explicitly refused with a paste-ready message naming the failed check.

Verified 3 ways:
  - gbrain_test + localhost → resets (6/0 pass)
  - production_data + GBRAIN_TEST_DB=1 → REFUSES with clear message
  - foo_e2e + GBRAIN_TEST_DB=1 → resets (test-shaped name passes)

**P2 — gbrain takes scorecard hides the unresolvable signal.** Early-return
on `resolved === 0` was triggered before the new sibling fields rendered.
A brain with only `quality='unresolvable'` verdicts — the spec's whole
production case — printed "No resolved bets yet" and exited. The
unresolvable_rate field was unreachable from the human CLI unless the user
knew to pass `--json`.

Fix: gate the early-return on `resolved === 0 AND unresolvable_count === 0`.
Render `unresolvable` count + `unresolvable_rate` alongside `partial_rate`
when present. Threshold warn at 30% (mirrors PARTIAL_RATE_WARNING_THRESHOLD)
pointing at retrieval coverage, not prediction accuracy — the actionable
read for high-unresolvable brains.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: renumber v0.37.0.1 → v0.37.2.0 (v0.37.1.0 claimed by other PRs)

PRs #1214 and #1215 both claim v0.37.1.0; bumping past to the next free
slot. Migration v79 renamed `takes_unresolvable_quality_v0_37_0_1` →
`takes_unresolvable_quality_v0_37_2_0`. VERSION + package.json +
CHANGELOG + llms bundles + inline doc references all swept.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
garrytan added a commit that referenced this pull request May 20, 2026
…ve opt-out (supersedes #1206) (#1215)

* v0.37.1.0 feat: skill_brain_first doctor check + auto-fix + declarative opt-out

Cathedral wave superseding PR #1206. Doctor now scans every SKILL.md for
external-lookup tools (web_search / web_fetch / exa / perplexity / happenstance
/ crustdata / captain_api / firecrawl) and warns when the skill has no brain-
first compliance signal. gbrain doctor --fix auto-inserts the canonical
> **Convention:** see [conventions/brain-first.md](...) callout via the
dry-fix.ts MISSING_RULE_PATTERNS extension (sharing safety gates with the
existing REPLACE patterns).

Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged
Garry's Palantir tweet as risky because no model knew he built it, but the
brain already had "designed the entire Finance product UI" and "150+ PSDs
from April-December 2006." Static check catches authorship; v0.37+ runtime
gate (filed in TODOS.md) closes the dispatch side.

Key design decisions locked via /plan-eng-review + codex outside-voice review:
- A1: frontmatter ships only brain_first: exempt (no required/n/a enum)
- A2: snapshot+diff audit at ~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl
  with transition-only writes (stable brains = 0 lines/run)
- A3: scaffold template pre-inserts callout; skillify check fails (exit 1)
  on external + no callout + no exempt
- A4: position-relative gate is BODY-ONLY (frontmatter tools: [web_search]
  declaration doesn't false-flag the skill)
- Q1: single pure analyzeSkillBrainFirst() helper consumed by 3 surfaces
- CMT1: no upgrade migration — doctor surfaces hint, --fix applies via
  dry-fix safety gates (user stays in loop)
- CMT2: dropped tools+writes_pages auto-exemption (was hiding mixed-class
  skills like idea-ingest/meeting-ingestion/data-research)

Trio: VERSION + package.json + CHANGELOG aligned at 0.37.1.0. 56 unit cases
+ 12 E2E cases pass. 170 related existing tests pass unchanged. Self-dogfood:
gbrain doctor against this repo's skills/ reports skill_brain_first: ok
across 43 skills (compliant or exempt). functional-area-resolver and
strategic-reading skills gained brain_first: exempt to validate the
declarative opt-out in production code (both name perplexity in dispatcher
prose without calling it).

Co-Authored-By: garrytan-agents <noreply@github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update CLAUDE.md for v0.37.1.0 skill_brain_first wave

Added Key Files entries for the four new modules:
- src/core/skill-frontmatter.ts (shared parser)
- src/core/skill-brain-first.ts (analyzer + FORMERLY_HARDCODED_EXEMPT)
- src/core/skill-fix-gates.ts (extracted safety primitives)
- src/core/audit-skill-brain-first.ts (snapshot+diff JSONL)

Extended existing entries:
- src/core/filing-audit.ts: rewired to shared parser
- src/core/dry-fix.ts: MISSING_RULE_PATTERNS INSERT pattern type
- src/commands/doctor.ts: skill_brain_first check + tweet-shield framing
- src/commands/skillify-check.ts: required item 12 + scaffold pre-insert

Added test inventory entries:
- test/skill-brain-first.test.ts (56 unit cases)
- test/e2e/skill-brain-first.test.ts (12 E2E cases)

Regenerated llms-full.txt via bun run build:llms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): skill_brain_first guard uses doctor --fast to skip engine connect

CI run #76881161092 failed because scripts/check-skill-brain-first.sh
invoked plain `gbrain doctor --json`, which routes through connectEngine().
With no ~/.gbrain/config.json present (CI's case — runner is bun-only,
no brain init), connectEngine() exits 1 with "No brain configured." and
emits zero stdout. The python parser sees an empty file and returns
parse_error, failing the verify gate.

Fix: pass --fast to doctor. --fast routes through runDoctor(null, ...)
which runs the filesystem-only check set (resolver_health,
skill_conformance, skill_brain_first) and emits the standard
single-line JSON envelope the parser expects. skill_brain_first is
filesystem-only by design (scans SKILL.md, no DB touch), so --fast is
the correct knob, not a workaround.

Verified by reproducing the CI failure mode locally with
GBRAIN_HOME=/tmp/empty-... — gate now passes both with and without
a configured brain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebump v0.37.1.0 → v0.37.3.0 (queue collision with #1214)

PR #1214 (brainstorm + lsd) claimed v0.37.1.0 concurrently with #1215.
Skipping 0.37.2.0 leaves a buffer for #1214's adjacent slot. Trio
(VERSION + package.json + CHANGELOG header + inline "To take advantage
of v0.37.3.0" block) aligned at 0.37.3.0.

No behavior changes — version metadata only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <noreply@github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mgunnin added a commit to mgunnin/gbrain that referenced this pull request May 28, 2026
* upstream/master:
  v0.38.2.0 fix(doctor): bounded frontmatter scan + partial-state surfacing (supersedes garrytan#1287) (garrytan#1297)
  v0.38.1.0 feat(agents): provider-agnostic subagent loop + remote MCP dispatch + budget meter (garrytan#1289)
  v0.38.0.0 ingestion cathedral — gbrain capture + write-through + IngestionSource contract (garrytan#1275)
  v0.37.11.0: fresh-install PGLite embedding setup fix wave (garrytan#1286)
  v0.37.10.0 feat(init): env-detection + interactive picker + preflight invariants (garrytan#1278)
  v0.37.9.0 fix(frontmatter): canonical-style normalization for tag arrays (garrytan#1252)
  v0.37.8.0 feat: voyage-code-3 discoverability + reindex-code cost-preview fix (garrytan#1267)
  v0.37.7.0 fix wave: federated brains + autopilot safety + OAuth confidential clients (garrytan#1253)
  v0.37.6.0 feat(ai): OpenRouter recipe + generic default_headers seam (cherry-pick garrytan#1210) (garrytan#1246)
  v0.37.5.0 fix(markdown): YAML-aware NESTED_QUOTES validator (stops flagging valid YAML) (garrytan#1229)
  feat: pgGraph-inspired CI scaffolding wave (v0.37.4.0) (garrytan#1228)
  v0.37.3.0 feat: skill_brain_first doctor check + auto-fix + declarative opt-out (supersedes garrytan#1206) (garrytan#1215)
  v0.37.2.0: takes_resolution_consistency CHECK accepts 'unresolvable' (garrytan#1211)
  v0.37.1.0 feat: brainstorm + lsd — bisociation idea generator grounded in your own brain (garrytan#1214)
  v0.37.0.0 feat(skillpack): registry cathedral — third-party publish + install + 10/10 quality bar (garrytan#1208)
  v0.36.6.0 feat: cross-modal search wave (text↔image + unified column + LLM intent) (garrytan#1165)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant