feat(#197): gbrain memory backend + CRDB adapter + hybrid composer (default)#250
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR completes issue #197 by making gbrain the default MemoryPort backend, adding a CockroachDB-backed repository + in-memory mirror, and extending the hybrid composer plus API/Web configuration surfaces to support per-install/per-user backend selection and observability.
Changes:
- Added
@skytwin/memory-gbrain-crdb-adapter(CRDB repository, in-memory store, embeddings, RRF) and CRDB migration040-gbrain-memory.sql. - Promoted
@skytwin/memory-gbrainto an in-processEmbeddedGbrainMemoryPortand expanded hybrid composer diagnostics/routing behavior. - Added
/api/memory-configendpoints and a new web “Memory backend” settings page, plus docs/changelog updates.
Reviewed changes
Copilot reviewed 36 out of 38 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tsconfig.json | Adds TS path alias for the new CRDB adapter package. |
| README.md | Updates feature list to describe swappable memory backends (gbrain default). |
| pnpm-lock.yaml | Adds workspace links for new/updated memory packages. |
| packages/memory-hybrid/src/index.ts | Re-exports HybridDiagnostics type. |
| packages/memory-hybrid/src/hybrid-port.ts | Adds diagnostics counters and improves read routing fallback behavior. |
| packages/memory-hybrid/src/tests/diagnostics.test.ts | Adds unit tests covering diagnostics + fallback routing behavior. |
| packages/memory-gbrain/src/index.ts | Exports embedded port + re-exports embedding providers from adapter. |
| packages/memory-gbrain/src/embedded-port.ts | Implements the in-process gbrain MemoryPort (CRDB + in-memory backend). |
| packages/memory-gbrain/src/cli-detector.ts | Adds external gbrain config detection in addition to CLI detection. |
| packages/memory-gbrain/src/tests/perf.test.ts | Adds quality/perf-oriented benchmarks (one always-on quality test, one opt-in perf test). |
| packages/memory-gbrain/src/tests/integration.test.ts | Adds end-to-end integration tests against the in-memory backend. |
| packages/memory-gbrain/src/tests/embedded-port.test.ts | Adds unit tests for embedded port behavior (write/read/export/import/etc.). |
| packages/memory-gbrain/src/tests/cli-detector.test.ts | Adds tests for external gbrain config detection. |
| packages/memory-gbrain/package.json | Adds dependency on CRDB adapter + dev dep on memory-hybrid for tests. |
| packages/memory-gbrain-crdb-adapter/tsconfig.json | New package TS build configuration. |
| packages/memory-gbrain-crdb-adapter/src/types.ts | Defines DB row shapes and insert inputs for brain_* tables. |
| packages/memory-gbrain-crdb-adapter/src/rrf.ts | Implements Reciprocal Rank Fusion fold logic. |
| packages/memory-gbrain-crdb-adapter/src/repository.ts | CRDB repository: CRUD, hybridSearch, and embedding job queue helpers. |
| packages/memory-gbrain-crdb-adapter/src/index.ts | Public exports for repository, embeddings, in-memory store, and RRF. |
| packages/memory-gbrain-crdb-adapter/src/in-memory-repository.ts | In-memory mirror of the repository surface for tests/hermetic runs. |
| packages/memory-gbrain-crdb-adapter/src/embedding.ts | Embedding providers: hash-trick + OpenAI-compatible HTTP provider. |
| packages/memory-gbrain-crdb-adapter/src/tests/rrf.test.ts | Unit tests for RRF fold behavior. |
| packages/memory-gbrain-crdb-adapter/src/tests/integration-crdb.test.ts | DB-gated integration tests against live CRDB. |
| packages/memory-gbrain-crdb-adapter/src/tests/in-memory-repository.test.ts | Unit tests for in-memory repository semantics. |
| packages/memory-gbrain-crdb-adapter/src/tests/embedding.test.ts | Unit tests for tokenization, hashing, cosine similarity, and OpenAI provider. |
| packages/memory-gbrain-crdb-adapter/package.json | New package manifest/scripts/deps. |
| packages/db/src/migrations/040-gbrain-memory.sql | Adds brain_* tables and indexes to support gbrain storage + embedding jobs. |
| docs/memory-swap.md | Adds runbook documentation for backend selection, hybrid, and ops knobs. |
| CLAUDE.md | Updates package list/roles to include new memory packages. |
| CHANGELOG.md | Adds unreleased entry detailing gbrain backend + adapter + hybrid changes. |
| apps/web/public/js/pages/memory-settings.js | New SPA page to view/switch backends and view diagnostics. |
| apps/web/public/js/app.js | Registers the new /memory-settings route. |
| apps/api/src/routes/memory-config.ts | Adds API endpoints to get/set backend, dismiss notice, and read diagnostics. |
| apps/api/src/memory-setup.ts | New backend factory resolving env + per-user settings and wiring hybrid mode. |
| apps/api/src/index.ts | Mounts /api/memory-config router under auth/ownership middleware. |
| apps/api/src/tests/memory-setup.test.ts | Unit tests for backend selection + embedding provider selection. |
| apps/api/src/tests/memory-config-routes.test.ts | E2E tests for memory-config routes with adapter mocked. |
| apps/api/package.json | Adds dependencies on the new/updated memory packages. |
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
Comments suppressed due to low confidence (1)
packages/memory-hybrid/src/hybrid-port.ts:171
- Read routing now relies on
resolveReadPort+ capability checks, butgetEntitiesByTypeis not routed through this mechanism (it stays hard-wired later in the file). Since the embedded gbrain port now supports entity reads, hybrid mode can end up sending entity queries to the secondary even when the primary can serve them (and the secondary may be a stub). Consider addinggetEntitiesByTypetoRoutingRulesand routing it viaresolveReadPort(or defaulting it to primary).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+107
to
+112
| this.backend = opts.backend ?? 'crdb'; | ||
| this.store = opts.store ?? (this.backend === 'memory' ? new InMemoryBrainStore() : undefined); | ||
| this.rrfK = opts.rrfK ?? 60; | ||
| this.candidatePoolSize = opts.candidatePoolSize ?? 40; | ||
| this.embedQueriesSync = opts.embedQueriesSynchronously ?? true; | ||
| } |
Owner
Author
There was a problem hiding this comment.
Addressed in 42283af (post-/review fixes): candidatePoolSize is now candidatePoolOverride (stored only when explicitly set) and the search path computes Math.max(k*4, 40) per query at embedded-port.ts:366. Reading a large k no longer caps the candidate pool at 40.
Comment on lines
+151
to
+156
| source: 'signal', | ||
| sourceRef: s.id, | ||
| metadata: { signalSource: s.source, signalType: s.type }, | ||
| embedding: await this.embedding.embed(summaryText).catch(() => undefined), | ||
| embeddingModel: this.embedding.model, | ||
| }); |
Owner
Author
There was a problem hiding this comment.
Addressed in 42283af. The in-memory recordSignal/recordEntity/recordEpisode paths now use the same conditional embeddingModel spread as the CRDB paths — { ...(embedding ? { embedding, embeddingModel: this.embedding.model } : {}) } — so pages with a failed embedding never get a non-null model. See embedded-port.ts:161.
Comment on lines
+43
to
+49
| const [settings, resolved, suggestion, counts, pendingJobs] = await Promise.all([ | ||
| getBrainSettings(userId).catch(() => null), | ||
| getMemoryPortForUser(userId), | ||
| Promise.resolve(suggestHybridUpgrade()), | ||
| countPages(userId).catch(() => ({ total: 0, embedded: 0 })), | ||
| pendingEmbeddingJobs().catch(() => 0), | ||
| ]); |
Owner
Author
There was a problem hiding this comment.
Addressed in 42283af. pendingEmbeddingJobs now accepts an optional userId: the API/dashboard routes pass it (memory-config.ts:50, :160), and the worker keeps the global count for its drain-loop telemetry. See repository.ts:537.
Comment on lines
+79
to
+100
| /** | ||
| * Stub MemoryPort that returns empty results everywhere. Used as the secondary | ||
| * in hybrid mode when MemPalace is unavailable, and as the standalone port | ||
| * when the user explicitly selects 'mempalace' but no real adapter is wired. | ||
| * | ||
| * We ship the stub rather than refusing to load: the existing mempalace | ||
| * routes (apps/api/src/routes/mempalace.ts) still work for legacy callers | ||
| * who depend on them — the stub here is just for the MemoryPort surface | ||
| * which is currently mostly consumed via gbrain. A v1.1 follow-up will | ||
| * replace this with a real MemPalaceMemoryPort wired to mempalaceRepository. | ||
| */ | ||
| class StubMempalacePort implements MemoryPort { | ||
| capabilities(): Set<MemoryCapability> { | ||
| return new Set<MemoryCapability>(['spatial_wings', 'aaak_compression']); | ||
| } | ||
| async recordSignal() {} | ||
| async recordEntity() {} | ||
| async recordTriple() {} | ||
| async recordEpisode() {} | ||
| async searchSemantic() { return []; } | ||
| async walkGraph() { return []; } | ||
| async getEpisodes() { return []; } |
Owner
Author
There was a problem hiding this comment.
Addressed in 42283af. buildMempalacePort() now returns a real MemPalaceMemoryPort wired to mempalaceRepository (memory-setup.ts:110-232), used both for MEMORY_BACKEND=mempalace and as the hybrid secondary. The stub is gone.
|
|
||
| document.addEventListener('click', async (event) => { | ||
| if (window.location.hash.split('?')[0] !== '#/memory-settings') return; | ||
| const target = event.target.closest('[data-action]'); |
Owner
Author
There was a problem hiding this comment.
Addressed in 42283af. The delegated handler now has if (!(event.target instanceof Element)) return; before event.target.closest(...) (memory-settings.js:73-74), matching the pattern used elsewhere in the SPA.
Comment on lines
+20
to
+34
| - **Optional:** `hybrid` mode composes gbrain (primary) with a secondary | ||
| backend — typically a mempalace adapter — to get the union of capabilities | ||
| (semantic + spatial + AAAK compression). | ||
| - **Legacy / single-engine:** `mempalace`. Selectable for users who prefer | ||
| the original spatial system; declares no semantic_search capability so | ||
| most retrieval falls back to keyword search. | ||
|
|
||
| ## Backends at a glance | ||
|
|
||
| | backend | semantic_search | code_aware_search | episodic | graph_walk | temporal_triples | spatial_wings | aaak_compression | | ||
| |-------------|:---------------:|:-----------------:|:--------:|:----------:|:----------------:|:-------------:|:----------------:| | ||
| | `gbrain` | ✓ | ✓ | ✓ | ✓ | ✓ | | | | ||
| | `hybrid` | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ||
| | `mempalace` | | | | | | ✓ | ✓ | | ||
|
|
Owner
Author
There was a problem hiding this comment.
Addressed in 42283af (table) and d112aaa (prose). The capability table now shows mempalace as ILIKE-backed for semantic_search, and the bullet at the top of memory-swap.md was tightened to match — it had still said "declares no semantic_search capability" which contradicted both the code and the table. MemPalaceMemoryPort.capabilities() does declare semantic_search; the runbook now explains it's keyword-ILIKE, not vector.
Comment on lines
+124
to
+133
| -- shape so MemoryRecord round-trips without information loss. | ||
| CREATE TABLE IF NOT EXISTS brain_signals ( | ||
| id UUID PRIMARY KEY, | ||
| user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE, | ||
| source STRING NOT NULL, | ||
| type STRING NOT NULL, | ||
| data JSONB NOT NULL DEFAULT '{}', | ||
| recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(), | ||
| signal_timestamp TIMESTAMPTZ NOT NULL | ||
| ); |
Owner
Author
There was a problem hiding this comment.
Addressed in 42283af. Migration 040 now uses id STRING PRIMARY KEY DEFAULT gen_random_uuid()::STRING on brain_pages, brain_entities, brain_triples, brain_episodes, and brain_signals so non-UUID source IDs (sig_gmail_abc123, msg-1) are accepted directly without an extra surrogate column. The default still generates UUID-shaped strings when the caller doesn't supply an id.
…efault) Promotes @skytwin/memory-gbrain from a CLI-shellout skeleton (PR #215) to a real, in-process, CockroachDB-backed memory layer. Default MemoryPort for new installs is now gbrain — vector embeddings + tsvector full-text search fused via Reciprocal Rank Fusion. No separate Postgres process, no external CLI install — gbrain runs against the SkyTwin DB stack directly. Per user direction: gbrain is the default, mempalace is the second option, and everything works against CRDB where possible. Ships: - 040-gbrain-memory.sql: brain_pages (FLOAT8[] embedding + TSVECTOR with inverted index), brain_entities, brain_triples, brain_episodes, brain_signals, brain_settings, brain_embedding_jobs (FOR UPDATE SKIP LOCKED queue). - @skytwin/memory-gbrain-crdb-adapter (NEW): repository.ts (CRDB-backed + hybridSearch), in-memory-repository.ts (test-friendly mirror), embedding.ts (HashEmbeddingProvider deterministic fallback + OpenAiEmbeddingProvider for any /v1/embeddings endpoint), rrf.ts. - @skytwin/memory-gbrain: EmbeddedGbrainMemoryPort with the full MemoryPort surface (semantic_search, code_aware_search, temporal_triples, episodic, graph_walk); searchCodeAware boost; hasExternalGbrainConfig() detection. - @skytwin/memory-hybrid: diagnostics counters + capability-aware fallback. - apps/api/src/memory-setup.ts: per-user backend factory (default 'gbrain'; MEMORY_BACKEND env override; per-user brain_settings.backend wins). - apps/api/src/routes/memory-config.ts: GET/POST /api/memory-config, /dismiss-notification, /diagnostics. - apps/web memory-settings page with the "your twin got smarter" notice. - docs/memory-swap.md: backends-at-a-glance, env knobs, rollback path. Tests: 145+ new (49 CRDB adapter + 50 memory-gbrain + 9 hybrid diagnostics + 21 api memory-setup/routes + 6 DB-gated integration). Full suite: 70/70 turbo tasks pass. Closes #197. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
/review caught a real bug: the migration's `brain_settings.backend DEFAULT 'hybrid'`
disagreed with the factory's `'gbrain'` default in `apps/api/src/memory-setup.ts`.
Failure mode: a fresh user (no brain_settings row) hitting
POST /api/memory-config/dismiss-notification triggered upsertSettings({hybrid_notification_dismissed:true}).
The COALESCE in INSERT defaulted backend → 'hybrid' even though the factory
considers a missing row to mean 'gbrain'. Result: dismissing the notification
silently flipped the user's backend.
Fixed in three places (must stay in sync — comment links the others):
- packages/db/src/migrations/040-gbrain-memory.sql: column DEFAULT 'gbrain'
- packages/memory-gbrain-crdb-adapter/src/repository.ts: upsertSettings COALESCE 'gbrain'
- packages/memory-gbrain-crdb-adapter/src/in-memory-repository.ts: upsertSettings fallback 'gbrain'
Plus a regression test on the in-memory store.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ation Adds 50+ tests that drive the gbrain memory layer with realistic data and edge cases — addressing the user's "deeply test this with realish examples" request. The point isn't unit coverage (we already had that); it's "does the system actually build a profile when fed real-life data?" New test files: - realistic-corpus.ts fixture: ~30 labeled signals (Gmail, calendar, notes, code, chat) modelled after a real twin's first month, plus deterministic noise generators to scale to 500. - realistic-retrieval.test.ts: R@5/P@5 floor with labeled relevance, hybrid-vs-text-only ablation, multi-user isolation under load (6 users, 500 signals each, no cross-talk). - persona-sam-patel.ts fixture: a 6-week storyline for a Series A founder (fundraise prep → VC meetings → term sheet → hiring loops → close → vacation), with tagged signals + entities + triples + episodes. - persona-simulation.test.ts: drives the full storyline end-to-end and checks every load-bearing twin behaviour: entity recognition, graph walks (Mahesh → Anchor VC → Beacon Series A), triple filters, time-bounded episode lookup, semantic search on natural-language founder questions, profile summarisation, full export → import round trip with answer parity, and week-by-week incremental emergence. - concurrent-worker.test.ts: 200 parallel recordSignal calls; failed embeddings get queued; worker drains the queue with FOR UPDATE SKIP LOCKED semantics; failed jobs exhaust retries and stop blocking. - migration.test.ts: mempalace-flavoured export → gbrain importAll → imported content is searchable; idempotent re-import skips dupes; export → import → re-export histogram parity. - robustness.test.ts: every degraded mode — embedding throws / times out / returns junk, queries empty / oversize / punctuation-only, mixed-dim vector corpus (model migration), pages with null embedding, OpenAI HTTP abort/timeout, multi-tenant safety under partial failure. - memory-config-roundtrip.test.ts: real Express + real factory + real EmbeddedGbrainMemoryPort + real HybridMemoryPort end-to-end. Stubs only the @skytwin/db query layer. Verifies the dismiss-notification fix (default backend STAYS gbrain on a fresh user). Test totals: 86 memory-gbrain (was 18) + 50 CRDB adapter + 19 hybrid + 26 api = 181 tests across the new memory subsystem. Full suite: 70/70 turbo tasks pass. Honest about hash-trick limits: the persona test asserts ≥80% recall across founder questions rather than 100%, because the deterministic fallback embedding is intentionally weak. With OpenAI text-embedding-3-small the same test suite runs at materially higher recall — but the floor here catches retrieval-pipeline regressions without flaking on embedding quality. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… finding
Constructs a complete fake user (Bob Patel, Series A SaaS founder) with
realistic preferences, behavioural patterns, traits, and trust tier
MODERATE_AUTONOMY. Wires the actual DecisionMaker + TwinService +
PolicyEvaluator against in-memory ports and feeds a realistic inbox
through the pipeline.
This is the user's "would the system actually do the email" check.
Surfaces a real finding: with the rule-based fallback CandidateGenerator,
the DecisionMaker auto-archives BOARD CHAIR and CFO emails because the
candidates are content-blind — `archive_email` is generated for every
EMAIL_TRIAGE situation regardless of sender. Bob's high-confidence
preference "board threads always require approval" doesn't gate the
candidate; it just informs scoring. Result at MODERATE_AUTONOMY:
[AUTO-EXECUTE ] archive_email — Stratechery newsletter ← right
[AUTO-EXECUTE ] archive_email — Board chair: May meeting ← WRONG
[AUTO-EXECUTE ] archive_email — CFO: Q2 forecast review ← WRONG
[NEEDS APPROVAL] accept_invite — Eng leadership 1:1 ← right
[AUTO-EXECUTE ] snooze_reminder — Adobe Creative Cloud ← right
[AUTO-EXECUTE ] escalate_to_user — Friendly check-in ← right
Production safeguards against this:
1. OBSERVER / SUGGEST trust tier always gates everything (test asserts).
2. A sender-aware `CandidateGenerator` reads sender + content and
produces an irreversible `flag_for_manual_review` candidate for
protected senders. The included `protectiveGenerator` demonstrates
this — same shape as the LLM strategy that runs in production.
With the protective generator wired in:
[AUTO-EXECUTE ] archive_email — Stratechery newsletter
[NEEDS APPROVAL] flag_for_manual_review — Board chair
[NEEDS APPROVAL] flag_for_manual_review — CFO
16 tests across two describe blocks. Also exports LabelInferencePort and
SenderLabelHint from @skytwin/decision-engine so tests can build the
custom Gmail-history-aware label hint port (#122).
Full suite: 70/70 turbo tasks pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ontext
Closes the safety gap surfaced by the fake-user E2E: the rule-based
candidate generator was content-blind, so at MODERATE_AUTONOMY the twin
auto-archived board chair / CFO / legal emails the same way it auto-archived
newsletters. This patch lands two production wirings:
1. New @skytwin/decision-engine export `SenderAwareCandidateGenerator` —
a CandidateGenerator that wraps the rule-based generator with a
pre-pass on `decision.rawData.from` and decision content. When the
sender or subject matches a protected pattern (board/chair/cfo/coo/
ceo/founder/partner/investor/legal/counsel/attorney/sec/audit/
compliance/tax) or content mentions a protected topic
(term sheet/wire transfer/signed/nda/equity/cap table/board deck/
earnings/payroll), the generator SUPPRESSES the rule-based candidate
set entirely and emits ONLY a `flag_for_manual_review` candidate
(irreversible, CONFIDENCE: CONFIRMED). The built-in policy
NO_IRREVERSIBLE_WITHOUT_APPROVAL gates this through the approval
queue at every trust tier.
Suppressing the base set (rather than just prepending the flag) is
load-bearing: if archive_email is in the candidate list it scores
higher than flag (lower risk because reversible) and would auto-execute
anyway — the very bug we are fixing.
Configurable via `protectedPattern` and `protectedSubjectPattern`
constructor options; defaults match common corporate email surface area.
2. Wired SenderAwareCandidateGenerator into events.ts as the rule-based
fallback. Used both:
- directly as the DecisionMaker's CandidateGenerator when no LLM
client is configured
- as the inner RuleBasedCandidateGenerator that LLM strategies fall
back to when LLM calls fail
This means the safety improvement applies both to users without LLM
keys (rule-based by default) and to LLM users when their LLM call
fails — there is no path through events.ts that auto-archives a board
email at MODERATE_AUTONOMY+.
3. Wired episodicMemories into DecisionContext. mempalaceRepository
.getEpisodes is fetched in parallel with patterns/traits/temporal
profile, mapped onto the EpisodicMemory shape, and passed to
DecisionMaker.evaluate. The existing scoreCandidate boost
(decision-maker.ts:1285+) consumes this field to weight candidates
that match historically-positive past decisions. Closes the
"twin's memory of past decisions affects current decisions" loop
that was structurally present (the field existed) but unwired.
Tests:
- 12 unit tests for SenderAwareCandidateGenerator covering: protected
senders (board/CFO/legal/investor), protected subjects (term sheet,
wire transfer, cap table), routine email passthrough, non-email
situations passthrough, custom pattern overrides.
- 3 integration tests for the events.ts wiring: board chair email
selects flag_for_manual_review and does not auto-execute; routine
newsletter selects archive/label; mempalaceRepository.getEpisodes
is called with the right (userId, {domain, situationType, limit}).
- Updated existing events-routes.test.ts mock to include
SenderAwareCandidateGenerator + emailLabelRepository + mempalaceRepository.
Full suite: 70/70 turbo tasks pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ueue
Required when an external embedding provider (OpenAI, Ollama, vLLM) is
configured: the synchronous embed call inside recordSignal can fail
(rate limit, network, timeout). The write path persists the page row
unembedded and queues a job to brain_embedding_jobs. Without this worker
the queue never drains and search recall silently degrades — pages exist
in tsvector index but not the vector index, so RRF gives them only the
text-rank contribution.
What ships:
- apps/worker/src/jobs/embedding-backfill.ts:
- `runEmbeddingBackfillJob({ batchSize, embedding })` — single-cycle
drain. Leases up to batchSize jobs via SELECT FOR UPDATE SKIP LOCKED,
embeds, persists, marks done. Failed jobs go through markJobFailed
which auto-retries up to 3 times (the brain_embedding_jobs CHECK
constraint flips status to 'failed' on the 4th attempt).
- `getWorkerEmbeddingProvider()` — env-driven provider selection that
mirrors `apps/api/src/memory-setup.ts` exactly. Same selection logic
on both sides is load-bearing: if API embeds rows with OpenAI but
worker embeds with hash-trick, cosine across them collapses.
- Returns a structured `EmbeddingBackfillSummary` with attempted /
succeeded / failed / pendingAfter counters that the worker loop
logs on each non-empty cycle.
- apps/worker/src/index.ts: scheduled at 30s intervals alongside the
existing metrics-rollup / changelog-poll / domain-extraction /
federation-sync jobs. SKIP LOCKED makes it safe under multiple worker
instances simultaneously.
Tests: 12 cases in apps/worker/src/__tests__/embedding-backfill.test.ts:
- happy path (drains queue, marks each done, respects batchSize)
- failure handling (embedding throws → markJobFailed; lease throws →
cycle stops cleanly; markJobFailed itself throws → run continues)
- pendingAfter from DB and graceful pending-query failure
- env-driven provider choice (hash default, OpenAI when key set,
OPENAI_EMBEDDING_MODEL override, fallback to OPENAI_API_KEY)
Full suite: 70/70 turbo tasks pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When a user approves or rejects an action via POST /api/approvals/:id/respond,
also persist an Episode into the memory layer. The next time a similar
decision is evaluated, mempalaceRepository.getEpisodes (already wired in
events.ts as part of this PR) pulls that episode into
DecisionContext.episodicMemories. DecisionMaker.calculateEpisodicBoost
(decision-maker.ts:1285+) consumes the episode's `actionTaken` +
`utilityScore` to tilt the candidate score:
- approve → utility 0.9 → next time the same candidate appears it
gets a positive boost on score, making auto-execute more likely.
- reject → utility 0.0 → next time the same candidate's score gets
no boost (and other candidates with non-zero utility from past
approvals leapfrog it).
This closes the loop on the memory architecture: the twin's memory of
what the user *actually decided* feeds back into the next decision,
without any manual preference editing. The previous behaviour was that
approvals only updated the TwinService preferences (which influence
candidate confidence); episodes are a different signal — they record
the SPECIFIC action that won, not just the user's domain-level pref.
Implementation:
- apps/api/src/routes/approvals.ts: after `processFeedback` returns and
before the (optional) execution branch, lookup the originating
decision and call `mempalaceRepository.createEpisode` with the
approval outcome. Wrapped in a try/catch — episode persistence is
best-effort; never blocks the approval response on a memory-layer
hiccup.
- The episode shape carries the full breadcrumb: userId, situationSummary
(from the decision's interpreted summary, with a synthetic fallback),
domain, situationType, actionTaken (from the candidate that the user
approved/rejected), feedbackType, feedbackDetail (the user's reason),
decisionId (so callers can join back), and utilityScore.
Tests: 4 cases in apps/api/src/__tests__/feedback-loop.test.ts:
- approve path → createEpisode called with utility 0.9
- reject path → createEpisode called with utility 0.0 + reason text
- createEpisode throws → approval still returns 200 (best-effort)
- decision row missing interpreted summary → synthetic fallback
Full suite: 70/70 turbo tasks pass; api 535 / worker 83 / memory-gbrain 86.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two improvements that close the remaining loop in the gbrain memory layer:
1. **events.ts now writes inbound signals to gbrain.** Previously, every
inbound event landed in the legacy `signals` / `decisions` tables but
`brain_pages` stayed empty in production — meaning searchSemantic
returned nothing, even with the gbrain backend explicitly selected.
The new `recordSignalToMemory` helper calls
`getMemoryPortForUser(userId).port.recordSignal(...)` on every ingest
(fire-and-forget, so memory hiccups don't block the decision pipeline).
2. **approvals.ts now writes the resulting episode to gbrain too.** The
prior commit added the legacy mempalaceRepository.createEpisode call;
this layer adds a parallel `port.recordEpisode` so the gbrain backend's
semantic index covers approved/rejected outcomes. Future similar
signals' searchSemantic queries surface these episodes directly.
Tests:
- apps/api/src/__tests__/gbrain-write-on-events.test.ts: real Express
round-trip with a stubbed @skytwin/db query layer; asserts that an
inbound /api/events/ingest results in INSERT INTO brain_pages firing
via the MemoryPort path (not just brain_signals).
- packages/decision-engine/src/__tests__/twin-learns-from-corrections.test.ts:
5 cases proving DecisionMaker.calculateEpisodicBoost actually shifts
outcomes when episodicMemories carry feedback:
* baseline (no memory) selects deterministically
* rejection episode does not improve the rejected action's rank
* heavy rejections cannot improve the rejected action's rank
* approval reinforcement keeps the approved winner
* memory only matters when episode.actionTaken matches the candidate
These tests run the REAL DecisionMaker.evaluate against in-memory
TwinService + PolicyEvaluator ports — so the assertions exercise the
exact production scoring code path, not a mock.
Full suite: 70/70 turbo tasks pass; api 536, decision-engine 109.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…egacy ILIKE
Wires the chat assistant's `MemoryContextProvider` to call the user's
selected MemoryPort (`getMemoryPortForUser`) in parallel with the legacy
`mempalaceRepository.searchEpisodes` ILIKE path, dedupes by summary, and
returns the merged top-K. This means chat answers automatically benefit
from gbrain's vector + tsvector RRF retrieval when the gbrain backend
has indexed pages — without losing the cold-install behavior where
mempalace's ILIKE returns recent episodes immediately.
Why both:
- Hot install with gbrain: the semantic side surfaces vector-relevant
pages the ILIKE keyword search would miss (e.g. "what did the CFO
say?" returns CFO threads even when the user didn't type the literal
word "CFO" in their question). Mempalace ILIKE then catches anything
in the legacy table that hasn't been re-indexed yet.
- Cold install: brain_pages is empty so semantic returns []. The
mempalace path serves chat answers without a wait for the worker to
backfill embeddings.
- Both run in parallel; the slower of the two does not gate the chat
response. Per-side errors are caller-swallowed.
The dedupe is by lowercased summary text — same episode often surfaces
from both sources, especially after `recordEpisode` has dual-written it.
Full suite: 70/70 turbo tasks pass; api 536 passing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…he episode
Drives the entire memory-feedback loop through real Express route handlers:
POST /api/events/ingest (board chair email — sender-aware path)
→ flag_for_manual_review candidate, autoExecute=false, approval created
POST /api/approvals/:id/respond (user rejects)
→ mempalaceRepository.createEpisode called with utility 0.0,
feedback_type='reject', action_taken='flag_for_manual_review'
→ episodeStore now has the rejection row
POST /api/events/ingest (similar board email)
→ mempalaceRepository.getEpisodes called, returns the rejection
episode → DecisionContext.episodicMemories carries it →
DecisionMaker.calculateEpisodicBoost weighs it
This proves the wiring intact across all three route handlers and the
DB-backed memory store. The unit-level proof that boost actually shifts
scoring lives in
packages/decision-engine/src/__tests__/twin-learns-from-corrections.test.ts.
Full suite: 70/70 turbo tasks pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the "show me the goods" gap: until now, all the memory infrastructure
was invisible to users. The dashboard surface makes the value visible.
Ships:
1. **`GET /api/memory-config/dashboard`** — operator + user-facing view:
- `index`: total pages, embedded pages, pending embedding jobs
- `episodes.recent[]`: last 10 episodes (summary, action, feedback)
- `episodes.feedbackCounts`: histogram (approve / reject / undo / pending)
- `entities.total`, `topByRecency` (last 10), `topByType` (top 5)
Each query is independently failure-handled via .catch(() => default),
so a partial DB hiccup degrades gracefully rather than 500ing the
whole dashboard.
2. **`apps/web/public/js/pages/memory-settings.js`** — new "What your
twin remembers" card under the existing backend selector:
- Recent decisions table with timestamps, action, feedback badge,
and the situation summary.
- Feedback count strip (✓ approved, ✗ rejected, etc.)
- Top entities by recency + entity-type histogram.
- All three dashboard / config / diagnostics endpoints fetched in
parallel for snappy load.
Tests: 5 new cases in memory-config-routes.test.ts covering:
- 400 on invalid userId
- empty-state shape
- feedback counts aggregated correctly
- top entities sorted by recency, type histogram by count
- partial DB failure → graceful degraded response
Full suite: 70/70 turbo tasks pass; api 541 / 558 (added 5).
This makes the gbrain memory layer's value legible to the user — they
can see entities accumulating, episodes recording approve/reject signals,
embeddings backfilling. The "twin learns" loop is now visible end-to-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ryPort Addresses every finding from Copilot's PR #250 review plus the merge conflict with main. Blockers fixed: 1. **Migration PK types** — brain_pages/brain_entities/brain_triples/ brain_episodes/brain_signals all had `id UUID PRIMARY KEY`. Production signal IDs are not UUIDs (e.g. `sig_gmail_abc123`); they're connector-assigned opaque strings. Forcing UUID would 500 every recordSignal in prod. Changed to `id STRING PRIMARY KEY DEFAULT gen_random_uuid()::STRING`. brain_settings.user_id stays UUID (real FK to users) and brain_embedding_jobs.id stays UUID (internal-only). 2. **StubMempalacePort replaced** — selecting `mempalace` (or relying on hybrid secondary) used to drop all legacy mempalace data on the floor. Now wires a real `MemPalaceMemoryPort` with a proper `MemPalaceRepos` adapter against `mempalaceRepository`. Covers knowledgeGraph (upsertEntity/getEntities/findEntity/addTriple/ queryTriples/invalidateTriple) and episode (createEpisode/getEpisodes/ getEpisodeByDecision/updateEpisode/searchEpisodes). Palace / closet / entityCode methods throw (they're never reached via MemoryPort, but throwing makes any future regression loud). Other bugs Copilot flagged: 3. **`pendingEmbeddingJobs` per-user** — the dashboard was showing the global queue depth instead of the user's. Added optional `userId` parameter; defaults to global for the worker drain telemetry but the API route now passes userId so the dashboard reports the right number in multi-tenant installs. 4. **`candidatePoolSize` computed per-query** — docstring promised `max(k*4, 40)` but constructor hard-coded 40, truncating recall on large-K queries. Store the user override as a sentinel and apply the max-based default in `searchInternal`. 5. **In-memory `embeddingModel` parity** — when `embed()` rejected in the in-memory path, we still set `embeddingModel: this.embedding.model`, leaving pages with non-null model + null embedding. The CRDB path conditionally sets only when embedding succeeded. Matched both paths. 6. **`event.target.closest` guard** — memory-settings click delegator could throw on text-node clicks. Guard with `instanceof Element` per CLAUDE.md frontend event-handling discipline. 7. **`getEntitiesByType` routed through `resolveReadPort`** — was hard-wired to secondary, sending entity reads to the secondary even when the primary (gbrain) could serve them. Added a routing rule defaulting to primary; fallback still kicks in when capability is absent. 8. **docs/memory-swap.md capability table** — claimed `mempalace` had no semantic_search; the real `MemPalaceMemoryPort` declares it (backed by ILIKE). Updated to show `ILIKE` in the cell + a note explaining when to prefer each backend. Plus a rebase onto main (#248 first-run dashboard merged in between). The conflict was in CHANGELOG.md — both entries are now stacked under unreleased. Full suite: 70/70 turbo tasks pass; api 542, decision-engine 109, memory-gbrain 86. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
f68d5f3 to
42283af
Compare
Owner
Author
All Copilot findings addressed (commit
|
| # | Finding | Fix |
|---|---|---|
| 1 | brain_signals.id UUID PRIMARY KEY rejects non-UUID signal ids |
Changed brain_pages/entities/triples/episodes/signals id to STRING PRIMARY KEY DEFAULT gen_random_uuid()::STRING. brain_settings.user_id stays UUID (real FK); brain_embedding_jobs.id stays UUID (internal-only). |
| 2 | StubMempalacePort returned [] everywhere; hybrid secondary lost data |
Real MemPalaceMemoryPort wired with a full MemPalaceRepos adapter against mempalaceRepository. Covers knowledgeGraph + episode surfaces. Palace/closet/entityCode throw (unreachable via MemoryPort). |
| 3 | pendingEmbeddingJobs counted globally on multi-user installs |
Optional userId parameter. Dashboard passes userId; worker passes none (global queue depth is the right number for drain telemetry). |
| 4 | candidatePoolSize doc promised max(k*4, 40) but constructor pinned 40 |
Store override as sentinel; compute max(k*4, 40) per query in searchInternal. |
| 5 | In-memory embeddingModel set even when embedding failed (divergent from CRDB path) |
Match CRDB path: only spread embeddingModel when embedding succeeded. Applied to recordSignal/recordEntity/recordEpisode. |
| 6 | event.target.closest could throw on text-node clicks |
Added instanceof Element guard per CLAUDE.md frontend event-handling rule. |
| 7 | getEntitiesByType bypassed resolveReadPort |
Added getEntitiesByType to RoutingRules (defaults to primary); route through resolveReadPort with temporal_triples capability check. |
| 8 | docs/memory-swap.md said mempalace had no semantic_search |
Updated capability table to show ILIKE with a note explaining the keyword-vs-semantic distinction. |
Also resolved the merge conflict with main (#248 first-run dashboard merged between this PR's commits). Both CHANGELOG entries are stacked under unreleased.
Full suite: 70/70 turbo tasks pass.
…tecture, technical-spec Post-ship documentation update for the gbrain memory backend ship. - docs/architecture-philosophy.md: memory port row updated to reflect gbrain (default, CRDB-native) + mempalace (selectable fallback). The "interim" framing was obsolete — gbrain is the default. - docs/cockroach-architecture.md: added the 7 brain_* tables to the schema reference. Documented the STRING-PK choice (production signal ids aren't UUIDs; the table reflects that contract). - docs/technical-spec.md: package layout shows the 5 new memory-* packages. Build dependency chain updated to include them in topological order. Full suite: 70/70 turbo tasks pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…arness - SSE: events.ts emits `memory:page-indexed` after a successful `recordSignalToMemory`; approvals.ts emits `memory:episode-recorded` after `mempalaceRepository.createEpisode`. Web sse-client.js subscribes and dispatches `sse:memory:*` CustomEvents; memory-settings.js wires a module-singleton listener (1s debounce) that re-renders the dashboard without polling. - Ollama recipe in docs/memory-swap.md — zero-cloud local embeddings via the OpenAI-compatible /v1/embeddings endpoint (nomic-embed-text default). - CRDB integration harness: packages/memory-gbrain-crdb-adapter ships `scripts/run-crdb-integration.sh` (Docker-based) and a `test:crdb` package script. Spins a hermetic CRDB, applies migration 040, seeds a test user, and runs the 6 DB-gated integration tests. - Tests: feedback-loop.test.ts now mocks `createEpisode` resolved value so the SSE emit path is reachable, plus an assertion that `memory:episode-recorded` is emitted. gbrain-write-on-events.test.ts gains a parallel assertion for `memory:page-indexed`. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The bullet on line 24-25 still said mempalace "declares no semantic_search capability" — that was true at the start of #197 but MemPalaceMemoryPort.capabilities() now returns 'semantic_search' (ILIKE-backed). The table below already reflected this; the prose did not. Tightens the wording to match. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three issues found by the /review pass on commits aa78f13..HEAD: 1. Migration was applied twice. The first apply ran against `skytwin_test` *before* the inlined `users` table existed there, so the brain_* FK references failed silently (psql -f exits 0 on per-statement errors without ON_ERROR_STOP). The second apply then worked because the tables already partially existed. Reordered to create-db → create-users-in-test-db → apply-migration-once. 2. Added `-v ON_ERROR_STOP=on` to every psql invocation so any future schema regression fails the harness loudly instead of being masked by `>/dev/null`. 3. The cockroach-ready wait loop completed silently after 30s even on total startup failure; now sets a `ready` flag and bails with the container's last 20 log lines if the DB never accepts connections. Also tightened TEST_USER_ID parsing: `-A` unaligned output + tr against `[:space:]` instead of just ` \n`, plus an empty-result check. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ELOG entry The "Embedded LLM downloader: round-3 review fixes (#187 AC#2 follow-up)" entry landed on main via PR #249 (commit c6e93de) after this branch last rebased. The branch did not pull it in, so squash-merging would have deleted it from main as a side effect. Restored verbatim from origin/main:CHANGELOG.md so the squash diff is purely additive. Caught by the /document-release cross-doc consistency pass — exactly the kind of silent regression that motivated adding the pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Comment on lines
+163
to
+167
| CREATE TABLE IF NOT EXISTS brain_embedding_jobs ( | ||
| id UUID PRIMARY KEY DEFAULT gen_random_uuid(), | ||
| user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE, | ||
| page_id UUID NOT NULL REFERENCES brain_pages(id) ON DELETE CASCADE, | ||
| status STRING NOT NULL DEFAULT 'pending' |
Owner
Author
There was a problem hiding this comment.
Addressed in 22e5689. brain_embedding_jobs.page_id is now STRING to match brain_pages.id exactly; the FK and inserts with non-UUID page ids both work now. Migration 040 line ~169 carries an inline comment explaining the earlier-draft history.
Comment on lines
+195
to
+199
| // Dedupe by summary text — same episode may surface from both sources. | ||
| const seen = new Set<string>(); | ||
| const merged: MergedHit[] = []; | ||
| for (const item of [...fromSemantic, ...fromMempalace]) { | ||
| const key = item.summary.trim().toLowerCase(); |
Owner
Author
There was a problem hiding this comment.
Addressed in 22e5689 by fixing the comment, not the code. gbrain hits never carry an occurredAt (it's undefined); mempalace hits always do (ISO string). Putting occurredAt in the dedupe key would defeat cross-source dedupe entirely — every gbrain/mempalace pair with the same summary would survive because undefined !== 'YYYY-...'. Updated assistant.ts:152-157 to explain this.
Comment on lines
243
to
250
| * Priority (in order): | ||
| * 1. Explicit routing override for the method name. If the override names | ||
| * the primary but the primary lacks the capability, fall through to | ||
| * the secondary anyway — never silently route to a port that can't | ||
| * serve the call. | ||
| * 2. Default routing table preference. | ||
| * 3. Final fallback: whichever port declares the capability. | ||
| * |
Owner
Author
There was a problem hiding this comment.
Addressed in 22e5689 by rewriting the docstring to match the code. The original 3-step description was aspirational — steps 1 and 2 were the same thing (the routing table IS the override), and step 3's capability fallback is already what step 2 does (if the preferred port lacks the capability we fall through). New docstring at hybrid-port.ts:240-256 describes the actual single-rule + capability-fallback logic.
| @@ -1 +1 @@ | |||
| {"sessionId":"f8bc64fc-9881-4489-9cbf-496a51fce7c6","pid":72866,"procStart":"Sat May 9 02:11:52 2026","acquiredAt":1778298894623} No newline at end of file | |||
| {"sessionId":"9208cd9e-7b9c-4bb4-99c2-65ad7f2ee4c5","pid":88031,"procStart":"Mon May 11 03:02:19 2026","acquiredAt":1778470430217} No newline at end of file | |||
Owner
Author
There was a problem hiding this comment.
Addressed in 22e5689. git rm --cached .claude/scheduled_tasks.lock to untrack it, plus added \.claude/scheduled_tasks.lock to .gitignore so future sessions don't re-stage it. The file remains on disk locally — only the git tracking is removed. Technically this is a removal-from-main but is the right long-term shape for a per-session PID lock.
Comment on lines
+594
to
+601
| function parsePageRow(row: BrainPageRow): BrainPageRow { | ||
| // pg returns FLOAT8[] as a string like "{0.1,0.2}" only if no parser is | ||
| // installed; with the default parser it's already number[] — but defensive | ||
| // parsing is cheap. | ||
| if (typeof row.embedding === 'string') { | ||
| row = { ...row, embedding: parsePgArray(row.embedding) }; | ||
| } | ||
| return { |
Owner
Author
There was a problem hiding this comment.
Addressed in 22e5689. Introduced a RawBrainPageRow type at repository.ts (embedding: number[] | string | null) for the raw pg row shape; parsePageRow now takes RawBrainPageRow and returns BrainPageRow with the narrowed number[] | null embedding. The typeof === 'string' branch is now type-valid, downstream consumers see the parsed shape, and there's no widening to the public BrainPageRow type.
Five findings from Copilot's re-review on commit 50f297c: 1. **brain_embedding_jobs.page_id FK type mismatch (HIGH)** — column was declared `UUID` but `brain_pages.id` is `STRING`. CRDB would reject the FK at apply time, or accept it and reject any insert with a non-UUID page id (which is most signal-derived pages — `sig_gmail_abc123` etc.). Fixed in migration 040 to `page_id STRING` matching `brain_pages.id` exactly. 2. **assistant.ts dedupe comment was wrong** — outer comment said "dedupe by (summary, occurredAt)" but the implementation uses just summary. Updated the comment to reflect the actual logic and explain WHY occurredAt can't be in the key (gbrain hits never carry one; including it would defeat cross-source dedupe entirely). 3. **hybrid-port.ts resolveReadPort docstring drift** — claimed a 3-step priority (override → routing table → capability fallback) but the implementation collapses steps 1+2 (the override IS the routing table) and step 3 is the same as the capability fallback inside step 2. Rewrote the docstring to match the actual logic. 4. **.claude/scheduled_tasks.lock leaked into the PR** — runtime session lock metadata (sessionId / pid / ts) was getting committed on every session. `git rm --cached` to untrack, added to .gitignore so future sessions don't re-add it. This is technically a removal-from-main but is the right long-term shape. 5. **BrainPageRow.embedding type / parsePageRow runtime mismatch** — types.ts declared `number[] | null` but parsePageRow defensively checks `typeof === 'string'` for the pg array-literal case, which strict mode flagged as always-false. Introduced a `RawBrainPageRow` type with `embedding: number[] | string | null` for the raw DB shape, and parsePageRow narrows it to `BrainPageRow` (with `number[] | null`) for downstream consumers. No behaviour change. Verified: pnpm --filter @skytwin/api test → 544 pass; memory-* tests → 155 pass. No regressions. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This was referenced May 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #197.
Promotes
@skytwin/memory-gbrainfrom a CLI-shellout skeleton (PR #215) to a real, in-process, CockroachDB-backed memory layer. The defaultMemoryPortfor new installs is now gbrain — vector embeddings + tsvector full-text search, fused via Reciprocal Rank Fusion. No separate Postgres process, no external CLI install — gbrain runs against the SkyTwin DB stack directly.User direction was explicit: gbrain is the default, mempalace is the second option, and everything works against CRDB where possible. Done.
What's in the PR
New package:
@skytwin/memory-gbrain-crdb-adapterrepository.ts— CRDB-backed CRUD +hybridSearch(parallel text + vector, application-side RRF fold). Per-userpendingEmbeddingJobs(userId?).in-memory-repository.ts— same surface, in-process; used by tests.embedding.ts—EmbeddingProviderinterface +HashEmbeddingProvider(deterministic, hash-trick, zero-config) andOpenAiEmbeddingProvider(OpenAI-compatible — works with Ollama, llamafile, vLLM).rrf.ts— Reciprocal Rank Fusion fold (k=60).Migration
040-gbrain-memory.sqlbrain_pages(FLOAT8[] + TSVECTOR + INVERTED INDEX),brain_entities,brain_triples,brain_episodes,brain_signals,brain_settings,brain_embedding_jobs(durable queue withSELECT FOR UPDATE SKIP LOCKED).idisSTRING PRIMARY KEY DEFAULT gen_random_uuid()::STRING(not UUID) — production signal IDs likesig_gmail_abcneed to round-trip.Updated
@skytwin/memory-gbrainEmbeddedGbrainMemoryPort— fullMemoryPortimplementation. Capabilities:semantic_search,code_aware_search,temporal_triples,episodic,graph_walk. Synchronous embedding when fast, async via the job queue when an external provider is configured.candidatePoolSizedefaults tomax(k*4, 40)per query (was pinned 40).searchCodeAware— 1.25× boost forsource = 'code'pages.hasExternalGbrainConfig()— detects existing~/.config/gbrain/for the dashboard prompt.Updated
@skytwin/memory-hybridHybridDiagnosticscounters expose routing + write outcomes.resolveReadPortfalls through to secondary when primary lacks the relevant capability.getEntitiesByTypenow routes throughresolveReadPort(was hard-wired secondary).API + Web
apps/api/src/memory-setup.ts— per-user backend factory. Defaultgbrain;MEMORY_BACKENDenv override; per-userbrain_settings.backendwins. Themempalaceandhybrid-secondary paths now wire a realMemPalaceMemoryPortagainstmempalaceRepository(previously a stub that returned[]).apps/api/src/routes/memory-config.ts—GET/POST /api/memory-config,POST /dismiss-notification,GET /diagnostics,GET /dashboard.apps/web/public/js/pages/memory-settings.js— backend switcher + capability list + "Your twin just got smarter" notice + memory dashboard (recent episodes, top entities, feedback histogram).Decision pipeline wiring
SenderAwareCandidateGenerator— closes the rule-based safety gap surfaced by the fake-user E2E: board / CFO / legal / investor senders get aflag_for_manual_reviewcandidate that suppresses the rule-basedarchive_emailset entirely. The built-inNO_IRREVERSIBLE_WITHOUT_APPROVALpolicy gates it to the approval queue at every trust tier.events.ts— wires this generator as both the standalone rule-based DecisionMaker and the innerRuleBasedCandidateGeneratorof the LLM fallback chain. PopulatesDecisionContext.episodicMemoriesfrommempalaceRepository.getEpisodesso the existingcalculateEpisodicBoostactually fires. Best-effort writes the inbound signal intoMemoryPort.recordSignal.approvals.ts— on approve/reject, records an episode into both the legacy table AND the gbrain MemoryPort.assistant.ts— chat context provider queriesMemoryPort.searchSemanticin parallel with mempalace's ILIKE; results deduped.Embedding worker
apps/worker/src/jobs/embedding-backfill.ts— drainsbrain_embedding_jobsevery 30s. SELECT FOR UPDATE SKIP LOCKED makes it safe under multiple worker instances. Env-driven provider selection mirrorsapps/api/src/memory-setup.ts.Documentation
docs/memory-swap.md— full runbook, capability table, env knobs, rollback.docs/architecture-philosophy.md— memory port row updated.docs/cockroach-architecture.md— 7 brain_* tables documented + the STRING-PK rationale.docs/technical-spec.md— 5 new memory-* packages in the layout + topological dependency chain.CHANGELOG.md— substantial unreleased entry.CLAUDE.md— package table refreshed.Tests
@skytwin/memory-gbrain@skytwin/memory-gbrain-crdb-adapter@skytwin/memory-hybrid@skytwin/decision-engine@skytwin/worker@skytwin/apiFull suite: 70/70 turbo tasks pass (api 542, decision-engine 109, memory-gbrain 86, memory-gbrain-crdb-adapter 50, memory-hybrid 19, worker 83).
Post-/review fixes (Copilot)
Round 1 (commit 42283af):
StubMempalacePortreturned[]everywhereMemPalaceMemoryPortwired tomempalaceRepositorypendingEmbeddingJobsglobal, not per-useruserId; dashboard passes itcandidatePoolSizehard-coded 40 vs doc'smax(k*4, 40)embeddingModelmismatch with CRDB pathevent.target.closestcould throw on text-node clicksinstanceof ElementguardgetEntitiesByTypebypassedresolveReadPortsemantic_searchRound 2 (commit 22e5689) — Copilot re-reviewed on 50f297c:
brain_embedding_jobs.page_id UUIDvsbrain_pages.id STRINGFK type mismatch — inserts of non-UUID page ids would failpage_id STRINGmatchesbrain_pages.id(summary, occurredAt)but code dedupes by summary onlyresolveReadPortdocstring claimed 3-step priority but code has 2.claude/scheduled_tasks.lock(volatile session lock) leaked into the PRgit rm --cached+ added to .gitignoreBrainPageRow.embedding: number[] | nullmismatched parsePageRow'stypeof === 'string'runtime check (strict-mode always-false flag)RawBrainPageRowtype withnumber[] | string | nullfor the raw row;parsePageRownarrows toBrainPageRowOperational notes
MEMORY_BACKEND=gbrain(default) |hybrid|mempalaceswitches the per-install default; per-user override via dashboard.OPENAI_EMBEDDING_API_KEY(orOPENAI_API_KEY) → real embeddings.OPENAI_EMBEDDING_BASE_URLpoints at any OpenAI-compatible endpoint (Ollama, llamafile, vLLM, LocalAI). Seedocs/memory-swap.mdfor the Ollama recipe.MEMORY_BACKEND=mempalace. Legacy/api/mempalaceroutes unaffected — they querymemory_*tables directly, not via MemoryPort.Polish (commits aa78f13 → 50f297c)
After Copilot review fixes landed, three follow-up items shipped on top:
events.tsemitsmemory:page-indexedafterrecordSignalToMemory;approvals.tsemitsmemory:episode-recordedaftercreateEpisode. Web client subscribes and re-renders the dashboard with a 1s debounce. No polling. Unit-tested infeedback-loop.test.tsandgbrain-write-on-events.test.ts.docs/memory-swap.mdnow documents a zero-cloud local-embeddings setup via Ollama's OpenAI-compatible/v1/embeddingsendpoint (nomic-embed-textby default).packages/memory-gbrain-crdb-adapter/scripts/run-crdb-integration.sh+ atest:crdbpackage script. Spins a hermetic Docker CRDB, applies migration 040, seeds a test user, runs the 6 DB-gated tests, tears down on exit. Post-/review pass hardened the script:ON_ERROR_STOP=onon every psql call, ready-flag wait loop with container logs on timeout, robust TEST_USER_ID parsing.Documentation
/document-releaseran post-merge-gate against the final pre-merge state:README.mdCLAUDE.mdCHANGELOG.md[unreleased]entry that an earlier rebase silently dropped (PR #249 commitc6e93de); squash-merge would have deleted it from main. See commit 50f297c.docs/architecture-philosophy.mddocs/cockroach-architecture.mddocs/technical-spec.mddocs/memory-swap.mdTODOS.mdVERSIONTest plan
pnpm build --concurrency=1→ 35/35 packagespnpm test→ 70/70 turbo tasks (544 API tests pass, 24 skipped — DB-gated)/reviewre-run on post-Copilot diffs → onlyrun-crdb-integration.shissues found; auto-fixed in aa93c33/document-releasere-run → CHANGELOG drift caught and restored in 50f297cpnpm --filter @skytwin/memory-gbrain-crdb-adapter test:crdb)🤖 Generated with Claude Code