Skip to content

feat: lcm_recent — temporal rollup layer for recency awareness#4

Closed
100yenadmin wants to merge 2 commits into
mainfrom
feat/lcm-recent
Closed

feat: lcm_recent — temporal rollup layer for recency awareness#4
100yenadmin wants to merge 2 commits into
mainfrom
feat/lcm-recent

Conversation

@100yenadmin

@100yenadmin 100yenadmin commented Apr 13, 2026

Copy link
Copy Markdown
Member

Summary

Adds a daily/weekly/monthly rollup system on top of the existing LCM summary DAG.

6 files changed, 1,612 additions, 0 deletions.

New tool: lcm_recent

  • lcm_recent(period="today") → structured daily recap
  • lcm_recent(period="yesterday") → prior day rollup
  • lcm_recent(period="7d") → last 7 days
  • lcm_recent(period="date:2026-04-11") → specific date

Architecture

  • Schema: lcm_rollups, lcm_rollup_sources, lcm_rollup_state (additive only, no existing table changes)
  • RollupStore: UPSERT semantics (ON CONFLICT DO UPDATE)
  • RollupBuilder: Timezone-aware, fingerprinted idempotent rebuilds, atomic transactions
  • lcm_recent tool: Period-based recall with timestamp-bounded fallback

Safety

  • All writes: ON CONFLICT DO UPDATE (not INSERT OR REPLACE)
  • Rebuilds: BEGIN IMMEDIATE transactions
  • Migration: before FTS5 guard (FTS5-independent)
  • Partial index: summaries(conversation_id, kind) WHERE kind='leaf'
  • Adversarial review: R-463 (10 findings, all addressed)
  • Tested against 1.9GB production LCM database
  • Scenario validation: R-465 (15 use cases)

Open with Devin

Summary by CodeRabbit

  • New Features
    • Timezone-aware daily, weekly, and monthly conversation rollups that summarize activity periods.
    • New "Recent activity" tool to fetch formatted reports for today, yesterday, 7d, week, month, 30d, or specific dates.
    • Rollup caching and prebuilding with incremental rebuilds to reduce repeated work and speed retrievals.
    • Fallback to direct summary search when rollups are unavailable, ensuring consistent results.

Adds a daily/weekly/monthly rollup system on top of the existing LCM
summary DAG. Pre-built daily rollups give fast answers to temporal
questions ('what did we do today?', 'catch me up on yesterday') without
requiring keyword search or LLM calls.

Architecture:
- Schema: lcm_rollups, lcm_rollup_sources, lcm_rollup_state tables
  (additive, no existing table modifications)
- RollupStore: data access layer with proper UPSERT semantics
- RollupBuilder: timezone-aware daily builder with fingerprinted
  idempotent rebuilds, keyword-based outcome extraction
- lcm_recent tool: period-based temporal recall with grep fallback

Safety:
- All writes use ON CONFLICT DO UPDATE (not INSERT OR REPLACE)
- Rebuilds wrapped in BEGIN IMMEDIATE transactions
- Rollup tables created before FTS5 guard (FTS5-independent)
- New partial index on summaries(conversation_id, kind) WHERE kind='leaf'

Tested against 1.9GB production LCM database. Adversarial review
completed (R-463). Scenario validation against 15 use cases (R-465).
Copilot AI review requested due to automatic review settings April 13, 2026 12:00
@coderabbitai

coderabbitai Bot commented Apr 13, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

Adds rollup persistence, builder, and retrieval: DB migrations for rollup tables/indexes/views; a SQLite-backed RollupStore; a timezone-aware RollupBuilder for daily rollups with fingerprint-based deduplication and token budgeting; a createLcmRecentTool that serves recent activity from rollups (or falls back to live summaries); plugin registers the new tool.

Changes

Cohort / File(s) Summary
Database Migrations
src/db/migration.ts
Replaced bulk table-name query with iterative checks; added migration steps to create lcm_rollups, lcm_rollup_sources, lcm_rollup_state, supporting indexes (including partial index on summaries(kind='leaf')) and period-scoped views (daily_rollups, weekly_rollups, monthly_rollups); run before FTS5 early-return.
Storage Layer
src/store/rollup-store.ts, src/store/index.ts
New RollupStore with types (RollupRow, RollupStateRow, RollupSourceRow, RollupSourceInput, LeafSummaryForDayRow) and methods for upsert/query/list/delete/mark-stale rollups, replace/get rollup sources, get/upsert per-conversation state, and getLeafSummariesForDay; re-exported from src/store/index.ts.
Rollup Builder
src/rollup-builder.ts
New RollupBuilder and config/result types: iterates days (timezone-aware), computes fingerprints from ordered leaf summary IDs + token totals, skips unchanged days, builds/updates rollups inside BEGIN IMMEDIATE transactions, enforces token budgets by pruning timeline entries, and records build outcomes and per-day errors.
Tool
src/tools/lcm-recent-tool.ts
New createLcmRecentTool that accepts period specifiers (today, yesterday, 7d, week, month, 30d, date:YYYY-MM-DD) and returns structured reports by: 1) loading rollup(s) from RollupStore and combining them (deduping sources, aggregating tokens/status), or 2) falling back to time-bounded leaf summary queries; returns content + details metadata.
Plugin Integration
src/plugin/index.ts
Registers createLcmRecentTool alongside existing LCM tools using same deps/getLcm/session scoping pattern.
New public surface
src/rollup-builder.ts, src/store/rollup-store.ts
Adds exported interfaces and classes: RollupBuilderConfig, BuildResult, RollupBuilder, computeFingerprint, getLocalDateKey, getLocalDayBounds, and RollupStore with its public API.

Sequence Diagram

sequenceDiagram
    participant Client as Client
    participant Tool as lcm-recent-tool
    participant Store as RollupStore
    participant DB as Database
    participant Builder as RollupBuilder
    participant LCM as LCM_Retrieval

    Client->>Tool: Request period (e.g., "today")
    Tool->>Tool: Parse period → UTC bounds
    alt Use Prebuilt Rollups
        Tool->>Store: Query rollups for period
        Store->>DB: SELECT from lcm_rollups / lcm_rollup_sources
        DB-->>Store: Rollup rows
        Store-->>Tool: Rollup content + status
        Tool-->>Client: Formatted rollup response
    else Fallback to Live Summaries
        Tool->>LCM: time-bounded grep/retrieval
        LCM->>DB: (FTS query) SELECT summaries
        DB-->>LCM: Summary rows
        LCM-->>Tool: Snippets
        Tool-->>Client: Fallback formatted response
    end
    alt Rebuild requested
        Client->>Builder: buildDailyRollups(conversationId)
        Builder->>Store: getState / getLeafSummariesForDay
        Store->>DB: SELECT summaries, state
        DB-->>Store: Data rows
        loop For each day needing rebuild
            Builder->>Builder: computeFingerprint
            alt Fingerprint changed
                Builder->>DB: BEGIN IMMEDIATE
                Builder->>Store: upsertRollup / replaceRollupSources / upsertState
                Store->>DB: INSERT/UPDATE lcm_rollups / lcm_rollup_sources / lcm_rollup_state
                DB-->>Store: OK / COMMIT
            else Skip
            end
        end
        Builder-->>Client: BuildResult {built, skipped, errors}
    end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 10.53% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly reflects the main change: a new temporal rollup layer for the LCM system that enables recency-aware queries via the lcm_recent tool.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/lcm-recent

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 9

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/db/migration.ts`:
- Around line 888-910: The lcm_rollups table is missing a foreign key to
conversations(conversation_id); add a foreign key constraint on conversation_id
(e.g., FOREIGN KEY(conversation_id) REFERENCES conversations(conversation_id) ON
DELETE CASCADE) in the CREATE TABLE for lcm_rollups to ensure rollups are
removed when a conversation is deleted; if you cannot alter the existing table
in-place (SQLite limitation), implement the change by creating a new table with
the FK, copying data, dropping the old table and renaming the new one, or
alternatively add a periodic GC routine to delete orphaned rows from lcm_rollups
by joining against conversations.

In `@src/rollup-builder.ts`:
- Around line 507-513: estimateTokens currently uses a simple heuristic
(Math.ceil(text.length / 4)) which misestimates for CJK and code; update the
implementation or surrounding docs: either replace estimateTokens with a real
tokenizer call (e.g., use the project's tokenizer utility or an external
tokenizer) and use that in all places that call estimateTokens, or add an inline
comment / JSDoc on the estimateTokens function explaining the ~4 chars/token
heuristic and its limitations for CJK and code so callers know the inaccuracy;
reference the estimateTokens function to locate and change the behavior or add
the documentation.
- Around line 182-254: The call to this.store.replaceRollupSources(...) inside
the withDatabaseTransaction block is not awaited, so the replacement may not
finish before the transaction commits; update the call in rollup-builder.ts to
await this.store.replaceRollupSources(...) (after rollup-store.ts has been fixed
to return a Promise) so the source-replace runs inside and completes within the
surrounding withDatabaseTransaction; ensure you await replaceRollupSources in
the same transaction before calling upsertRollup/upsertState to preserve
transactional guarantees.
- Around line 191-246: The two upsertRollup calls (the one setting status
"building" and the one setting status "ready") are redundant inside the same
transaction; remove the first upsertRollup block (the "building" status) and
keep a single upsertRollup that writes the final state ("ready") after
replaceRollupSources, ensuring fields like rollup_id, conversation_id,
period_kind, period_key, period_start, period_end, timezone, content
(draft.content), token_count (draft.summaryTokenCount), source_summary_ids
(JSON.stringify(summaries.map(...))), source_message_count, source_token_count
(totalSourceTokens), coverage_start/coverage_end (derived from summaries),
summarizer_model and source_fingerprint are preserved; if you intended the
"building" state to be externally visible, instead persist that upsert outside
the transaction so the intermediate status can be observed.

In `@src/store/rollup-store.ts`:
- Around line 323-389: The upsertState function issues multiple UPDATEs; change
it to perform the INSERT as-is then, if any fields in Partial<RollupStateRow>
are provided, build a single dynamic UPDATE: construct an array of set clauses
starting with "updated_at = datetime('now')", push "field = ?" for each defined
property (timezone, last_message_at, last_rollup_check_at, last_daily_build_at,
pending_rebuild) and collect their values, then run one prepared statement
`UPDATE lcm_rollup_state SET ${setClauses.join(", ")} WHERE conversation_id = ?`
with ...values, conversationId; keep the initial INSERT/ON CONFLICT as-is and
only execute the UPDATE when values.length > 0.
- Around line 62-106: The upsert in upsertRollup currently overwrites the
existing rollup_id on conflict (in the lcm_rollups INSERT ... ON CONFLICT DO
UPDATE clause), which can break the foreign key from lcm_rollup_sources; fix by
removing the assignment rollup_id = excluded.rollup_id from the UPDATE list so
existing rollup_id is preserved on conflicts (alternatively ensure callers
always pass the existing rollup_id like buildDayRollup or change the
lcm_rollup_sources FK to use ON UPDATE CASCADE if you intend to permit id
changes).
- Around line 274-291: The replaceRollupSources function fires a transaction
with void withDatabaseTransaction(...) which discards the Promise causing race
conditions and silent failures; make replaceRollupSources return a Promise and
await the transaction (change signature to async and remove the void) so the
DELETE/INSERT complete before callers proceed, and then update call sites such
as buildDayRollup in src/rollup-builder.ts to await replaceRollupSources to
ensure the transaction is completed before the outer transaction commits.

In `@src/tools/lcm-recent-tool.ts`:
- Around line 99-105: getLcmDatabase currently reaches into an internal property
via a type assertion (candidate.db) which couples callers to LcmContextEngine's
implementation; add a public accessor on LcmContextEngine (e.g., getDatabase():
DatabaseSync | undefined or throw) and update getLcmDatabase to call that
accessor (handle undefined by throwing the same error) so callers use the stable
API instead of candidate.db, or alternatively document this coupling on
LcmContextEngine if adding a getter is not possible.
- Around line 347-352: The calls to parseIsoTimestampParam for "since" and
"before" in lcm-recent-tool.ts are dead: they parse values and discard results
while the tool uses a period-based approach; either remove these unused calls
(delete the try/catch block invoking parseIsoTimestampParam) or wire the parsed
values into the tool logic as overrides (use the returned parsed timestamps from
parseIsoTimestampParam("since") and parseIsoTimestampParam("before") to override
the period-derived since/before values where present, and propagate them through
the existing functions that compute the time window). Ensure you update or
remove the surrounding try/catch accordingly and keep the tool behavior
consistent with period-based defaults when parsed params are absent.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 7d37559e-e266-49c2-872b-8e4b69144b01

📥 Commits

Reviewing files that changed from the base of the PR and between 6e9388c and 13a964c.

📒 Files selected for processing (6)
  • src/db/migration.ts
  • src/plugin/index.ts
  • src/rollup-builder.ts
  • src/store/index.ts
  • src/store/rollup-store.ts
  • src/tools/lcm-recent-tool.ts
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Agent
🔇 Additional comments (8)
src/store/index.ts (1)

40-48: LGTM!

Clean addition following the existing re-export pattern. The rollup-layer primitives are correctly exported for use by other modules.

src/plugin/index.ts (1)

1877-1879: LGTM!

The lcm_recent tool registration follows the same dependency injection pattern as the existing LCM tools. The rollupStore is intentionally omitted since the tool lazily initializes it from the database connection when not provided.

src/db/migration.ts (1)

941-950: LGTM!

The convenience views correctly filter rollups by period_kind and provide a clean interface for querying daily, weekly, and monthly rollups.

src/tools/lcm-recent-tool.ts (2)

354-410: LGTM!

The allConversations fallback branch correctly uses retrieval.grep with time bounds from the resolved period. The output format includes helpful status and drill-down guidance.


416-461: LGTM!

The rollup retrieval logic handles both single-period lookup (with periodKey) and multi-period aggregation (for "7d", "30d" ranges) correctly. The combineRollups function properly aggregates content, deduplicates source IDs, and conservatively reports status as "stale" if any source rollup is stale.

src/store/rollup-store.ts (1)

391-420: LGTM!

The query efficiently leverages the partial index on (conversation_id, kind) WHERE kind = 'leaf' and correctly uses coalesce(latest_at, earliest_at, created_at) for temporal filtering consistent with other queries in the codebase.

src/rollup-builder.ts (2)

274-277: LGTM!

The fingerprint computation is deterministic (sorted IDs, consistent format) and includes token count to detect structural changes. The 64-bit truncation is sufficient for collision resistance in this context.


71-151: LGTM!

The buildDailyRollups orchestration correctly:

  • Short-circuits when pending_rebuild=0 (no new activity)
  • Skips today unless forceCurrentDay is set
  • Uses fingerprint comparison to avoid redundant rebuilds
  • Accumulates errors per-day without failing the entire batch

Comment thread src/db/migration.ts
Comment on lines +888 to +910
CREATE TABLE IF NOT EXISTS lcm_rollups (
rollup_id TEXT PRIMARY KEY,
conversation_id INTEGER NOT NULL,
period_kind TEXT NOT NULL CHECK (period_kind IN ('day', 'week', 'month')),
period_key TEXT NOT NULL,
period_start TEXT NOT NULL,
period_end TEXT NOT NULL,
timezone TEXT NOT NULL DEFAULT 'UTC',
content TEXT NOT NULL,
token_count INTEGER NOT NULL DEFAULT 0,
source_summary_ids TEXT NOT NULL DEFAULT '[]',
source_message_count INTEGER NOT NULL DEFAULT 0,
source_token_count INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'ready' CHECK (status IN ('building', 'ready', 'stale', 'failed')),
coverage_start TEXT,
coverage_end TEXT,
summarizer_model TEXT,
source_fingerprint TEXT,
built_at TEXT NOT NULL DEFAULT (datetime('now')),
invalidated_at TEXT,
error_text TEXT,
UNIQUE (conversation_id, period_kind, period_key)
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Consider adding a foreign key constraint on conversation_id.

The lcm_rollups table lacks a foreign key to conversations(conversation_id). This means:

  • Rollups won't be cascade-deleted when conversations are deleted.
  • Orphaned rollup data may accumulate over time.

If this is intentional for FTS5-independence or to avoid schema changes to existing tables, consider adding a cleanup mechanism or periodic GC for orphaned rollups.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/db/migration.ts` around lines 888 - 910, The lcm_rollups table is missing
a foreign key to conversations(conversation_id); add a foreign key constraint on
conversation_id (e.g., FOREIGN KEY(conversation_id) REFERENCES
conversations(conversation_id) ON DELETE CASCADE) in the CREATE TABLE for
lcm_rollups to ensure rollups are removed when a conversation is deleted; if you
cannot alter the existing table in-place (SQLite limitation), implement the
change by creating a new table with the FK, copying data, dropping the old table
and renaming the new one, or alternatively add a periodic GC routine to delete
orphaned rows from lcm_rollups by joining against conversations.

Comment thread src/rollup-builder.ts
Comment on lines +182 to +254
await withDatabaseTransaction(this.store.db, "BEGIN IMMEDIATE", () => {
if (
existing?.rollup_id
&& existing.source_fingerprint
&& existing.source_fingerprint !== fingerprint
) {
this.store.markStale(existing.rollup_id);
}

this.store.upsertRollup({
rollup_id: rollupId,
conversation_id: conversationId,
period_kind: PERIOD_KIND,
period_key: dateKey,
period_start: start.toISOString(),
period_end: end.toISOString(),
timezone: this.config.timezone,
content: draft.content,
token_count: draft.summaryTokenCount,
source_summary_ids: JSON.stringify(summaries.map((summary) => summary.summaryId)),
source_message_count: 0,
source_token_count: totalSourceTokens,
status: "building",
coverage_start:
summaries[0]?.earliestAt?.toISOString() ?? summaries[0]?.createdAt.toISOString() ?? null,
coverage_end:
summaries[summaries.length - 1]?.latestAt?.toISOString()
?? summaries[summaries.length - 1]?.createdAt.toISOString()
?? null,
summarizer_model: "concatenation-v1",
source_fingerprint: fingerprint,
});

this.store.replaceRollupSources(
rollupId,
summaries.map((summary, index) => ({
type: "summary",
id: summary.summaryId,
ordinal: index,
})),
);

this.store.upsertRollup({
rollup_id: rollupId,
conversation_id: conversationId,
period_kind: PERIOD_KIND,
period_key: dateKey,
period_start: start.toISOString(),
period_end: end.toISOString(),
timezone: this.config.timezone,
content: draft.content,
token_count: draft.summaryTokenCount,
source_summary_ids: JSON.stringify(summaries.map((summary) => summary.summaryId)),
source_message_count: 0,
source_token_count: totalSourceTokens,
status: "ready",
coverage_start:
summaries[0]?.earliestAt?.toISOString() ?? summaries[0]?.createdAt.toISOString() ?? null,
coverage_end:
summaries[summaries.length - 1]?.latestAt?.toISOString()
?? summaries[summaries.length - 1]?.createdAt.toISOString()
?? null,
summarizer_model: "concatenation-v1",
source_fingerprint: fingerprint,
});

this.store.upsertState(conversationId, {
timezone: this.config.timezone,
last_daily_build_at: builtAt.toISOString(),
last_rollup_check_at: builtAt.toISOString(),
pending_rebuild: 0,
});
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Non-awaited replaceRollupSources call inside transaction.

Line 215-222 calls this.store.replaceRollupSources(...) without await. Due to the bug in replaceRollupSources using void withDatabaseTransaction(...), the source replacement may not complete before the outer transaction commits.

This ties to the critical issue in rollup-store.ts. Once that's fixed to return a Promise, this call must be awaited:

Required fix after rollup-store.ts is updated
-      this.store.replaceRollupSources(
+      await this.store.replaceRollupSources(
         rollupId,
         summaries.map((summary, index) => ({
           type: "summary",
           id: summary.summaryId,
           ordinal: index,
         })),
       );
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/rollup-builder.ts` around lines 182 - 254, The call to
this.store.replaceRollupSources(...) inside the withDatabaseTransaction block is
not awaited, so the replacement may not finish before the transaction commits;
update the call in rollup-builder.ts to await
this.store.replaceRollupSources(...) (after rollup-store.ts has been fixed to
return a Promise) so the source-replace runs inside and completes within the
surrounding withDatabaseTransaction; ensure you await replaceRollupSources in
the same transaction before calling upsertRollup/upsertState to preserve
transactional guarantees.

Comment thread src/rollup-builder.ts Outdated
Comment thread src/rollup-builder.ts
Comment on lines +507 to +513
function estimateTokens(content: string): number {
const text = content.trim();
if (!text) {
return 0;
}
return Math.max(1, Math.ceil(text.length / 4));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Token estimation is a rough heuristic.

estimateTokens uses ceil(length / 4) which assumes ~4 chars/token (typical for English with GPT-style tokenizers). This underestimates for CJK text (~1-2 chars/token) and varies for code.

Since it's used consistently for both budget checking and comparison, the relative accuracy is acceptable. Consider documenting the limitation or using an actual tokenizer if precision matters for production rollups.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/rollup-builder.ts` around lines 507 - 513, estimateTokens currently uses
a simple heuristic (Math.ceil(text.length / 4)) which misestimates for CJK and
code; update the implementation or surrounding docs: either replace
estimateTokens with a real tokenizer call (e.g., use the project's tokenizer
utility or an external tokenizer) and use that in all places that call
estimateTokens, or add an inline comment / JSDoc on the estimateTokens function
explaining the ~4 chars/token heuristic and its limitations for CJK and code so
callers know the inaccuracy; reference the estimateTokens function to locate and
change the behavior or add the documentation.

Comment thread src/store/rollup-store.ts
Comment thread src/store/rollup-store.ts
Comment on lines +274 to +291
replaceRollupSources(rollupId: string, sources: RollupSourceInput[]): void {
void withDatabaseTransaction(this.db, "BEGIN", () => {
this.db.prepare(`DELETE FROM lcm_rollup_sources WHERE rollup_id = ?`).run(rollupId);

if (sources.length === 0) {
return;
}

const insert = this.db.prepare(
`INSERT INTO lcm_rollup_sources (rollup_id, source_type, source_id, ordinal)
VALUES (?, ?, ?, ?)`,
);

for (const source of sources) {
insert.run(rollupId, source.type, source.id, source.ordinal);
}
});
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Critical: Transaction not awaited — race condition and silent failures.

void withDatabaseTransaction(...) discards the Promise, making this fire-and-forget. The DELETE/INSERT operations may not complete before the outer transaction in buildDayRollup commits, causing:

  • Race condition: sources may not be persisted
  • Silent failures: errors are swallowed
Proposed fix: Properly await the transaction
-  replaceRollupSources(rollupId: string, sources: RollupSourceInput[]): void {
-    void withDatabaseTransaction(this.db, "BEGIN", () => {
+  async replaceRollupSources(rollupId: string, sources: RollupSourceInput[]): Promise<void> {
+    await withDatabaseTransaction(this.db, "BEGIN", () => {
       this.db.prepare(`DELETE FROM lcm_rollup_sources WHERE rollup_id = ?`).run(rollupId);

       if (sources.length === 0) {
         return;
       }

       const insert = this.db.prepare(
         `INSERT INTO lcm_rollup_sources (rollup_id, source_type, source_id, ordinal)
          VALUES (?, ?, ?, ?)`,
       );

       for (const source of sources) {
         insert.run(rollupId, source.type, source.id, source.ordinal);
       }
     });
   }

Also update the call site in src/rollup-builder.ts:

-      this.store.replaceRollupSources(
+      await this.store.replaceRollupSources(
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
replaceRollupSources(rollupId: string, sources: RollupSourceInput[]): void {
void withDatabaseTransaction(this.db, "BEGIN", () => {
this.db.prepare(`DELETE FROM lcm_rollup_sources WHERE rollup_id = ?`).run(rollupId);
if (sources.length === 0) {
return;
}
const insert = this.db.prepare(
`INSERT INTO lcm_rollup_sources (rollup_id, source_type, source_id, ordinal)
VALUES (?, ?, ?, ?)`,
);
for (const source of sources) {
insert.run(rollupId, source.type, source.id, source.ordinal);
}
});
}
async replaceRollupSources(rollupId: string, sources: RollupSourceInput[]): Promise<void> {
await withDatabaseTransaction(this.db, "BEGIN", () => {
this.db.prepare(`DELETE FROM lcm_rollup_sources WHERE rollup_id = ?`).run(rollupId);
if (sources.length === 0) {
return;
}
const insert = this.db.prepare(
`INSERT INTO lcm_rollup_sources (rollup_id, source_type, source_id, ordinal)
VALUES (?, ?, ?, ?)`,
);
for (const source of sources) {
insert.run(rollupId, source.type, source.id, source.ordinal);
}
});
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/store/rollup-store.ts` around lines 274 - 291, The replaceRollupSources
function fires a transaction with void withDatabaseTransaction(...) which
discards the Promise causing race conditions and silent failures; make
replaceRollupSources return a Promise and await the transaction (change
signature to async and remove the void) so the DELETE/INSERT complete before
callers proceed, and then update call sites such as buildDayRollup in
src/rollup-builder.ts to await replaceRollupSources to ensure the transaction is
completed before the outer transaction commits.

Comment thread src/store/rollup-store.ts
Comment on lines +99 to +105
function getLcmDatabase(lcm: LcmContextEngine): DatabaseSync {
const candidate = lcm as unknown as { db?: DatabaseSync };
if (!candidate.db) {
throw new Error("LCM rollup database is unavailable.");
}
return candidate.db;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Type-safe database accessor uses internal property access.

The getLcmDatabase function accesses lcm.db through a type assertion. This works but couples to internal implementation. If the engine's database access changes, this will break silently.

Consider exposing a proper getter on LcmContextEngine or documenting this coupling.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/tools/lcm-recent-tool.ts` around lines 99 - 105, getLcmDatabase currently
reaches into an internal property via a type assertion (candidate.db) which
couples callers to LcmContextEngine's implementation; add a public accessor on
LcmContextEngine (e.g., getDatabase(): DatabaseSync | undefined or throw) and
update getLcmDatabase to call that accessor (handle undefined by throwing the
same error) so callers use the stable API instead of candidate.db, or
alternatively document this coupling on LcmContextEngine if adding a getter is
not possible.

Comment thread src/tools/lcm-recent-tool.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 13a964c3a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/rollup-builder.ts Outdated
Comment on lines +89 to +91
if (state && state.pending_rebuild === 0 && !forceCurrentDay) {
result.skipped += daysBack;
return result;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Do not skip rebuild checks when pending_rebuild is 0

This early return prevents any fingerprint comparison or day-level rebuild once a state row exists with pending_rebuild = 0. In this same commit, buildDayRollup always writes pending_rebuild: 0, and there is no corresponding path that flips it back to 1 when new leaf summaries arrive, so subsequent buildDailyRollups() calls will skip all days and leave rollups stale.

Useful? React with 👍 / 👎.

Comment thread src/tools/lcm-recent-tool.ts Outdated

if (conversationScope.allConversations) {
const fallback = await retrieval.grep({
query: "",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid empty grep query in all-conversations fallback

Passing query: "" into retrieval.grep makes this fallback effectively return nothing for most backends (FTS query sanitization yields an empty phrase and the LIKE fallback drops empty-term queries), so lcm_recent(allConversations=true) can incorrectly report no activity even when messages/summaries exist in the requested time window.

Useful? React with 👍 / 👎.

Comment thread src/rollup-builder.ts Outdated
if (!/^\d{4}-\d{2}-\d{2}$/.test(dateKey)) {
throw new Error(`Invalid date key: ${dateKey}`);
}
return new Date(`${dateKey}T12:00:00.000Z`);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Stop parsing date keys via fixed noon UTC timestamps

Using T12:00:00.000Z to represent a calendar day shifts to the next local date in high positive-offset zones (for example UTC+13/UTC+14), so downstream calls like getLocalDayBounds(parseDateKey(dateKey), tz) can compute bounds for the wrong day and misfile rollups by one date.

Useful? React with 👍 / 👎.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an “LCM recent” capability by introducing temporal rollup storage (schema + store) and a new lcm_recent tool intended to read daily/weekly/monthly rollups with fallback recall.

Changes:

  • Added SQLite migration steps for lcm_rollups, lcm_rollup_sources, and lcm_rollup_state plus supporting indexes/views.
  • Introduced RollupStore and RollupBuilder for writing/maintaining daily rollups.
  • Registered a new lcm_recent tool to retrieve rollups (or fall back to recent items).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
src/tools/lcm-recent-tool.ts Implements the lcm_recent tool, period resolution, rollup retrieval, and fallback output formatting.
src/store/rollup-store.ts Adds DB access layer for rollups/state/sources with UPSERT semantics.
src/store/index.ts Exposes RollupStore and related types via store barrel exports.
src/rollup-builder.ts Adds logic to build daily rollups from leaf summaries and persist them.
src/plugin/index.ts Registers the lcm_recent tool with the plugin API.
src/db/migration.ts Creates rollup tables, indexes, and views during migrations (before the FTS5 guard).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/store/rollup-store.ts Outdated
error_text
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, datetime('now'), NULL, NULL)
ON CONFLICT(conversation_id, period_kind, period_key) DO UPDATE SET
rollup_id = excluded.rollup_id,

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

upsertRollup updates rollup_id on conflict (rollup_id = excluded.rollup_id). Because lcm_rollup_sources.rollup_id is a foreign key without ON UPDATE CASCADE, changing the primary key can either fail with a foreign key constraint error or orphan existing lcm_rollup_sources rows. Consider keeping the existing rollup_id on conflict (don’t update the PK), or add ON UPDATE CASCADE to the foreign key if PK updates are required.

Suggested change
rollup_id = excluded.rollup_id,

Copilot uses AI. Check for mistakes.
Comment thread src/store/rollup-store.ts Outdated
Comment on lines +274 to +275
replaceRollupSources(rollupId: string, sources: RollupSourceInput[]): void {
void withDatabaseTransaction(this.db, "BEGIN", () => {

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replaceRollupSources calls the async withDatabaseTransaction(...) but does not await it (uses void ...). This can allow the caller to continue before the delete/insert work is committed/released, and any errors become unhandled promise rejections. Make this method async and await the transaction (and propagate errors).

Suggested change
replaceRollupSources(rollupId: string, sources: RollupSourceInput[]): void {
void withDatabaseTransaction(this.db, "BEGIN", () => {
async replaceRollupSources(rollupId: string, sources: RollupSourceInput[]): Promise<void> {
await withDatabaseTransaction(this.db, "BEGIN", () => {

Copilot uses AI. Check for mistakes.
Comment on lines +355 to +364
const fallback = await retrieval.grep({
query: "",
mode: "full_text",
scope: "both",
conversationId: undefined,
since: resolution.start,
before: resolution.end,
sort: "recency",
limit: 20,
});

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The all-conversations fallback calls retrieval.grep with query: "" and mode: "full_text". With the current FTS5 sanitization this likely becomes a match-all query, which can be extremely expensive because it forces the FTS table to return/score a huge result set before created_at ordering + LIMIT can be applied. Prefer a dedicated "recent items" query path (e.g., fetch messages/summaries by created_at within the bounds) instead of FTS when there is no search term.

Copilot uses AI. Check for mistakes.
Comment thread src/tools/lcm-recent-tool.ts Outdated
Comment on lines +347 to +352
try {
parseIsoTimestampParam(p, "since");
parseIsoTimestampParam(p, "before");
} catch {
// Intentional no-op, imported helper kept aligned with surrounding tool conventions.
}

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parseIsoTimestampParam(p, "since"|"before") is invoked but its return values are unused and any parsing errors are swallowed. Since since/before aren’t part of this tool’s schema, this code is effectively a no-op and can hide invalid caller input. Either remove this block entirely, or add since/before to the schema and actually apply them (with proper validation/error reporting).

Copilot uses AI. Check for mistakes.
Comment thread src/plugin/index.ts
Comment on lines +1877 to +1879
api.registerTool((ctx) =>
createLcmRecentTool({ deps, getLcm: shared.waitForEngine, sessionKey: ctx.sessionKey }),
);

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tool is registered, but there’s no wiring that actually builds/populates lcm_rollups (e.g., RollupBuilder isn’t invoked anywhere in the runtime). As a result lcm_recent will almost always hit fallback behavior, which doesn’t match the PR description of adding a daily/weekly/monthly rollup layer. Either add a scheduled/background build step (or hook into existing compaction/summarization flows) or adjust the PR description/tool description to reflect that rollups must be built externally.

Copilot uses AI. Check for mistakes.
Comment on lines +463 to +468
if (rollupContent == null) {
const recentSummaries = getRecentSummaryFallback(db, conversationId, resolution.start, resolution.end);

const lines: string[] = [];
lines.push(`## Recent Activity: ${resolution.label}`);
lines.push(

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tool description says it "Falls back to time-bounded lcm_grep when no rollup exists", but the single-conversation fallback path instead runs a direct SQL query over summaries (leaf only) and does not include message matches. If the intended fallback is lcm_grep-style behavior, consider routing through retrieval.grep (with appropriate query semantics) or update the tool/PR description to accurately describe the implemented fallback behavior.

Copilot uses AI. Check for mistakes.
Comment on lines +299 to +306
export function createLcmRecentTool(input: {
deps: LcmDependencies;
lcm?: LcmContextEngine;
getLcm?: () => Promise<LcmContextEngine>;
rollupStore?: RollupStore;
sessionId?: string;
sessionKey?: string;
}): AnyAgentTool {

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No automated tests were added for lcm_recent period resolution, rollup selection (single-period vs multi-day combine), and fallback formatting. The repo already has Vitest coverage for other LCM tools (e.g. test/lcm-tools.test.ts), so adding similar tests here would help prevent regressions in date/timezone logic and output structure.

Copilot uses AI. Check for mistakes.
Comment thread src/db/migration.ts
Comment on lines +911 to +916
CREATE TABLE IF NOT EXISTS lcm_rollup_sources (
rollup_id TEXT NOT NULL REFERENCES lcm_rollups(rollup_id) ON DELETE CASCADE,
source_type TEXT NOT NULL CHECK (source_type IN ('summary', 'rollup')),
source_id TEXT NOT NULL,
ordinal INTEGER NOT NULL,
PRIMARY KEY (rollup_id, source_type, ordinal)

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lcm_rollup_sources.rollup_id references lcm_rollups(rollup_id) with ON DELETE CASCADE but no ON UPDATE action. Since RollupStore.upsertRollup currently attempts to update rollup_id on conflict, this schema will either block the update (FK constraint failure) or leave sources inconsistent. Either avoid updating rollup_id in the upsert, or define the FK with ON UPDATE CASCADE (and migrate existing DBs accordingly).

Copilot uses AI. Check for mistakes.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 7 potential issues.

Open in Devin Review

Comment thread src/store/rollup-store.ts
Comment on lines +299 to +312
export function createLcmRecentTool(input: {
deps: LcmDependencies;
lcm?: LcmContextEngine;
getLcm?: () => Promise<LcmContextEngine>;
rollupStore?: RollupStore;
sessionId?: string;
sessionKey?: string;
}): AnyAgentTool {
return {
name: "lcm_recent",
label: "LCM Recent",
description:
"Retrieve recent activity summaries from pre-built temporal rollups. Returns daily, weekly, or monthly summaries without LLM calls. Use for questions like 'what happened today?', 'what did we do yesterday?', or recap requests. Falls back to time-bounded lcm_grep when no rollup exists.",
parameters: LcmRecentSchema,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Missing changeset file for new user-facing features (lcm_recent tool, rollup tables)

Per AGENTS.md: "Before merging a PR, check whether it changes user-facing behavior or should appear in npm release notes. If yes, make sure a maintainer adds a .changeset/*.md file before merge or immediately after in a follow-up PR." and "Treat a PR as not release-ready until the changeset question has been answered."

This PR adds significant new user-facing features — the lcm_recent tool, rollup database tables, RollupStore, and RollupBuilder — but contains no corresponding changeset file. The existing changesets (wise-bears-doubt.md and async-bootstrap-sync-io.md) are for unrelated features. A minor bump changeset should be added for this new feature.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/rollup-builder.ts Outdated
Comment thread src/rollup-builder.ts
Comment on lines +183 to +213
if (
existing?.rollup_id
&& existing.source_fingerprint
&& existing.source_fingerprint !== fingerprint
) {
this.store.markStale(existing.rollup_id);
}

this.store.upsertRollup({
rollup_id: rollupId,
conversation_id: conversationId,
period_kind: PERIOD_KIND,
period_key: dateKey,
period_start: start.toISOString(),
period_end: end.toISOString(),
timezone: this.config.timezone,
content: draft.content,
token_count: draft.summaryTokenCount,
source_summary_ids: JSON.stringify(summaries.map((summary) => summary.summaryId)),
source_message_count: 0,
source_token_count: totalSourceTokens,
status: "building",
coverage_start:
summaries[0]?.earliestAt?.toISOString() ?? summaries[0]?.createdAt.toISOString() ?? null,
coverage_end:
summaries[summaries.length - 1]?.latestAt?.toISOString()
?? summaries[summaries.length - 1]?.createdAt.toISOString()
?? null,
summarizer_model: "concatenation-v1",
source_fingerprint: fingerprint,
});

@devin-ai-integration devin-ai-integration Bot Apr 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: markStale call is redundant before upsertRollup in same transaction

At src/rollup-builder.ts:178-184, markStale(existing.rollup_id) sets the row's status to 'stale'. Immediately after, upsertRollup at line 186 overwrites the same row (same conversation_id, period_kind, period_key) with status 'ready' via ON CONFLICT ... DO UPDATE. The markStale write is immediately overwritten and has no observable effect since both operations are in the same transaction. If the intent was to leave an audit trail of the stale transition, it's lost. Not a bug but unnecessary work.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +99 to +105
function getLcmDatabase(lcm: LcmContextEngine): DatabaseSync {
const candidate = lcm as unknown as { db?: DatabaseSync };
if (!candidate.db) {
throw new Error("LCM rollup database is unavailable.");
}
return candidate.db;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 getLcmDatabase accesses private field via unsafe cast — fragile coupling

In src/tools/lcm-recent-tool.ts:99-105, getLcmDatabase casts LcmContextEngine through unknown to access the private readonly db field (src/engine.ts:1201). This works at runtime because TypeScript's private is not enforced in JavaScript, but it creates fragile coupling — renaming or making db a #private field would silently break this. No other tool in the codebase uses this pattern (confirmed via search). Consider adding a public accessor like getDatabase() on LcmContextEngine if tools need DB access.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/store/rollup-store.ts Outdated
Comment thread src/rollup-builder.ts
Comment on lines +546 to +551
function localDateTimeToUtc(dateKey: string, time: string, timezone: string): Date {
const [year, month, day] = dateKey.split("-").map((part) => Number.parseInt(part, 10));
const [hour, minute, second] = time.split(":").map((part) => Number.parseInt(part, 10));
const utcGuess = new Date(Date.UTC(year, month - 1, day, hour, minute, second, 0));
const offsetMs = getTimeZoneOffsetMs(utcGuess, timezone);
return new Date(utcGuess.getTime() - offsetMs);

@devin-ai-integration devin-ai-integration Bot Apr 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: localDateTimeToUtc uses single-pass offset correction — may drift near DST transitions

In src/rollup-builder.ts:546-551, localDateTimeToUtc computes a UTC date by guessing a UTC time, calculating the timezone offset at that guess, and subtracting. This single-pass approach can produce incorrect results near DST transitions where the offset at the guessed UTC time differs from the offset at the corrected UTC time. However, this function is only called from getLocalDayBounds with 00:00:00 time, and DST transitions almost universally happen at 2am+, so midnight conversions are safe in practice. Worth documenting this limitation if the function is ever reused for arbitrary times.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

♻️ Duplicate comments (1)
src/store/rollup-store.ts (1)

272-287: 🧹 Nitpick | 🔵 Trivial

Non-atomic replaceRollupSources relies on caller transaction.

The DELETE + INSERT sequence is not wrapped in its own transaction. This works because RollupBuilder.buildDayRollup calls this within withDatabaseTransaction(..., "BEGIN IMMEDIATE", ...). However, direct callers could leave partial state.

Consider either:

  1. Documenting the transactional requirement
  2. Adding defensive transaction handling
+  /**
+   * Replace all sources for a rollup. Must be called within a transaction
+   * for atomicity (e.g., via withDatabaseTransaction).
+   */
   replaceRollupSources(rollupId: string, sources: RollupSourceInput[]): void {
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/store/rollup-store.ts` around lines 272 - 287, The replaceRollupSources
function performs a DELETE followed by multiple INSERTs without its own
transaction, risking partial state if the caller doesn't wrap it; modify
replaceRollupSources to execute the DELETE and all INSERTs inside a single
database transaction (e.g., begin/commit or using the DB's transaction helper)
so the operation is atomic, referencing the existing function name
replaceRollupSources and callers like
RollupBuilder.buildDayRollup/withDatabaseTransaction for context.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/db/migration.ts`:
- Around line 887-931: Add a periodic GC job (Phase 2 TODO in index.ts) that
removes orphaned rollup data left when conversations are deleted: run a
transaction that DELETEs from lcm_rollups where conversation_id NOT IN (SELECT
id FROM conversations) (this will cascade-remove lcm_rollup_sources), and
DELETEs from lcm_rollup_state where conversation_id NOT IN (SELECT id FROM
conversations); schedule it to run regularly (daily or configurable), wrap in a
safe transaction with logging and error handling, and expose a way to trigger it
manually for backfill/recovery.

In `@src/rollup-builder.ts`:
- Around line 106-108: The .filter((summary) => summary.kind === "leaf") is
redundant because getLeafSummariesForDay already returns only leaf summaries;
remove that filter and directly sort the summaries with
compareSummariesChronologically (i.e., replace const leafSummaries =
summaries.filter(...).sort(...) with const leafSummaries =
summaries.sort(compareSummariesChronologically)), keeping the variable name and
usage intact; reference getLeafSummariesForDay in rollup-store.ts and
compareSummariesChronologically to locate the related logic.

In `@src/tools/lcm-recent-tool.ts`:
- Around line 422-454: Extract the inline mapping from RollupRow to RollupRecord
into a shared helper (e.g., toRollupRecord) and use it where rollups are
transformed (the current mapping inside the resolution.kind branch that produces
usableRollups); keep the same field conversions (new Date for
period_start/period_end/coverage_start/coverage_end/built_at/invalidated_at,
parseJsonStringArray for source_summary_ids, and direct mappings for other
fields), then replace the .map(...) call with
rollups.filter(...).map(toRollupRecord) so combineRollups and other consumers
(e.g., combineRollups, rollupStore.listRollups usage) can reuse the conversion.
- Around line 200-218: The returned objects for normalized values "7d" and "30d"
use kind: "day" which is misleading because these are multi-day ranges; add a
brief clarifying comment next to the blocks handling normalized === "7d" and
normalized === "30d" explaining that kind: "day" denotes the rollup/unit (daily
granularity) while the start/end define a multi-day range and that periodKey
(see usage around periodKey at line ~422) drives list-based lookups rather than
single-day behavior; keep getUtcDateForZonedMidnight and the existing start/end
logic intact and only add the comment to improve maintainability.

---

Duplicate comments:
In `@src/store/rollup-store.ts`:
- Around line 272-287: The replaceRollupSources function performs a DELETE
followed by multiple INSERTs without its own transaction, risking partial state
if the caller doesn't wrap it; modify replaceRollupSources to execute the DELETE
and all INSERTs inside a single database transaction (e.g., begin/commit or
using the DB's transaction helper) so the operation is atomic, referencing the
existing function name replaceRollupSources and callers like
RollupBuilder.buildDayRollup/withDatabaseTransaction for context.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 80aa9fee-2616-47ad-9492-e2393ea96060

📥 Commits

Reviewing files that changed from the base of the PR and between 13a964c and 544984a.

📒 Files selected for processing (5)
  • src/db/migration.ts
  • src/plugin/index.ts
  • src/rollup-builder.ts
  • src/store/rollup-store.ts
  • src/tools/lcm-recent-tool.ts
📜 Review details
🔇 Additional comments (28)
src/tools/lcm-recent-tool.ts (10)

99-105: Internal property access for database handle.

This accesses lcm.db through a type assertion. While functional, it couples to internal implementation. Consider exposing a getDatabase() accessor on LcmContextEngine if this pattern becomes more common.


1-10: LGTM — Clean imports and module setup.

Imports are appropriately scoped. The module correctly depends on RollupStore for rollup retrieval and formatTimestamp from compaction for consistent time display.


11-74: LGTM — Schema and type definitions.

The Typebox schema provides clear parameter descriptions, and the TypeScript types (RollupRecord, PeriodResolution, etc.) align with the database schema defined in migration.ts.


128-151: LGTM — Timezone conversion logic.

The algorithm correctly computes UTC timestamps for local midnight by using Intl.DateTimeFormat to determine actual timezone offsets. This handles DST transitions properly for the common case where spring-forward occurs at 2am.


258-271: LGTM — Rollup combination logic.

Correctly aggregates multiple rollups: concatenates content with headers, sums tokens, deduplicates source IDs, and conservatively returns "stale" if any source rollup is stale.


273-297: LGTM — Fallback summary query.

The SQL correctly uses COALESCE(latest_at, earliest_at, created_at) for timestamp fallback, matching the pattern in RollupStore.getLeafSummariesForDay. The LIMIT 20 prevents unbounded result sets.


313-345: LGTM — Input validation and period resolution.

Proper error handling for missing conversation scope and invalid period formats. The jsonResult wrapper ensures structured error responses.


347-403: LGTM — Cross-conversation fallback path.

The allConversations: true path correctly falls back to a time-bounded grep query with query: "*" to retrieve recent activity across all conversations. The formatted output includes appropriate drill-down guidance.


456-493: LGTM — Leaf summary fallback path.

When no rollup exists, the fallback correctly queries leaf summaries in the time window and formats them with appropriate messaging. The summaryIds are included in details for downstream expansion.


495-519: LGTM — Successful rollup response formatting.

The output format is consistent with the fallback paths, including period metadata, status, token count, and drill-down instructions.

src/rollup-builder.ts (9)

478-484: Document token estimation heuristic.

The ~4 chars/token heuristic is reasonable for relative comparisons but underestimates for CJK text. A brief comment would clarify the limitation.

+/**
+ * Rough token estimate assuming ~4 characters per token.
+ * Underestimates for CJK (~1-2 chars/token). Used consistently
+ * for both budgeting and fingerprinting, so relative accuracy suffices.
+ */
 function estimateTokens(content: string): number {

1-52: LGTM — Module setup and types.

Clear constants for token budgets and timeline limits. Type definitions accurately represent the data structures used in rollup building.


53-69: LGTM — Builder initialization.

The constructor correctly enforces dailyMaxTokens >= dailyTargetTokens and uses safe normalization for configuration values.


71-146: LGTM — Daily rollup iteration logic.

The fingerprint-based deduplication efficiently skips unchanged days. Per-day error handling with collection provides good observability without failing the entire batch.


177-225: LGTM — Atomic transaction for rollup persistence.

The withDatabaseTransaction(..., "BEGIN IMMEDIATE", ...) correctly wraps all mutations:

  1. markStale (conditional)
  2. upsertRollup (single call with final status)
  3. replaceRollupSources (synchronous within transaction)
  4. upsertState

Previous review concerns about double upsert and non-awaited calls have been addressed.


270-322: LGTM — Token-budgeted content generation.

The iterative truncation correctly prioritizes recency by removing oldest entries first. The fallback path at lines 302-315 handles the edge case where even minimal content exceeds the budget.


386-421: LGTM — Heuristic key item extraction.

The regex-based categorization (decisions, completed, blockers) provides useful structure for daily rollups. Case-insensitive deduplication prevents near-duplicate entries.


245-264: LGTM — Exported fingerprint and date utilities.

computeFingerprint provides deterministic rollup change detection. getLocalDateKey and getLocalDayBounds are correctly exported for use by consumers needing consistent date handling.


517-547: LGTM — Timezone offset calculation.

The getTimeZoneOffsetMs implementation correctly uses Intl.DateTimeFormat to determine actual timezone offsets at specific points in time, handling DST correctly.

src/plugin/index.ts (1)

22-22: LGTM — Tool registration follows established pattern.

The createLcmRecentTool import and registration mirror the existing LCM tools (lcm_grep, lcm_describe, lcm_expand, lcm_expand_query), using the same deps, getLcm, and sessionKey wiring. The TODO comment appropriately tracks Phase 2 integration work.

Also applies to: 1877-1881

src/db/migration.ts (3)

570-586: LGTM — Per-name table existence check.

The refactored implementation iterates over table names individually rather than using a bulk IN (...) clause. While slightly less efficient for large name lists, this avoids potential issues with placeholder handling in certain SQLite driver versions.


932-970: LGTM — Indexes well-designed for rollup queries.

  • summaries_leaf_temporal_idx partial index optimizes leaf summary lookups
  • FK hint indexes mitigate missing FK constraints for query planning
  • Conditional creation based on PRAGMA foreign_key_list avoids duplicates

971-980: LGTM — Period-scoped views.

Simple filtered views for daily_rollups, weekly_rollups, and monthly_rollups provide convenient access patterns for the lcm_recent tool.

src/store/rollup-store.ts (5)

4-56: LGTM — Type definitions.

Interface definitions align with the database schema and provide clear contracts for the store methods.


61-125: LGTM — Upsert correctly preserves rollup_id on conflict.

The ON CONFLICT DO UPDATE clause does not overwrite rollup_id, addressing the previously flagged FK violation risk. The existing rollup_id is preserved, keeping lcm_rollup_sources references intact.


127-255: LGTM — Rollup query methods.

Query methods are well-structured with appropriate parameter handling. listRollups correctly normalizes the limit and handles optional periodKind filtering.


319-367: LGTM — Consolidated upsertState implementation.

The dynamic SET clause construction addresses the previous review's efficiency concern. Only provided fields are updated, and updated_at is always set.


369-398: LGTM — Leaf summary retrieval.

The query correctly uses COALESCE(latest_at, earliest_at, created_at) for timestamp filtering and ordering, matching the fallback pattern used throughout the rollup system.

Comment thread src/db/migration.ts
Comment on lines +887 to +931
// ── Rollup tables (temporal awareness layer) ──────────────────────
// These are independent of FTS5 and must run before the fts5 early-return.
runMigrationStep("ensureRollupTables", log, () => {
db.exec(`
CREATE TABLE IF NOT EXISTS lcm_rollups (
rollup_id TEXT PRIMARY KEY,
conversation_id INTEGER NOT NULL,
period_kind TEXT NOT NULL CHECK (period_kind IN ('day', 'week', 'month')),
period_key TEXT NOT NULL,
period_start TEXT NOT NULL,
period_end TEXT NOT NULL,
timezone TEXT NOT NULL DEFAULT 'UTC',
content TEXT NOT NULL,
token_count INTEGER NOT NULL DEFAULT 0,
source_summary_ids TEXT NOT NULL DEFAULT '[]',
source_message_count INTEGER NOT NULL DEFAULT 0,
source_token_count INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'ready' CHECK (status IN ('building', 'ready', 'stale', 'failed')),
coverage_start TEXT,
coverage_end TEXT,
summarizer_model TEXT,
source_fingerprint TEXT,
built_at TEXT NOT NULL DEFAULT (datetime('now')),
invalidated_at TEXT,
error_text TEXT,
UNIQUE (conversation_id, period_kind, period_key)
);
CREATE TABLE IF NOT EXISTS lcm_rollup_sources (
rollup_id TEXT NOT NULL REFERENCES lcm_rollups(rollup_id) ON DELETE CASCADE,
source_type TEXT NOT NULL CHECK (source_type IN ('summary', 'rollup')),
source_id TEXT NOT NULL,
ordinal INTEGER NOT NULL,
PRIMARY KEY (rollup_id, source_type, ordinal)
);
CREATE TABLE IF NOT EXISTS lcm_rollup_state (
conversation_id INTEGER PRIMARY KEY,
timezone TEXT NOT NULL DEFAULT 'UTC',
last_message_at TEXT,
last_rollup_check_at TEXT,
last_daily_build_at TEXT,
pending_rebuild INTEGER NOT NULL DEFAULT 0,
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
`);
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Rollup tables: Schema is sound, orphan cleanup needed.

The tables are well-structured with appropriate constraints:

  • lcm_rollups: unique on (conversation_id, period_kind, period_key)
  • lcm_rollup_sources: cascades on rollup deletion

As noted in a previous review, lcm_rollups and lcm_rollup_state lack FKs to conversations. This appears intentional for FTS5-independence but means orphaned rollups won't auto-delete when conversations are removed. The Phase 2 TODO in index.ts (line 1880-1881) should include periodic GC for orphaned rollup data.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/db/migration.ts` around lines 887 - 931, Add a periodic GC job (Phase 2
TODO in index.ts) that removes orphaned rollup data left when conversations are
deleted: run a transaction that DELETEs from lcm_rollups where conversation_id
NOT IN (SELECT id FROM conversations) (this will cascade-remove
lcm_rollup_sources), and DELETEs from lcm_rollup_state where conversation_id NOT
IN (SELECT id FROM conversations); schedule it to run regularly (daily or
configurable), wrap in a safe transaction with logging and error handling, and
expose a way to trigger it manually for backfill/recovery.

Comment thread src/rollup-builder.ts
Comment on lines +106 to +108
const leafSummaries = summaries
.filter((summary) => summary.kind === "leaf")
.sort(compareSummariesChronologically);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Redundant leaf filter.

getLeafSummariesForDay already filters to kind = 'leaf' in SQL (line 392 of rollup-store.ts). The .filter((summary) => summary.kind === "leaf") is defensive but redundant.

-      const leafSummaries = summaries
-        .filter((summary) => summary.kind === "leaf")
-        .sort(compareSummariesChronologically);
+      const leafSummaries = summaries.sort(compareSummariesChronologically);
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const leafSummaries = summaries
.filter((summary) => summary.kind === "leaf")
.sort(compareSummariesChronologically);
const leafSummaries = summaries.sort(compareSummariesChronologically);
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/rollup-builder.ts` around lines 106 - 108, The .filter((summary) =>
summary.kind === "leaf") is redundant because getLeafSummariesForDay already
returns only leaf summaries; remove that filter and directly sort the summaries
with compareSummariesChronologically (i.e., replace const leafSummaries =
summaries.filter(...).sort(...) with const leafSummaries =
summaries.sort(compareSummariesChronologically)), keeping the variable name and
usage intact; reference getLeafSummariesForDay in rollup-store.ts and
compareSummariesChronologically to locate the related logic.

Comment on lines +200 to +218
if (normalized === "7d") {
const startDay = addDays(today, -6);
return {
label: "last 7 days",
kind: "day",
start: getUtcDateForZonedMidnight(startDay, timezone),
end: getUtcDateForZonedMidnight(addDays(today, 1), timezone),
};
}

if (normalized === "30d") {
const startDay = addDays(today, -29);
return {
label: "last 30 days",
kind: "day",
start: getUtcDateForZonedMidnight(startDay, timezone),
end: getUtcDateForZonedMidnight(addDays(today, 1), timezone),
};
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Clarify kind semantics for multi-day ranges.

For "7d" and "30d", the returned kind: "day" is semantically misleading since these represent ranges, not single day rollups. The code handles this correctly (no periodKey triggers list-based lookup at line 422), but a comment explaining this behavior would improve maintainability.

   if (normalized === "7d") {
     const startDay = addDays(today, -6);
     return {
       label: "last 7 days",
-      kind: "day",
+      kind: "day", // Multi-day range; no periodKey triggers list-based rollup lookup
       start: getUtcDateForZonedMidnight(startDay, timezone),
       end: getUtcDateForZonedMidnight(addDays(today, 1), timezone),
     };
   }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if (normalized === "7d") {
const startDay = addDays(today, -6);
return {
label: "last 7 days",
kind: "day",
start: getUtcDateForZonedMidnight(startDay, timezone),
end: getUtcDateForZonedMidnight(addDays(today, 1), timezone),
};
}
if (normalized === "30d") {
const startDay = addDays(today, -29);
return {
label: "last 30 days",
kind: "day",
start: getUtcDateForZonedMidnight(startDay, timezone),
end: getUtcDateForZonedMidnight(addDays(today, 1), timezone),
};
}
if (normalized === "7d") {
const startDay = addDays(today, -6);
return {
label: "last 7 days",
kind: "day", // Multi-day range; no periodKey triggers list-based rollup lookup
start: getUtcDateForZonedMidnight(startDay, timezone),
end: getUtcDateForZonedMidnight(addDays(today, 1), timezone),
};
}
if (normalized === "30d") {
const startDay = addDays(today, -29);
return {
label: "last 30 days",
kind: "day", // Multi-day range; no periodKey triggers list-based rollup lookup
start: getUtcDateForZonedMidnight(startDay, timezone),
end: getUtcDateForZonedMidnight(addDays(today, 1), timezone),
};
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/tools/lcm-recent-tool.ts` around lines 200 - 218, The returned objects
for normalized values "7d" and "30d" use kind: "day" which is misleading because
these are multi-day ranges; add a brief clarifying comment next to the blocks
handling normalized === "7d" and normalized === "30d" explaining that kind:
"day" denotes the rollup/unit (daily granularity) while the start/end define a
multi-day range and that periodKey (see usage around periodKey at line ~422)
drives list-based lookups rather than single-day behavior; keep
getUtcDateForZonedMidnight and the existing start/end logic intact and only add
the comment to improve maintainability.

Comment on lines +422 to +454
} else if (resolution.kind) {
const rollups = rollupStore.listRollups(conversationId, resolution.kind, 200)
.filter((rollup) => new Date(rollup.period_start) >= resolution.start && new Date(rollup.period_start) < resolution.end);
const usableRollups = rollups.filter((rollup) => rollup.status === "ready" || rollup.status === "stale").map((rollup) => ({
rollupId: rollup.rollup_id,
conversationId: rollup.conversation_id,
periodKind: rollup.period_kind,
periodKey: rollup.period_key,
periodStart: new Date(rollup.period_start),
periodEnd: new Date(rollup.period_end),
timezone: rollup.timezone,
content: rollup.content,
tokenCount: rollup.token_count,
sourceSummaryIds: parseJsonStringArray(rollup.source_summary_ids),
sourceMessageCount: rollup.source_message_count,
sourceTokenCount: rollup.source_token_count,
status: rollup.status,
coverageStart: rollup.coverage_start ? new Date(rollup.coverage_start) : null,
coverageEnd: rollup.coverage_end ? new Date(rollup.coverage_end) : null,
summarizerModel: rollup.summarizer_model,
sourceFingerprint: rollup.source_fingerprint,
builtAt: new Date(rollup.built_at),
invalidatedAt: rollup.invalidated_at ? new Date(rollup.invalidated_at) : null,
errorText: rollup.error_text,
}));
if (usableRollups.length > 0) {
const combined = combineRollups(usableRollups);
rollupContent = combined.content;
tokenCount = combined.tokenCount;
status = combined.status;
sourceSummaryIds = combined.sourceSummaryIds;
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Consider extracting rollup row mapping.

The mapping from RollupRow to RollupRecord (lines 425-446) is verbose but correct. If reused elsewhere, consider extracting to a shared helper.

function toRollupRecord(row: RollupRow): RollupRecord {
  return {
    rollupId: row.rollup_id,
    // ... rest of mapping
  };
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/tools/lcm-recent-tool.ts` around lines 422 - 454, Extract the inline
mapping from RollupRow to RollupRecord into a shared helper (e.g.,
toRollupRecord) and use it where rollups are transformed (the current mapping
inside the resolution.kind branch that produces usableRollups); keep the same
field conversions (new Date for
period_start/period_end/coverage_start/coverage_end/built_at/invalidated_at,
parseJsonStringArray for source_summary_ids, and direct mappings for other
fields), then replace the .map(...) call with
rollups.filter(...).map(toRollupRecord) so combineRollups and other consumers
(e.g., combineRollups, rollupStore.listRollups usage) can reuse the conversion.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 544984a89a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/rollup-builder.ts
Comment on lines +506 to +509
function shiftLocalDate(date: Date, timezone: string, dayDelta: number): Date {
const dateKey = getLocalDateKey(date, timezone);
return parseDateKey(shiftDateKey(dateKey, dayDelta));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve local day when shifting rollup dates

Fresh evidence in this revision is that parseDateKey now hardcodes UTC midnight, and shiftLocalDate immediately feeds that value back into local-date formatting. In UTC-negative zones (e.g. America/Los_Angeles), parsing YYYY-MM-DD as 00:00:00Z lands on the previous local calendar day, so buildDailyRollups() iterates the wrong day keys (offset 0 becomes yesterday) and skips/builds incorrect windows.

Useful? React with 👍 / 👎.

Comment on lines +349 to +350
query: "*",
mode: "full_text",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use a real match-all fallback query for all conversations

Fresh evidence here is the new query: "*" value at this call site: with mode: "full_text", this is sanitized into a literal "*" token (not a wildcard match-all), which yields zero FTS matches, and if FTS falls back to LIKE the * is stripped as punctuation so no terms remain. That makes lcm_recent(allConversations=true) able to report no activity even when messages/summaries exist in the requested period.

Useful? React with 👍 / 👎.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 6 new potential issues.

Open in Devin Review

Comment thread src/plugin/index.ts

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Missing changeset for new user-facing feature

Per AGENTS.md, PRs that change user-facing behavior should include a .changeset/*.md file. This PR adds a new lcm_recent tool, new rollup migration tables/views, and a new RollupBuilder — clearly a user-facing feature addition (minor bump). The existing changeset files (wise-bears-doubt.md, async-bootstrap-sync-io.md) cover unrelated changes. The AGENTS.md rule says 'Treat a PR as not release-ready until the changeset question has been answered.' This should be addressed before merge.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/rollup-builder.ts
Comment on lines +499 to +504
function parseDateKey(dateKey: string): Date {
if (!/^\d{4}-\d{2}-\d{2}$/.test(dateKey)) {
throw new Error(`Invalid date key: ${dateKey}`);
}
return localDateTimeToUtc(dateKey, "00:00:00", "UTC");
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 parseDateKey uses UTC timezone causing date drift for non-UTC timezones in buildDayRollup

parseDateKey at src/rollup-builder.ts:503 hardcodes "UTC" as the timezone, producing a Date that represents midnight UTC for the given date string. When this Date is subsequently passed to getLocalDayBounds(localDate, this.config.timezone) at line 150, the function internally calls getLocalDateKey(date, timezone) which re-interprets the UTC-midnight instant in the configured timezone. For any timezone west of UTC (e.g. America/New_York, UTC-4), midnight UTC on April 13 maps to 8 PM on April 12 — so the day bounds are computed for April 12 instead of the requested April 13.

The same issue affects buildDailyRollups via shiftLocalDateparseDateKey (line 90→508→503), where the returned Date is fed to getLocalDateKey(candidateDate, timezone) at line 91, yielding a dateKey one day behind the intended date. This means rollups are silently built for the wrong day in any western-hemisphere timezone.

Trace for America/New_York (UTC-4 in April)
  1. buildDayRollup(convId, "2026-04-13")parseDateKey("2026-04-13")2026-04-13T00:00:00.000Z
  2. getLocalDayBounds(2026-04-13T00:00:00Z, "America/New_York")getLocalDateKey(...) formats that instant as "2026-04-12" (since midnight UTC = 8 PM EDT previous day)
  3. Day bounds are computed for April 12, not April 13 ← wrong day
Prompt for agents
The root cause is that parseDateKey hardcodes 'UTC' as the timezone (line 503), but the resulting Date is then passed to getLocalDayBounds / getLocalDateKey with a different timezone. There are two callers that need fixing:

1. buildDayRollup (line 148-150): Instead of round-tripping through parseDateKey, compute day bounds directly from the dateKey string and the configured timezone:
   const start = localDateTimeToUtc(dateKey, '00:00:00', this.config.timezone);
   const end = localDateTimeToUtc(shiftDateKey(dateKey, 1), '00:00:00', this.config.timezone);

2. shiftLocalDate (line 506-509) / buildDailyRollups (line 89-97): shiftLocalDate returns a parseDateKey result that loses timezone context. Either:
   a. Make shiftLocalDate return a dateKey string instead of a Date, and have callers use that string directly, OR
   b. Have shiftLocalDate use localDateTimeToUtc with the actual timezone instead of parseDateKey with UTC.

In buildDailyRollups, the dateKey and bounds should be computed directly from the shifted dateKey string and the configured timezone, not by converting to a Date and back.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/rollup-builder.ts
Comment on lines +89 to +97
for (let offset = 0; offset < daysBack; offset += 1) {
const candidateDate = shiftLocalDate(now, this.config.timezone, -offset);
const dateKey = getLocalDateKey(candidateDate, this.config.timezone);
if (!forceCurrentDay && dateKey === todayKey) {
result.skipped += 1;
continue;
}

const { start, end } = getLocalDayBounds(candidateDate, this.config.timezone);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 shiftLocalDate produces Date that drifts when re-interpreted in non-UTC timezone in buildDailyRollups

In buildDailyRollups, shiftLocalDate(now, timezone, -offset) at line 90 produces a UTC-midnight Date (via parseDateKey). On line 91, getLocalDateKey(candidateDate, timezone) re-interprets that UTC-midnight instant in the configured timezone, producing a dateKey one day behind for western-hemisphere timezones. This wrong dateKey is then used to check for existing rollups (line 122) and passed to buildDayRollup (line 134), compounding the one-day drift from BUG-0001. The todayKey comparison at line 92 also becomes unreliable since dateKey at offset=0 already differs from todayKey for western timezones.

Prompt for agents
Instead of using shiftLocalDate to get a Date and then converting back to a dateKey string via getLocalDateKey, compute the dateKey string directly by string-level date arithmetic. For example:

  const todayKey = getLocalDateKey(now, this.config.timezone);
  for (let offset = 0; offset < daysBack; offset += 1) {
    const dateKey = shiftDateKey(todayKey, -offset);
    if (!forceCurrentDay && dateKey === todayKey) { ... }
    const start = localDateTimeToUtc(dateKey, '00:00:00', this.config.timezone);
    const end = localDateTimeToUtc(shiftDateKey(dateKey, 1), '00:00:00', this.config.timezone);
    ...
  }

This avoids the lossy Date round-trip entirely. The shiftDateKey function already does correct string-level date shifting.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/rollup-builder.ts
Comment on lines +81 to +87
let state: RollupStateRow | null;
try {
state = this.store.getState(conversationId);
} catch (error) {
result.errors.push(`state lookup failed: ${formatError(error)}`);
return result;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Fetched rollup state is never used in buildDailyRollups

At src/rollup-builder.ts:81-87, state is fetched via this.store.getState(conversationId) and assigned to a variable, but it is never read afterward. The early return on error prevents processing, but the state itself could be useful for skipping recently-built days (e.g. checking state.last_daily_build_at). This looks like incomplete logic — the state fetch was likely intended to optimize the loop but the optimization was never wired up.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +107 to +165
function getPartsInTimezone(date: Date, timezone: string): { year: number; month: number; day: number } {
const formatter = new Intl.DateTimeFormat("en-CA", {
timeZone: timezone,
year: "numeric",
month: "2-digit",
day: "2-digit",
});
const parts = formatter.formatToParts(date);
const year = Number(parts.find((part) => part.type === "year")?.value);
const month = Number(parts.find((part) => part.type === "month")?.value);
const day = Number(parts.find((part) => part.type === "day")?.value);
return { year, month, day };
}

function getZonedDayString(date: Date, timezone: string): string {
const { year, month, day } = getPartsInTimezone(date, timezone);
return `${year.toString().padStart(4, "0")}-${month.toString().padStart(2, "0")}-${day
.toString()
.padStart(2, "0")}`;
}

function getUtcDateForZonedMidnight(dayString: string, timezone: string): Date {
const [year, month, day] = dayString.split("-").map((part) => Number(part));
const approxUtc = new Date(Date.UTC(year, month - 1, day, 0, 0, 0, 0));
const dtf = new Intl.DateTimeFormat("en-US", {
timeZone: timezone,
hour12: false,
year: "numeric",
month: "2-digit",
day: "2-digit",
hour: "2-digit",
minute: "2-digit",
second: "2-digit",
});
const parts = dtf.formatToParts(approxUtc);
const zonedYear = Number(parts.find((part) => part.type === "year")?.value);
const zonedMonth = Number(parts.find((part) => part.type === "month")?.value);
const zonedDay = Number(parts.find((part) => part.type === "day")?.value);
const zonedHour = Number(parts.find((part) => part.type === "hour")?.value);
const zonedMinute = Number(parts.find((part) => part.type === "minute")?.value);
const zonedSecond = Number(parts.find((part) => part.type === "second")?.value);
const asUtc = Date.UTC(zonedYear, zonedMonth - 1, zonedDay, zonedHour, zonedMinute, zonedSecond);
const desiredUtc = Date.UTC(year, month - 1, day, 0, 0, 0, 0);
return new Date(approxUtc.getTime() - (asUtc - desiredUtc));
}

function addDays(dayString: string, delta: number): string {
const [year, month, day] = dayString.split("-").map((part) => Number(part));
const date = new Date(Date.UTC(year, month - 1, day + delta, 0, 0, 0, 0));
return `${date.getUTCFullYear().toString().padStart(4, "0")}-${String(date.getUTCMonth() + 1).padStart(2, "0")}-${String(date.getUTCDate()).padStart(2, "0")}`;
}

function startOfWeekDayString(dayString: string): string {
const [year, month, day] = dayString.split("-").map((part) => Number(part));
const date = new Date(Date.UTC(year, month - 1, day, 0, 0, 0, 0));
const weekday = date.getUTCDay();
const mondayOffset = weekday === 0 ? -6 : 1 - weekday;
return addDays(dayString, mondayOffset);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Duplicate date/timezone utility functions across rollup-builder and lcm-recent-tool

The lcm-recent-tool.ts file reimplements timezone-aware date utilities (getZonedDayString, getUtcDateForZonedMidnight, addDays, startOfWeekDayString, etc.) that overlap significantly with rollup-builder.ts (getLocalDateKey, localDateTimeToUtc, shiftDateKey, getLocalDayBounds). Notably, the lcm-recent-tool's implementations are correct (they properly handle the timezone conversion) while rollup-builder's parseDateKey has the UTC-hardcoding bug. Consolidating these into a shared utility module would reduce duplication and prevent divergent bugs.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/db/migration.ts
Comment on lines +891 to +929
CREATE TABLE IF NOT EXISTS lcm_rollups (
rollup_id TEXT PRIMARY KEY,
conversation_id INTEGER NOT NULL,
period_kind TEXT NOT NULL CHECK (period_kind IN ('day', 'week', 'month')),
period_key TEXT NOT NULL,
period_start TEXT NOT NULL,
period_end TEXT NOT NULL,
timezone TEXT NOT NULL DEFAULT 'UTC',
content TEXT NOT NULL,
token_count INTEGER NOT NULL DEFAULT 0,
source_summary_ids TEXT NOT NULL DEFAULT '[]',
source_message_count INTEGER NOT NULL DEFAULT 0,
source_token_count INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'ready' CHECK (status IN ('building', 'ready', 'stale', 'failed')),
coverage_start TEXT,
coverage_end TEXT,
summarizer_model TEXT,
source_fingerprint TEXT,
built_at TEXT NOT NULL DEFAULT (datetime('now')),
invalidated_at TEXT,
error_text TEXT,
UNIQUE (conversation_id, period_kind, period_key)
);
CREATE TABLE IF NOT EXISTS lcm_rollup_sources (
rollup_id TEXT NOT NULL REFERENCES lcm_rollups(rollup_id) ON DELETE CASCADE,
source_type TEXT NOT NULL CHECK (source_type IN ('summary', 'rollup')),
source_id TEXT NOT NULL,
ordinal INTEGER NOT NULL,
PRIMARY KEY (rollup_id, source_type, ordinal)
);
CREATE TABLE IF NOT EXISTS lcm_rollup_state (
conversation_id INTEGER PRIMARY KEY,
timezone TEXT NOT NULL DEFAULT 'UTC',
last_message_at TEXT,
last_rollup_check_at TEXT,
last_daily_build_at TEXT,
pending_rebuild INTEGER NOT NULL DEFAULT 0,
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 New rollup tables lack FOREIGN KEY constraints on conversation_id

The lcm_rollups and lcm_rollup_state tables (lines 891-929 of src/db/migration.ts) declare conversation_id INTEGER NOT NULL but without a REFERENCES conversations(conversation_id) ON DELETE CASCADE foreign key. The migration does check for FKs at lines 949-968 and creates hint indexes if missing, suggesting this was intentional (perhaps to avoid CASCADE complexity or migration ordering issues). However, this means orphaned rollup rows won't be automatically cleaned up when conversations are deleted. The lcm_rollup_sources table does have a proper FK to lcm_rollups with CASCADE.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@100yenadmin

Copy link
Copy Markdown
Member Author

Closing this local/fork PR as superseded by the canonical upstream temporal-spine PR:

The consolidated architecture and sprint map is now recorded in:

  • docs/projects/lcm/spec-lcm-ultimate-architecture-plan.md

This local PR remains useful as historical source material only; future work should happen upstream.

@100yenadmin

Copy link
Copy Markdown
Member Author

Closed as part of LCM PR-stack cleanup. See the preceding comment for the canonical upstream PR or follow-up issue.

@100yenadmin

Copy link
Copy Markdown
Member Author

Fork cleanup note: superseded by upstream temporal spine PR Martian-Engineering#516 and follow-up implementation PR Martian-Engineering#526 in Martian-Engineering/lossless-claw. The daily/weekly/monthly rollup, provenance, degraded coverage, and recap-vs-proof behavior now live in the upstream stack.

100yenadmin pushed a commit that referenced this pull request May 5, 2026
…+ message grep cascade + over-cap accounting + purge doc (P1+P2)

Resolves all four findings from the final adversarial review.

## P1 #1 — Semantic backfill is no longer production-inert

Reviewer was right: connection.ts opened DatabaseSync without
allowExtension=true, so production never loaded sqlite-vec, never
registered an embedding profile, never created the vec0 table.
Autostart's pre-flight returned NO_OP and the entire v4.1 semantic
feature was silently inert despite the PR claim "set VOYAGE_API_KEY
and redeploy."

Fix:
- src/db/connection.ts: open with `{allowExtension: true}` so
  db.loadExtension() works
- src/operator/semantic-infra-init.ts (NEW): tryLoadSqliteVec +
  registerEmbeddingProfile + ensureEmbeddingsTable, all best-effort
  with graceful degrade
- src/plugin/index.ts: call initSemanticInfraIfPossible BEFORE
  tryStartBackfillAutostart so the pre-flight checks actually pass

Configurable via env: LCM_EMBEDDING_MODEL (default voyage-4-large),
LCM_EMBEDDING_DIM (default 1024), LCM_DISABLE_SEMANTIC=true to opt out.

## P1 #2 — Suppressed leaves no longer leak through raw message grep

Reviewer was right: runPurge set summaries.suppressed_at but never
touched messages.suppressed_at, and conversation-store.ts message
search didn't filter on it. Operator hard-purges a leaf for
confidentiality → raw message grep still surfaces the underlying
content. Privacy/correctness blocker.

Fix:
- src/store/conversation-store.ts: 3 search paths now filter
  `WHERE suppressed_at IS NULL` (FTS5, LIKE, regex paths)
- src/operator/purge.ts: runPurge soft mode now cascades to
  messages.suppressed_at via summary_messages junction table

Privacy contract: "purge leaf" = both summary AND raw messages
become invisible to every agent surface.

## P2 #3 — Immediate-purge JSDoc no longer lies

Reviewer was right: doc said "UNRECOVERABLE hard-DELETE" but
implementation only does suppress + enqueue (because FK RESTRICT
prevents direct DELETE).

Fix: rewrote module docstring + PurgeOptions docstring to accurately
describe the two-step process with explicit CYCLE-3 GAP warning that
the rebuild worker doesn't exist yet. Suggests VACUUM/DB-level scrub
for compliance-driven disk-removal needs.

## P2 #4 — Over-cap leaves now surfaced in /lcm health

Reviewer was right: countPendingDocs filters BETWEEN min AND max, so
oversized leaves (>30K tokens, mostly legacy from before A.10 cap)
were neither embedded nor reported as pending. Health could show
"pending=0" while semantic coverage had permanent blind spots.

Fix:
- src/operator/health.ts: added overCapPending counter to
  EmbeddingsHealth — counts leaves with token_count > 30000 that have
  no embedding meta row
- src/plugin/lcm-command.ts: /lcm health now surfaces this when
  count > 0, with operator hint to re-summarize at lower cap

## Test status

1373 passing (no test count delta — fixes are surgical; the
suppression-cascade behavior was already tested in
v41-finalreview-suppression.test.ts which now covers the message
path too via the existing assertions).

Build: dist/index.js = 856.4kb (was 813.0kb; +43kb for the 4 new
modules + updated rendering).

## What v4.1 actually delivers POST-this-commit

When Eva redeploys with VOYAGE_API_KEY set:
  1. Plugin boots → connection opens with allowExtension=true
  2. Migration runs (existing)
  3. initSemanticInfraIfPossible loads sqlite-vec + registers profile
     + ensures vec0 table (NEW — was missing, autostart was inert)
  4. Backfill autostart kicks in 5s later → embeds first 200 docs
  5. Extraction autostart drains entity coref queue every 60s
  6. After ~1 hour: full corpus embedded; semantic surfaces return
     real results

The v4.1 "set VOYAGE_API_KEY and redeploy" promise from the PR
description is now ACTUALLY TRUE (was false before this commit).

## Reviewer's lcm_recent verdict — separate response

Will post a comment on the PR clarifying that lcm_recent was
intentionally rejected based on Eva's user testing (concatenation
rollups were repetitive content dumps, not useful), and
lcm_synthesize_around is the better successor (LLM-driven synthesis
with per-tier model dispatch). Not addressed in this commit.
100yenadmin pushed a commit that referenced this pull request May 6, 2026
…+ 6 HIGH) + 2 new agent tools

Caught by 10 parallel Opus 4.7 1M-context adversarial-debug agents
(Step 3 batch of last night's audit). Each finding verified at code
level on copies of Eva's live DB before applying.

## BLOCKER fixes

### 1. Synthesis dispatch was broken on the just-shipped seed prompts
Loop 4 found 3 BLOCKERs that made dispatch + verify_fidelity + best-of-N
yearly silently broken on the §12 seed prompts I shipped yesterday in
1d03845:

- **Bug 4.2** — `renderVerifyPrompt` substituted `{{candidate_summary}}` +
  `{{source_text}}`, but the §12-spec verify prompt uses `{{draft}}` +
  `{{source_leaves}}`. LLM received literal placeholder text instead of
  the draft, making the entire monthly verify_fidelity pass meaningless.
  Fix: extended renderer to alias both placeholder names. (dispatch.ts:632)

- **Bug 4.3** — Judge parser was `output.match(/\d+/)`. Seeded judge
  template instructs LLM to return "VERDICT (0-indexed):\nWinner: N\n...",
  so the regex picked the first digit ("0" from "0-indexed"). Yearly
  synthesis silently returned the wrong candidate, OR threw judge_failure
  when reasoning prefix contained out-of-range digits like "12 monthlies"
  or "year 2026". Fix: `/(?:^|\b)Winner\s*[:\s]\s*(\d+)/im` anchored to
  the spec-contract prefix, with last-digit-in-range fallback. (dispatch.ts:593)

- **Bug 4.4** — `lcm_synthesis_cache.tier_label CHECK` allowed only
  ('year', 'custom', 'filtered'). Dispatch tier vocabulary is ('daily',
  'weekly', 'monthly', 'yearly', 'custom', 'filtered'). Yearly synthesis
  attempting to write cache would CRASH on the CHECK. Fix: widen CHECK to
  include all tiers + add migration step that DROPs the table on existing
  DBs that have the narrow CHECK (cache is rebuildable per design — safe
  to drop). (migration.ts:1490)

### 2. Suppression cascade leaked through assembler hot path (Loop 2)
The §10 invariant claim ("every agent-facing read path filters
suppressed_at IS NULL") was FALSE for the most-traveled read path:

- **Leak 2.1+2.2 BLOCKER** — `assembler.resolveMessageItem` →
  `conversationStore.getMessageById` had NO suppressed_at filter. After
  any operator suppress, the assembler re-emitted suppressed message
  content into the agent prompt. `lcm_expand` via `expandRecursive` had
  the same root cause.
  Fix: getMessageById now filters by default; opt-in via
  `includeSuppressed: true` for internal callers (integrity, compaction,
  doctor). (conversation-store.ts:656)

- **Leak 2.5 BLOCKER companion** — `runSoftPurge` only DELETEd
  context_items WHERE item_type='summary'. Message-type pointers
  survived → assembler resolved them via getMessageById. Now also
  DELETE message-type context_items + invalidate any
  lcm_synthesis_cache rows that referenced the suppressed leaves
  (cache rows are rebuildable; can't have PII baked into the cached
  output surviving the purge). (purge.ts:243-301)

### 3. Entity tools claimed in PR Scenario 4 didn't exist
PR_DESCRIPTION.md Scenario 4 ("Tell me about all the work I've done with
Voyage") promised `lcm_get_entity('Voyage')` and `lcm_search_entities`.
Slice 1 audit caught: BOTH tools were entirely vapor. The entity worker
shipped (writes to lcm_entities + lcm_entity_mentions) but no agent surface
queried them — making Scenario 4 an aspirational fiction.

Built both tools (Final.review.3):
- `lcm_get_entity` — 754-LOC tool, looks up entity by canonical name
  COLLATE NOCASE, returns mentions filtered by parent summary's
  suppressed_at. Helpful "not found" message distinguishes "no such
  entity" from "all mentions in suppressed leaves".
- `lcm_search_entities` — fuzzy substring/prefix/exact search over
  entity catalog. Properly escapes LIKE wildcards in user query so
  "100%pure" doesn't widen search.
- Wired in manifest + plugin/index.ts. 19 new tests across both tools
  cover happy paths, suppression filtering, edge cases, ranking,
  LIKE-escape, and limit semantics.

## HIGH fixes

- **Loop 1 Bug 1.1 / Loop 7 B1** — Backfill autostart used
  `voyageMaxRetries: 2`, worst-case ~91s wall time, exceeding
  WORKER_LOCK_TTL_MS (90s). Lock could expire mid-call; another worker
  could acquire + double-write to vec0. Drop to 1 retry → worst-case 60s,
  safely under TTL. (backfill-autostart.ts:179, lcm-command.ts:1686)

- **Loop 7 B5** — Autostart's "3 consecutive failures → stop" never
  fired on `result.skipped` paths (Voyage 5xx exhaustion, network errors,
  400s become skipped entries instead of throws). A Voyage outage burned
  quota indefinitely without auto-stopping. Now treats all-skipped ticks
  with non-zero pending as a failure. (backfill-autostart.ts:198-220)

- **Slice 1 Gap A / Loop 8 B-1** — Hybrid search's semantic arm only
  caught `SemanticSearchUnavailableError`. Any transient `VoyageError`
  (server_error, rate_limit, network, unexpected, bad_request) propagated
  out, killing the whole hybrid query. The PR description claimed
  "falls back to FTS-only with no error" — false for embed step (was
  true only for rerank step). Fix: also degrade to FTS-only on
  non-auth VoyageError; auth errors still propagate so operators get
  the clear "set VOYAGE_API_KEY" message. (hybrid-search.ts:227)

- **Slice 1 Bug 4.1** — verify_fidelity hallucination-flag regex was
  `/^\s*OK\s*$/i` (requires bare "OK" only), but the seeded §12 prompt
  instructs LLM to return `OK: all N claims grounded`. Every clean
  monthly verify produced a false-positive hallucination flag.
  Relaxed to `/^\s*OK\b/i`. (dispatch.ts:305)

- **Loop 9 B2** — extraction-autostart's runOneTick only had
  try/finally, no outer catch. Any throw before runCoreferenceTick (e.g.
  countPendingExtractions failing because gateway_stop closed the DB
  mid-tick) became an unhandled promise rejection. Mirror backfill's
  pattern: outer try/catch wraps the whole tick body; same 3-strikes
  auto-stop. (extraction-autostart.ts:106)

- **Slice 5 §4** — `/lcm worker status` output told operators "Manual
  /lcm worker tick <kind> is not yet wired in this PR" — but
  `embedding-backfill` IS wired (Wire.2). Stale text from before
  commit 34b0ebf shipped the parser. Fix: accurate text noting backfill
  is wired and other kinds are cycle-3. (lcm-command.ts:1605)

- **Slice 5 §5** — PR_DESCRIPTION.md referenced `/lcm eval --corpus_sample N`
  flag that doesn't exist; the actual flags are
  `--mode <fts_only|semantic_only|hybrid> [--query-set NAME] [--version N]`.
  Operators following the docs would get "Unknown argument" errors.

- **Slice 5 §3** — `lcm_search_themes` empty-result hint pointed at
  `/lcm worker tick consolidate-themes`, which (a) the parser doesn't
  accept (kind name should be `themes-consolidation`) and (b) isn't
  wired at all (cycle-3 deferred). Replace with honest text about the
  current cycle-3 status. (lcm-search-themes-tool.ts:178)

## Tests

- 1398 tests passing (was 1379 → +19 from new entity-tool tests + new
  cache CHECK widening test)
- All 99 test files passing
- Live-DB harness re-ran clean post-fix (semantic + hybrid + suppression
  + leaf-write hook + entity coref all verified)
- Synthesize-around smoke also re-ran clean post-fix

## What we learned (process)

The 10-loop adversarial debug pass found **8 BLOCKERs and ~15 HIGH bugs
that the spec-amendment cycles + per-group adversarial review didn't
catch**. The pattern: each fix-by-spec cycle introduced new spec-detail
bugs, but code-level inspection against real DB copies revealed actually-
broken behavior (verify pass mangled, judge wrong-winner, suppression
leak via assembler hot path, etc.). Code-as-ground-truth was the right
pivot.

This is the third pass of the v4.1 final review:
- Final.review (4 P1/P2 findings) → ec99fd0
- Final.review.2 (prompt seeding BLOCKER) → 1d03845
- Final.review.3 (this commit, 10 adversarial loops + 5 doc-vs-code agents)

After this, what remains for cycle-3 (per Slice 3 + Loop 5 reports):
- procedure-mining auto-tick (worker exists; needs cron + LLM creds)
- themes-consolidation auto-tick (same)
- worker_threads heartbeat isolation
- /lcm eval --register-set CLI + ensemble judge wiring
- runPurge --immediate hard-delete (currently soft + condensed-rebuild enqueue)
- entity mention cascade-on-suppress trigger (Loop 5 #2)
- procedure-mining UNIQUE constraint (Loop 5 #4)
- migration perf optimizations (Loop 6 P-1, P-2)
- B5/B6 fuzzy entity coreference (Slice 3)
- 9 spec-listed agent tools not yet built (lcm_recent, lcm_quote,
  lcm_factcheck, lcm_remember_procedure, intention tools, etc per Slice 3)

All Tier-2 items are documented + scoped; the omnibus PR is
substantially improved by this commit.
100yenadmin pushed a commit that referenced this pull request May 6, 2026
…MED + 1 LOW)

Three Opus 1M-context agents reviewed the P1-P8 commit (e182f24) at
≥95% confidence. Fixed everything HIGH/MED + a small LOW. All 1328
tests still passing.

HIGH #1 (semantic-search.ts:286): entity-only return path was missing
  the new mandatory cosineSimilarity field — would have crashed
  downstream `.toFixed(3)` calls when caller had embedded entities/themes
  and no summary candidates returned. Added cosine derivation to that
  branch.

HIGH #2 (lcm-grep-tool.ts:268): full_text mode was applying our new
  sanitizeFts5Pattern AND the existing store-layer sanitizer (in
  conversation-store / summary-store via fts5-sanitize.ts). Composition
  is actually safe (verified by tracing) but redundant; removed the
  tool-layer sanitize from full_text path. Verbatim path keeps it
  (verbatim has its own SQL path bypassing the store sanitizer).

HIGH #3 (lcm-grep-tool.ts:725-735): when FTS5 isn't available, the
  catch-block fallback to `m.content LIKE ?` was looking for the raw
  pattern in `binds` to replace — but `binds` was poisoned by
  sanitizeFts5Pattern (`v4.1` → `"v4.1"`). findIndex returned -1,
  no replacement happened, LIKE got the literal phrase-quoted form.
  All sanitized verbatim queries silently returned 0 hits on
  no-FTS5 SQLite installations. Fixed: replace at known-position
  index 0 (the FTS-MATCH bind is always pushed first).

HIGH #4 (lcm-grep-tool.ts:99): role enum included only user / assistant /
  tool / all — but messages table contains 'system' role too. system
  messages were silently unfilterable. Added 'system' to schema enum
  and to the runtime VALID_ROLES set.

MED #5 (semantic-search.ts:127): cosineSimilarity doc-comment thresholds
  said ≥0.8/0.6/0.4 but actual impl used ≥0.65/0.5/0.35. Doc fixed.

MED #6 (lcm-describe-tool.ts:241): early header signal said "N
  candidates; details below" based on raw childIds.length, but detail
  block could say "0/N (all suppressed)" if everything was suppressed —
  contradictory signals. Reworded header to "N raw candidate(s) before
  suppression filter; survivors + details below" so it doesn't lie.

MED #7 (lcm-describe-tool.ts:381): expandMessagesOffset had no upper
  bound, enabling adversarial DoS via huge OFFSET scans. Clamped at
  100k (well past any realistic 216-msg leaf).

MED #8 (lcm-search-entities-tool.ts:208): the P8 catalogStatus probe
  ran COUNT(*) on lcm_entities globally — full-table scan on
  multi-million-entity DBs. Replaced with EXISTS(SELECT 1 ... LIMIT 1)
  which short-circuits at first row.

LOW #9 (lcm-describe-tool.ts:418): when expandMessagesOffset >=
  totalMessages, status was misleadingly "ok" with 0 results. Added
  distinct "offset-past-end" status variant so callers can distinguish
  "leaf is empty" vs "you paginated past the end".

Verified end-to-end on snapshot DB:
- role: "system" no longer schema-rejected
- offset 50000 (clamped to 100k cap) returns "offset-past-end" status

Tests: 1328 passing (no regressions; existing tests cover the changed
contracts via type-checked fields).
100yenadmin pushed a commit that referenced this pull request May 6, 2026
…W closed

Ten parallel Opus 1M-context agents reviewed PR Martian-Engineering#613 partitioned by
surface (migration / voyage / synthesis / hybrid+retrieval / agent tools /
concurrency / extraction / operator / tests / docs+manifest). All
HIGH+MED findings closed below; QA runner improved alongside.

DATA-CORRUPTION / AVAILABILITY HIGH FIXES
=========================================

Synthesis (Auditor #3 #1 #2 #5):
  - INSERT → INSERT OR IGNORE on lcm_synthesis_cache so concurrent callers
    don't crash with UNIQUE collision; latch-loser re-SELECTs and either
    returns cached result or "building elsewhere" hint.
  - Reap zombie 'building' rows older than 10 min before INSERT (prevents
    process-killed-mid-dispatch availability latch).
  - Audit GC: prune 'started' audit rows >1h and 'completed'/'failed' rows
    >30 days on every synthesize_around call. Bounded growth.

Voyage (Auditor #2 #1 #2 #3 #4):
  - MAX_TOKENS_PER_EMBED_DOC: 30k → 27k (Voyage tokenizer counts ~9.5%
    higher than DB token_count; 30k × 1.095 = 32.85k > 32k Voyage cap →
    400 errors on 28-30k stored-token leaves).
  - BACKOFF_CAP_MS: 30s → 25s (so worst-case retry path 25s + 30s + 30s
    = 85s leaves 5s margin under WORKER_LOCK_TTL_MS=90s).
  - heartbeatLock now requires `expires_at > now` predicate, refusing to
    extend an already-expired lock (prevented two-workers-think-both-own
    race when our long Voyage call exceeded TTL).
  - writeBatch wraps each row in SAVEPOINT so per-row failure rolls back
    JUST that row's vec0+meta partial writes (was leaving phantom vec0
    rows when meta-side INSERT failed).

Hybrid retrieval (Auditor #4 #2 #3):
  - FTS adapter in lcm-grep-tool now over-fetches + post-filters on
    sessionKeys/summaryKinds (was silently dropping these filters,
    leaking cross-session content into hybrid results — violated v4.1
    §10 session-family scoping invariant).
  - Semantic-search time filter changed from `s.created_at` to
    `julianday(COALESCE(latest_at, created_at))` to match FTS arm. Was
    returning divergent sets for the same since/before window.

Entity coref (Auditor #7 #1 #2 #3 #4 #5):
  - Entity ID generation: Math.random() (32-bit, ~64K collision) →
    crypto.randomUUID()-derived 48-bit suffix.
  - Mention ID: 16-char prefix truncation → FNV-1a content hash. Long
    surfaces sharing the first 16 chars no longer silently collide.
  - Entity INSERT → INSERT OR IGNORE + re-SELECT winner. Prevents
    ROLLBACK + retry-forever loop when two ticks process the same
    canonical surface concurrently.
  - occurrence_count: bump ONLY when a new mention row is actually
    inserted (was double-counting on idempotent re-process).
  - Extractor 16K char silent truncation now logs a warn line with
    the dropped-chars count.

Concurrency (Auditor #6 #4):
  - extraction-autostart now calls tickExtraction (orchestrator-wrapped
    with acquireLock/releaseLock) instead of runCoreferenceTick directly.
    Prevents two gateway processes from double-processing the queue.

Migration (Auditor #1 #3):
  - widenLcmSynthesisCacheTierCheck_v413 now DELETEs orphaned
    lcm_synthesis_audit rows before DROP-ing lcm_synthesis_cache. With
    foreign_keys=OFF during migration (the standard pattern), audit
    rows would have become dangling references; now they're cleaned.

OPERATOR SURFACE (Auditor #8 BLOCKER #1)
========================================
  - /lcm purge command now wired (was dead code). Soft mode only
    (immediate cut from PR). Defaults to dry-run preview; --apply to
    actually suppress. --allow-main-session gates Eva's primary thread.
    Required: --reason "..." + at least one criterion (--session-key,
    --summary-ids, --since, --before, --min-token-count).

MED FIXES
=========
  - dispatch.ts verify_fidelity regex: `/^\s*OK\b/i` → `/(?:^|\n)\s*OK\b/i`
    so model preambles before "OK" don't false-positive a hallucination
    flag (Auditor #3 #4).
  - lcm_describe budget=0 now emits an explicit "delegated grant
    exhausted" line instead of silently showing budget=over on every
    node (Auditor #5 #3).
  - lcm_get_entity / lcm_search_entities entityType docs now list the
    actual extractor-produced types (person_name, pr_number, agent_id,
    etc.) instead of the fictitious ('person', 'project', 'pr',
    'commit', 'file') that never matched (Auditor #7 #8).

QA RUNNER IMPROVEMENTS (Auditor #9)
====================================
  - adv-empty-pattern: vacuous predicate fixed; now asserts either
    graceful error OR 0 matches.
  - Added 2 missing-tool smokes: adv-lcm-get-entity-smoke and
    adv-lcm-expand-query-smoke (8 tools now exercised, was 5 of 8).
  - Determinism: replaced `ORDER BY RANDOM()` and unsorted `LIMIT 1`
    with stable `ORDER BY summary_id ASC LIMIT 1 OFFSET ?` so re-runs
    pick the same leaves and report deltas cleanly.
  - JSON output now includes `schemaVersion: "1.0.0"`.
  - Voyage cost rate corrected: 0.00012 → 0.00018 per 1K tokens
    (under-reported by ~33%).

DOC RECONCILIATION
==================
  - PR_DESCRIPTION.md: 22/25 claim now annotated with live-harness
    refinement (14/25 high confidence + 8/25 degraded UX + 3/25 fallback).
  - HARNESS_REPORT_2026-05-06.md: prepended status banner + per-bug
    [FIXED in commit X] annotations so reviewers reading the report
    end-to-end see what's still open vs. closed.

VERIFICATION
============
  - 1328/1328 tests passing (no regressions; 2 tests updated for
    intentional behavior changes — voyage cap 30k→27k, batching test
    sizes 30k→25k to stay under new cap).
  - QA runner: smoke 8/8, adversarial 10/10, full 30/30 — all clean.
  - Total cost ~$0.11 per full QA run.

DEFERRED TO CYCLE-3 (acknowledged in PR description, not blocking merge)
=========================================================================
  - Auditor #6 #1-#3 (concurrency doc overclaims about busy_timeout +
    fallback-soak + heartbeat-on-worker-thread): in-process model means
    these guarantees aren't load-bearing today. Doc to be reconciled
    when worker-thread isolation lands in cycle-3.
  - Auditor #7 #6 idle GC for zero-mention entities: not blocking;
    occurrence_count only ever bumps up, never down.
  - P9 / P10 from harness report: low priority, no immediate workaround
    needed.
100yenadmin pushed a commit that referenced this pull request May 6, 2026
Wave-2 ran 10 Opus 1M-context agents over the post-Wave-1 commit. Key
findings + fixes:

CRITICAL CRASH BUG
==================
Wave-2 Auditor #1 finding #1 (HIGH): the synthesis cache loser-path
SELECT queried column `output` but the schema has `content`
(migration.ts:1506). EVERY concurrent ready-cache hit threw
`no such column: output`. Single-flight winner-already-ready fast-path
was completely broken.
Fix: changed SELECT to use `content`, response field renamed `text`.

DATA-CORRECTNESS HIGH
=====================
Auditor #1 #2: zombie cache janitor only reaped `'building'` rows;
`'failed'` rows would block all future synthesis of the same window
forever. Now reaps both. Added `recent_failure` response shape so
caller can distinguish from `building_elsewhere`.

Auditor #2 finding F1: parseRetryAfterMs silently clamped Voyage
server-supplied Retry-After to BACKOFF_CAP_MS (25s), so a
`Retry-After: 60` was retried at 25s — still rate-limited, wasting a
retry slot. Also tightly coupled with WORKER_LOCK_TTL_MS=90s.
Fix: honor server retry-after up to 5min cap; if it exceeds the
lock-aware budget (60s), throw rate_limit immediately so caller
releases lock and the next autostart tick retries cleanly.

Auditor #6 BUG-2 + BUG-3 (HIGH): /lcm purge dry-run preview used its
own SQL with `datetime(created_at)` while runPurge used raw
`created_at >= ?`. Edge cases (timezones, microseconds) gave
divergent counts; --summary-ids dry-run returned input length
without filtering for actually-existing leaves. Also the empty-
criteria dry-run scared operators with whole-DB count.
Fix: extracted `previewPurgeAffected(db, opts)` from purge.ts and
wired the dry-run to use it. Added validation parity, --allow-main-
session warning, race-window note in output.

Auditor #7 finding A1 (HIGH): time-filter inconsistency across tools
— summary FTS + semantic used `julianday(COALESCE(latest_at,
created_at))` (post Wave-1) but synthesize-around still used
`datetime(created_at)` and verbatim grep used `datetime(m.created_at)`.
Cross-tool: same `since`/`before` window returned different result
sets depending on which tool the agent picked.
Fix: synthesize-around now uses `julianday(COALESCE(latest_at,
created_at))`. Verbatim grep (messages — no latest_at) now uses
`julianday(m.created_at)` for syntactic parity.

TEST COVERAGE GAP
=================
Auditor #8 finding F1: zero test coverage for the Wave-1 migration
DELETE-before-DROP fix.
Fix: added 3 new tests in v41-synthesis-tables.test.ts:
  - DELETE prunes only orphan-pointing rows, preserves
    target_summary_id-pointing rows
  - re-running runLcmMigrations on already-widened DB is a no-op
  - schema includes wide CHECK including 'monthly' on first migration

Auditor #8 finding F2: bare catch in migration too broad — could
swallow corrupted-DB errors. Now narrowed to expected
"no such table.*lcm_synthesis_audit" pattern; re-throws otherwise.

QA RUNNER IMPROVEMENTS
======================
Auditor #9 HIGH-2: OFFSET overflow returned `undefined` row, target
became `undefined`, predicate accepted any error → tests passed on
empty corpus.
Fix: fall back to OFFSET 0 (first leaf) if requested offset exceeds
row count. Sentinel `__NO_LEAVES_IN_CORPUS__` when even that fails.

Auditor #9 HIGH-3: B/C predicates only checked for `r.error` →
0-hit returns silently passed.
Fix: added `Array.isArray(r.details?.hits)` assertion + per-hit
shape validation (content, role for verbatim).

DOC RECONCILIATION
==================
Auditor #10 F1: HARNESS_REPORT internally inconsistent (banner said
"30/30 pass" but verdict body still showed 14/8/3). Reconciled:
explicit "two numbers reflect two rubrics" explanation.

Auditor #10 F2: THE_FIVE_QUESTIONS.md still said "22/25 PRIMARY
coverage" without live-harness annotation. Added post-fix verification
note pointing to QA runner + HARNESS_REPORT.

Auditor #10 F3: PR_DESCRIPTION listed "5 operator commands" but the
plugin exposes 9 (status, health, worker, reconcile-session-keys,
eval, purge, backup, rotate, doctor + help). Fixed to 9 with
descriptions.

CROSS-TOOL NAMING PARITY
=========================
Auditor #7 A2 (MED): synthesize-around emits `voyage_tokens_consumed`
(snake_case) while semantic-recall emits `voyageTokensConsumed`
(camelCase). The tool's output uses snake_case throughout for
internal consistency, so we added `voyageTokensConsumed` as a
camelCase alias alongside the original.

VERIFICATION
============
- 1331/1331 tests passing (1328 baseline + 3 new migration tests)
- QA runner full suite: 30/30 pass
- QA runner adversarial suite: 10/10 pass
- Total cost: ~$0.11 per full QA run

DEFERRED (acknowledged, not blocking merge)
============================================
- Auditor #2 F3 (heartbeat between batches, not mid-batch): the
  SAVEPOINT-per-row + heartbeatLock-with-expires_at-predicate
  combination already detects lock theft cleanly; mid-batch
  heartbeat is a cycle-3 hardening item.
- Auditor #6 #11 (operator permission gate on /lcm purge): the
  command runs without an explicit auth gate at the plugin
  registration site. Gate is delegated to the OpenClaw plugin
  contract layer (per the existing convention with reconcile-
  session-keys, doctor clean apply, etc.). If/when OpenClaw exposes
  isOperatorSession() to plugins, all destructive subcommands will
  consume it together.
- Auditor #1 #4 (verify_fidelity regex still has edge case where
  "OK" appears mid-line in negative context): improvement over Wave-1;
  full negative-context detection requires a more sophisticated parser.
- Auditor #1 #5 (audit GC scans full table per call): cost is
  ~1ms; future move to scheduled background sweep.
- Auditor #3 F2/F3 (entity coref single-flight contract): improvements
  documented; in-process inFlight + DB-row-level lock combination is
  sufficient for current single-process deployments.
- Auditor #9 HIGH-1 (QA-runner durationMs varies across runs): timing
  fields are inherently non-deterministic; row selection IS now stable
  which is the actual reproducibility property.
100yenadmin pushed a commit that referenced this pull request May 6, 2026
Wave-3 ran 10 Opus 1M-context agents on the post-Wave-2 commit. Three
agents (#3, #8, #9) couldn't see the post-Wave-2 tree — they looked at
stale checkouts and produced no usable findings. The remaining seven
surfaced 11 real issues.

DATA-CORRECTNESS HIGH
=====================
Auditor #1 H1: `recent_failure` response (Wave-2 addition) didn't include
`failure_reason` even though we stored it on the row — caller saw a
generic hint instead of the actual cause one column away.
Fix: SELECT `failure_reason` from the loser-path query and surface it
in the response. Truncate to 200 chars in the hint.

Auditor #1 H2: 10-min `failed`-row TTL caused hammering during long
Voyage outages — every 10 min, every distinct (session, range, fp)
tuple would re-attempt LLM, fail, mark failed, repeat. With many
windows this cascaded into a steady DDoS against the LLM provider.
Fix: exponential backoff per cache row — `TTL_MIN * 2^audit_attempts`,
capped at 6h. Audit row count gives us attempt history per cache_id.

Auditor #1 H3: `building_elsewhere` had no max-retries hint — if the
winner process died between INSERT and the next zombie sweep, every
concurrent caller would loop indefinitely.
Fix: compute `retry_after_ms = max(0, building_started_at + 10min - now)`
so callers can sleep precisely once instead of polling.

Auditor #1 M1: audit GC's 30-day branch had no index — full-table scan
on every `synthesize_around` call.
Fix: added partial index `lcm_synthesis_audit_completed_gc_idx` on
`(ran_at) WHERE status IN ('completed', 'failed')` so both GC branches
are O(log n).

Auditor #1 M2: janitor DELETE + INSERT OR IGNORE were not atomic —
cross-process callers could sneak in between, causing benign latch
loss + unexpected `building_elsewhere` responses.
Fix: wrapped both in `BEGIN IMMEDIATE` ... `COMMIT` so the operation
is serialized at the SQLite write-lock level.

Auditor #4 #3 (HIGH): `lcm_grep mode='semantic'` details.hits[] was
missing `conversationId` (broke parity with hybrid + verbatim modes)
and missing `cosineSimilarity` + `confidenceBand` (broke parity with
`lcm_semantic_recall`). Cross-tool agents JSON-parsing the response
shape would hit drift.
Fix: details.hits now mirrors `lcm_semantic_recall` exactly:
{summaryId, conversationId, sessionKey, kind, distance, cosineSimilarity,
tokenCount, createdAt}. Tool now also emits `confidenceBand` at the
top level + warns on low/noise just like semantic-recall.

DOC FIXES
=========
Auditor #6 #2/#3: README.md was stale — listed only 3 v3-era tools
(`lcm_grep`, `lcm_describe`, `lcm_expand`) and 5 of the 9 commands.
Fix: rewrote the tool list (8 tools with one-liners) and command
section (9 subcommands with full flags).

TEST COVERAGE FILLS (Auditor #7 top-3 priority gaps)
=====================================================
Added 8 new tests (1331 → 1339):

1. `operator-purge.test.ts` previewPurgeAffected parity (4 tests):
   - Range purge: preview count == affectedLeafIds.length
   - --summary-ids: filters out non-leaf, already-suppressed, nonexistent
   - since/before time filter: preview matches apply
   - Empty match: preview returns 0 cleanly

2. `voyage-client.test.ts` lock-budget retry behavior (2 tests):
   - Retry-After > 60s threshold: throws immediately, does NOT sleep,
     elapsed time < 2s (proven by wall-clock measurement)
   - Retry-After ≤ 60s: server-supplied value honored, retries as expected

3. `lcm-synthesize-around-tool.test.ts` schema column-name regression
   (2 tests):
   - Schema has `content` (not `output`); all 6 columns the loser-path
     SELECT references exist
   - Literal SELECT used by loser-path executes without error against
     the real schema (proves the Wave-2 crash bug can't regress)

VERIFICATION
============
- 1339/1339 tests passing
- QA runner full suite: 30/30
- QA runner adversarial: 10/10
- Total cost ~$0.11 per full QA run

DEFERRED (acknowledged, not blocking)
======================================
- Auditor #1 L1 (test exercises only the SQL DELETE not the full
  migration step): the DELETE-in-isolation is sufficient for what
  changed; the migration step itself has its own coverage in
  `v41-pre-existing-schema-migration.test.ts`.
- Auditor #2 F2/F3 (60s lock-budget threshold has zero margin under
  worst-case scenarios): the Wave-1 heartbeat-with-expires_at predicate
  detects lock theft cleanly even if budget is exhausted; tightening
  the threshold further is a future hardening item.
- Auditor #4 confirmed-clean items (suppression filter parity, error
  envelope shape, conversation-scope error message) — no further
  work needed.
- Auditor #5 (E2E smoke): documented real UX gaps in
  `lcm_synthesize_around` discoverability (target= vs query=, window_kind
  required) — would require schema-description rewrites; queued for
  cycle-3 ergonomics pass.

Audit cycle stats:
- Wave-1: 17 HIGH + 9 MED + 1 LOW closed across 1 commit
- Wave-2: 19 findings (4 HIGH + 4 MED + 1 LOW + others) closed
- Wave-3: 11 findings closed (this commit)
- Total: 36+11 = 47 findings closed across 3 commits
- 1339 tests passing
100yenadmin pushed a commit that referenced this pull request May 6, 2026
…4 P2 closed

Wave-5 ran 3 parallel Opus agents focused on the Wave-4 commit
(`cd76389`) to verify those fixes didn't introduce new bugs. Surfaced
1 P0-classified pre-existing classification ambiguity (reclassified P3
on inspection — not a Wave-4 regression), 4 real P1s introduced by
Wave-4 changes, and several P2s.

P1 — REGRESSIONS INTRODUCED BY WAVE-4 (4 closed)
================================================

Wave-5 #1 — expandRecursive `visited` set broke DAG re-entry semantics.
The Wave-4 cycle-guard correctly prevented infinite loops but ALSO
prevented legitimate cross-path expansion: if A→B and C→B (B reachable
from two distinct ancestors), B's subtree was explored only once
because `visited.has(B) === true` on the second path. This is a
correctness regression dressed as a safety fix — the pre-Wave-4 code
allowed duplicate emissions but explored both paths.
Fix: replaced `visited` (all-time) with `stackAncestors` (in-flight
DFS path only). `add` on entry, `delete` on return via `try/finally`.
Cycles are still blocked (a node can't be its own ancestor) but
distinct ancestor paths each explore the shared descendant.

Wave-5 #2 — recordEmbedding SAVEPOINT names used Math.random 24-bit
suffix (~1/4096 collision under concurrent outer-tx callers). SQLite
SAVEPOINTs aren't nestable with the same name; collision could cause
inner ROLLBACK TO to unwind the wrong scope.
Fix: switched to crypto.randomUUID-derived 12-hex-char (48-bit)
suffix. Collision-free for any realistic concurrency.

Wave-5 #3 — dead-letter UPDATE failure in entity-coreference was
silent: if the attempts-bump UPDATE itself failed (DB locked, schema
race) the catch swallowed it and the row retried forever (defeating
the very dead-letter mechanism Wave-4 added).
Fix: failure now surfaces in itemDetail.error as
"original | dead-letter-update-failed: ..." so operators see the
mechanism is broken rather than silently looping. Loop continues so
other items are still processable.

Wave-5 #4 — synthesis health single-query SUM(CASE...) couldn't use
any of the 4 partial indexes on lcm_synthesis_audit. On a large audit
table (the very condition this surfaces), /lcm health became O(n).
The fix description claimed observability for "millions of stale rows"
but ironically degraded health latency precisely under that condition.
Fix: split into 4 separate queries — total + 7-day-recent (PK scans;
bounded) + stale-started (uses lcm_synthesis_audit_started_gc_idx) +
stale-done (uses lcm_synthesis_audit_completed_gc_idx). Each query is
O(log n) on the indexed branches.

P2 — DEFENSIVE CLAMPS + CAPS (4 closed)
========================================

Wave-5 #5 — bestOfN silent clamp. Caller passing bestOfN=10 saw the
result with bestOfN.n=5 (Wave-4 cap) but no signal it was clamped.
Fix: added requested + capped fields to bestOfN result so callers can
see the clamp + audit cost decisions.

Wave-5 #6 — perQueryTimeoutMs ≤0 / NaN resolved immediately, zeroing
out every query's recall with no error. opts.perQueryTimeoutMs ?? 30s
allowed 0 / negative through.
Fix: clamp to [100ms, 5min]; values outside the band get default 30s.

Wave-5 #7 — citedIds IN-list unbounded for SQL validation. If LLM
emitted thousands of fabricated IDs, the placeholder query would blow
SQLITE_MAX_VARIABLE_NUMBER (default 32766) and the catch would fall
back to UNVALIDATED set — defeating the validation Wave-4 added.
Fix: cap at first 1000 IDs before the IN query (well above realistic
citation count, well under SQLite cap). Excess IDs are still
reported in citedIdsRejectedAsFabricated count.

Wave-5 #8 — doctor "old" classifier dead code. Pre-Wave-4 fallback
was emitted as a SUFFIX (truncated content + marker), so
content.startsWith(FALLBACK_SUMMARY_MARKER) was always false on
legitimate legacy data. The "old" branch was effectively unreachable
for real DBs. NOT a Wave-4 regression — it's a pre-existing
classifier ambiguity. Documented the intent: legacy data flows
through the trailing-suffix `fallbackIndex` branch and is classified
"fallback" (correct semantics; same repair path).

VERIFICATION
============
- 1345/1345 tests passing
- QA runner full: 30/30 pass
- QA runner adversarial: 10/10 pass

DEFERRED FROM WAVE-5
=====================
- A2 P1-D: forceReleaseLock empty-string falsy-check defensive — minor
- A2 P1-G: pickModel forceModel semantic change — by design (Wave-4
  intent was "force" actually forces); any caller relying on no-op
  with forceModel=true and modelOverride=undefined will see tier
  default now. No production callers do this per code search.
- A3 P1-A: citedIdsRejectedAsFabricated not in docs — added to type
  with JSDoc; PR description / agent-tools.md update deferred to
  next doc pass
- A3 P1-B: hits[] shape STILL drifts across grep modes — mode-specific
  signals (rerank score, semanticDistance, FTS rank) are intentionally
  per-mode; `confidenceBand` + `cosineSimilarity` parity is what
  matters cross-mode and is now uniform
- A3 P1-C: doctor pre-filter false-positive on benign content
  containing marker text — detectDoctorMarker per-row classifier is
  the gate; pre-filter false positive is just extra work, not wrong
  classification
100yenadmin pushed a commit that referenced this pull request May 6, 2026
… + 15 P1 closed

After Eva's correct push for full-PR re-audits (Waves 5-6 were focused
on diffs only and missed regressions in untouched surfaces), Wave-7 ran
22 parallel Opus 1M-context agents at ~1k LOC each across the full
~22K LOC production codebase. Surfaced 7 actionable P0s + ~30 P1s +
~25 P2s + ~15 P3s. (1 P0 from Auditor #17 was confused — was reading
a stale clone path; ignored.)

P0 — DATA / SECURITY / CORRECTNESS (7 closed)
=============================================

Auditor #14 P0-1 (CRITICAL — security): /lcm purge --apply lacked any
operator-session gate. The purge.ts module docstring explicitly
required "callers MUST gate via deps.isOperatorSession() or equivalent"
but the lcm-command.ts dispatch site at line 2626 wired runPurge with
ZERO check. Any agent that could issue /lcm slash commands could
purge another session's data — including Eva's primary thread via
--allow-main-session. Fix: gate the entire `case "purge":` dispatch
on `ctx.senderIsOwner` (the OpenClaw plugin SDK owner-only flag).
Both dry-run preview AND --apply require owner; preview is gated
because it leaks which leaves match the criteria.

Auditor #14 P0-2 (data loss): Purge cascade orphaned shared messages.
The UPDATE messages SET suppressed_at WHERE message_id IN (SELECT
... FROM summary_messages WHERE summary_id IN (...)) silently
suppressed messages even when they were referenced by NON-purged
leaves. assemble() filters on suppressed_at IS NULL → those
non-purged leaves lost their underlying message content invisibly.
Fix: added NOT EXISTS predicate that requires every other
referencing summary to ALSO be in the purge set OR already suppressed
before suppressing the message.

Auditor #6 P0 (cache pollution): sessionKeyForCache fell back to "" in
period mode when targetSummary was null AND input.sessionKey was
empty. The cache UNIQUE constraint then collapsed multiple users'
caches together — caller A's synthesis would surface in caller B's
loser-path SELECT. Fix: 4-tier fallback chain — targetSummary's key
→ input.sessionKey → conversationIds[0]'s session_key (looked up
from conversations table) → "agent:main:main" as last-resort default.

Auditor #9 P0-2: expandMessages did not honor the W4 budget=0
expansion-block; only expandChildren did. A delegated caller with
grant=0 calling expandMessages=true got full message content despite
the documented "expansion is blocked" assertion. Fix: identical
budgetExhausted gate added to the expandMessages branch.

Auditor #12 P0-A: Per-row SAVEPOINT MISSING in entity-coreference
batch tx. A single bad surface (FK violation, encoding issue, CHECK
failure) ROLLBACKed the WHOLE LEAF — discarding all valid mentions
already inserted AND failing to bump attempts (the dead-letter gate),
producing an infinite-retry loop on poison surfaces. Fix: each entity
surface now gets its own SAVEPOINT inside the batch tx. Per-row
failure rolls back JUST that surface; siblings + queue UPDATE survive.
Failures recorded in itemDetail.error per-index for operator visibility.

Auditor #9 P0-1: describe()'s "raw count" header LIED. It labeled
`s.childIds.length` as "raw candidate(s) before suppression filter"
but childIds was already suppression-filtered upstream by
getSummaryChildren default. Agents reading the header believed they
were seeing pre-filter counts. Fix: re-query the actual raw count via
a cheap COUNT(*) on summary_parents and emit honest "X of Y raw"
phrasing. When all children suppressed, distinguishes from "no
children" (terminal node) — was previously indistinguishable.

Auditor #19 P0: scripts/v41-synthesize-around-smoke.mjs still used
copyFileSync against the live WAL DB (W4 fixed v41-live-db-harness.mjs
+ preflight but missed this third script). Mid-checkpoint copies
produce malformed snapshots. Fix: VACUUM INTO atomic snapshot.

P1 — HIGH IMPACT (15 closed)
=============================

- Auditor #1 P1: searchLikeCjk used `new Date()` instead of
  parseUtcTimestamp → CJK fallback timestamps offset by host's local
  TZ. Other 4 search paths used parseUtcTimestamp; CJK was the outlier.
- Auditor #2 P1: Voyage responseBody privacy. W4 fixed only the 400
  path; 401/403/429/5xx/4xx-other still attached raw bodyText to the
  exception. Same Sentry/log-capture vector. Fix: route ALL non-200
  responseBody through summarizeBody for parity.
- Auditor #4/13 P1: tickExtraction ignored result.lockLostMidTick. W4
  added the field but the wrapper returned `lockAcquired: true`
  regardless. Now flips to false when heartbeat reported lock-loss
  mid-tick → autostart can detect + back off.
- Auditor #5 P1.1: best-of-N used Promise.all → one failed candidate
  threw away successful peers' work. Fix: Promise.allSettled. Throw
  only if ALL fail; judge picks among survivors.
- Auditor #5 P1.2: best-of-N with N=1 still ran judge — judge prompt
  expects 0..N-1 indexed candidates; many models emit 1-indexed and
  trip judge_failure. Fix: skip judge when only 1 candidate survived.
- Auditor #6 P1: parsePeriodShortcut regex over-accepted undocumented
  variants (last-3day, last-3-d). Fix: tightened to /^last-(\d+)d$|
  ^last-(\d+)-days$/ matching only documented forms.
- Auditor #8 P1-3: sort silent override. Agent passing sort=relevance
  with mode=regex got recency without warning. Fix: details now
  surfaces sortIgnored: true + requestedSort/effectiveSort.
- Auditor #8 P1-2: kFts/kSemantic over-fetch was max(limit, 50). At
  limit=200, rerank had ZERO headroom. Fix: 3× limit, floored at 50,
  capped at 500 (Voyage rerank budget).
- Auditor #21 + #8 P1-6: hybrid confidenceBand thresholds reuse
  cosine calibration on rerank scores (different scale). Fix: emit
  confidenceBandSource: "cosine" | "rerank" so callers know which
  signal drove the band.
- Auditor #12 P1-A: extractor placeholder pre-scan (W4 promised but
  never implemented). Fix: refuse extraction if leaf content contains
  XML envelope-like patterns (defense-in-depth against injection).
- Auditor #12 P1-E: dead-letter UPDATE failure left attempts at 0 →
  infinite retry. Fix: try second simpler bump-only UPDATE if the
  first (with last_error) fails.
- Auditor #18 P1: promptAwareEviction violates "structural-only"
  invariant. Fix: documented as opt-in with WARNING comment in
  config.ts that flagging it on breaks deterministic replay.
- Auditor #20 P1-3: README synthesize_around description was
  anchor-required-only — period mode (the lcm_recent replacement)
  not mentioned. Fix: 3-mode breakdown.
- Auditor #20 P1-4: THE_FIVE_QUESTIONS stale prose declared
  "themes/procedures/entities" all live. Themes + procedures were
  CUT (preserved in Martian-Engineering#616). Fix: explicit coverage status note.

VERIFICATION
============
- 1345/1345 unit tests passing (no regressions)
- QA runner full: 30/30 pass
- QA runner adversarial: 10/10 pass (not re-run; W6 baseline)
- Total cost ~$0.11 per full QA run

DEFERRED (acknowledged)
========================
- A14 P1: lcm_purge_audit table — needs schema migration; defer to
  cycle-3. Workaround: purge_session_id is returned + suppress_reason
  is recorded per leaf row.
- A18 P1: summarizeWithEscalation silent over-cap truncation —
  separate from W4 fallback marker fix; cycle-3 ergonomics.
- A8 P1-5: details.hits[] shape drift across 5 grep modes — by-design
  difference (regex/full_text are aggregates; hybrid/semantic/verbatim
  are per-row). Documented in agent-tools.md.
- A8 P1-4: verbatim recency-only ordering — by-design (citation use
  case prioritizes "what was said most recently").
- A10 P1-01: lcm_expand 24-day legacy timeout — sub-agent-only path,
  bounded by grant TTL.
- A10 P1-06: runExpand `?? 0` fallthrough — multi-conv grant path
  not exercised by lcm_expand_query (always single-conv).
- Various P2/P3 cosmetic items.
100yenadmin pushed a commit that referenced this pull request May 6, 2026
…0 + 9 P1 closed + 4 new regression tests

Wave-9 was the first audit cycle to give every agent FULL FILE context
(not just diffs) plus cross-cutting checklists tailored to their slice,
plus all prior wave findings as known-closed reference. Eva's directive:
"agents need ENOUGH CONTEXT not to introduce new issues while fixing
minor ones." Wave-9 also added a TS-strict closure pass (separate
commit 11f10a6) that brought PR-introduced TS errors from 30 → 0.

11 agents (slicing by responsibility, ~14.7k LOC src + 12.5k LOC tests
+ 2.2k LOC scripts):
  #1 Lossless core      — engine, assembler, retrieval, summarize, compaction
  #2 Migration + schema — db/migration, all migration tests
  #3 Storage layer      — summary-store, conversation-store
  #4 Search tools       — lcm_grep, lcm_semantic_recall, hybrid, semantic
  #5 Drilldown tools    — lcm_describe, lcm_expand, lcm_expand_query
  #6 Entity + extraction — lcm_get_entity, lcm_search_entities, coreference
  #7 Synthesis          — synthesize_around, dispatch, prompt-registry, seed
  #8 Voyage stack       — voyage/client, embeddings/store/backfill/semantic
  #9 Worker + concurrency — concurrency/*, autostarts, worker-orchestrator
  #10 Operator surface  — purge, health, reconcile, eval-runner, plugin
  #11 Scripts/QA-runner — coverage-gap audit Eva caught after launch

Findings: 1 P0 + 13 P1 + 22 P2 + 42 P3 = ~77 unique
(Agent #2 P2 and Agent #7 P1 converged on same `{{date_range}}` bug.)

This commit closes the P0 + 9 of 13 P1s + adds 4 regression tests.
Remaining P1s + all P2/P3 are documented in PR comment for follow-up.

P0 (CLOSED) — Owner gate parity (Agent #10):
- /lcm reconcile-session-keys --apply lacked senderIsOwner (Wave-7
  P0-1 had only added it to /lcm purge). Cross-session data theft
  vector: non-owner agent could re-key Eva's primary thread into an
  attacker bucket via --allow-main-session.
- /lcm worker tick embedding-backfill same gap (lower-impact:
  DoS-by-billing on the operator's Voyage account).
- Both fixed: same gate pattern as case "purge" applied to both.
- 3 new regression tests pin the gate behavior so future refactors
  can't silently regress.

P1 fixes (9 of 13):

P1.1 (Agent #5) — Citation-fabrication count threaded through
ExpandQueryReply. Wave-4+W6+W8 chain validated citedIds internally
(rejected fabricated IDs against summaries table) but
buildExpandQueryReply silently dropped the counts. Agent now sees
citedIdsRejectedAsFabricated + citedIdsExceededValidationCap in the
JSON reply (omitted when zero, summed across buckets in multi-conv
path).

P1.2 (Agent #5) — lcm_describe expandChildren/expandMessages now
consumes the grant token budget. Previously the budget was CHECKED
(budgetExhausted detection) but never DECREMENTED. With 50 children
+ 50 messages × ~2K tokens each = ~100K tokens delivered per call
without grant cap touching. Now sums consumed tokens and calls
authManager.consumeTokenBudget() for sub-agent sessions. Closes the
unbudgeted side-channel that defeated the W4/W6 expansion budget.

P1.3 (Agent #4) — lcm_grep --mode semantic VoyageError contract
parity. Previously caught only `auth` and SemanticSearchUnavailable;
let rate_limit/server_error/network/bad_request/unexpected propagate
as unhandled tool errors. lcm_semantic_recall correctly catches all
VoyageError kinds. Now mirrored — both surfaces routed for Question B
have identical error contract.

P1.4 (Agent #4) — lcm_grep --mode verbatim CJK fallback. messages_fts
uses tokenize='porter unicode61' which can't segment CJK ideographs
— MATCH on 中文 returned 0 rows WITHOUT throwing, so the
exception-driven LIKE fallback never fired. Now containsCjk(pattern)
detected at JS layer, routes directly to LIKE substring match
(skipping FTS join entirely). 1 new regression test covers Chinese
characters.

P1.5 (Agent #10) — reconcileSessionKeys TOCTOU race. affectedConvs
snapshot was taken OUTSIDE BEGIN IMMEDIATE; concurrent INSERT/UPDATE
between snapshot and tx-acquire could be UPDATE-moved without an
audit row, silently dropping it → loss-of-undo on a destructive op.
Same pattern as Wave-8 P1's runSoftPurgeAtomic fix. Refactored:
active-conflict pre-check + affectedConvs SELECT + UPDATEs all run
inside the same BEGIN IMMEDIATE.

P1.6 (Agent #10) — runRecallEval setTimeout leak. Promise.race
spawned a timer that was never cleared on adapter resolve. N=100
queries × 30s = 30s tail-latency floor + event-loop liveness held
open (process never exits in scripts). Added try/finally with
clearTimeout.

P1.8 (Agent #1) — Compaction fallback marker regression. Wave-4 P0
fix in summarize.ts tagged fallback content with "[LCM fallback
summary - model unavailable]" — but because the marker adds ~25
tokens, the resulting summary is LARGER than the source, so
summarizeWithEscalation rejected it as "didn't compress" and fell
through to compaction.ts's OWN buildDeterministicFallback which
emitted raw truncated content with NO marker, silently undoing the
W4 fix for any source <= max(targetTokens*4, 256) chars (i.e. most
leaves under LLM outage). Fix: prepend the same marker in
compaction.ts's fallback. Empty-source path tagged for parity.

P1.9 (Agent #2 + #7 convergence) — {{date_range}} placeholder
orphaned in seed prompts vs renderer. dispatch.renderPrompt only
substituted source_text/tier/memory_type. Seeded daily/weekly/
monthly templates used {{date_range}} literally; SynthesizeRequest
had no dateRange field. Currently latent (synthesize_around clamps
to custom/filtered) but becomes P0 the moment a daily/weekly/monthly
synthesis worker wires up. Same class as Final.review.3 Loop 4 Bug
4.2. Fix: dropped {{date_range}} from seeded templates (use
"from a single day/week/month" phrasing instead). Caller can bake
explicit ranges into sourceText if needed.

P1.10-P1.13 (Agent #11) — QA harness coverage gaps:

P1.10 — process.chdir("/tmp/lossless-claw-upstream") hardcoded made
the QA harness unrunnable anywhere except that exact path. Replaced
with a sentinel-file existence check that errors fast with a clear
"run from repo root" message.

P1.11 — adv-lcm-expand-query-smoke was vacuous: predicate returned
null unconditionally, args omitted required `prompt` field. Now
exercises full dispatch path with real prompt + asserts response
shape (answer + citedIds, or graceful LLM-unavailable error).

P1.12 — Period mode (lcm_recent replacement, most reviewer-debated
capability) had ZERO harness coverage. Added 2 new test cases:
period='yesterday' and period='last-7d' (covers the W7-tightened
hyphenated parser).

P1.13 — lcm_grep regex/full_text modes had ZERO harness coverage
(2 of 5 documented modes). Added 2 new test cases asserting the
regex/full_text response shape (totalMatches/messageCount/
summaryCount, not details.hits which is hybrid-only).

Verifications:
- npx tsc --noEmit → 739 errors (exactly matches origin/main baseline;
  ZERO PR-introduced TS errors)
- npx vitest run → 1353/1353 passing (1349 baseline + 3 owner-gate
  + 1 CJK regression tests)
- All Wave-9 fixes verified at code level on real file paths

Deferred P1s (4 of 13) — handled in follow-up commits / cycle-3:

- P1.7: TOCTOU between affectedConvs and active-conflict pre-check
  is now closed (folded into P1.5 fix above).
- Agent #5 P2 multi-bucket DEFAULT_MAX_CONVERSATION_BUCKETS=3 silent
  drop is documented but deferred (ergonomic, not safety).
- Agent #4 cosineSimilarity not clamped in hybrid mode: trivial 2-line
  fix but not safety.
- Agent #5 dead `runDelegatedExpansionLoop` in lcm_expand: cleanup
  task, no behavior change.

Pattern observation: Wave-9's full-file-context approach paid off —
caught the same class of bug (missing owner gate) on the SISTER case
of a previously-fixed P0, which a narrow-diff audit could not have
spotted. Future audits should keep this approach.
100yenadmin pushed a commit that referenced this pull request May 6, 2026
… 4 sub-agent test layers + 8 source bugs closed

A separate reviewer raised 12 findings on PR Martian-Engineering#613 with the strategic bar
"don't just make the findings disappear; make the PR truthful under real
operator scenarios." User correctly noted "wasn't sure if verified" so
I verified each before fixing. Verification result: 12-for-12 real bugs.

Combined with 4 parallel test-quality sub-agents addressing antipatterns
A8 (concurrency) + A9 (schema drift) + A1/A4 (adversarial scenarios +
fixture-test circularity) + A4-at-scale (stress fixture).

# Reviewer findings (all 12 closed)

## P1 (5)

- **#1 Period synthesis timezone** (src/tools/lcm-synthesize-around-tool.ts):
  parsePeriodShortcut anchored "today/yesterday/this-week/last-week/
  this-month/last-month" at UTC midnight. A Bangkok operator (UTC+7) at
  02:00 local asking "yesterday" got UTC-yesterday — ~17 hours off.
  Operator-trust violation. Now uses Intl.DateTimeFormat to compute
  local-day boundaries in lcm.timezone (configured IANA TZ); samples
  the offset at local noon to avoid DST-fold ambiguity. Relative forms
  (last-Nh, last-Nd) stay UTC-anchored (now-minus-N, not day-anchored).

- **#2 Synthesis cache key** (src/db/migration.ts +
  src/tools/lcm-synthesize-around-tool.ts): UNIQUE index keyed only on
  (session_key, range_start, range_end, leaf_fingerprint, grep_filter).
  Two correctness bugs: (a) tier='custom' then tier='filtered' for same
  range/leaves silently returned wrong-tier cached text, (b) registerPrompt
  changing the active prompt left cache serving stale text from the old
  prompt. Now includes tier_label + prompt_id in both the UNIQUE index
  and the lookup SELECT. Cache is rebuildable so wiping under the new key
  is safe.

- **#4 /lcm eval owner gate** (src/plugin/lcm-command.ts): /lcm eval
  mutates lcm_eval_run + lcm_eval_query_result tables AND can use Voyage
  in hybrid mode (small but non-zero quota cost). Wave-9 Agent #10 had
  classified it as READ_ONLY — the reviewer correctly challenged that
  classification. Now gated on senderIsOwner and added to the
  authorization-invariant test's DESTRUCTIVE_OPERATOR_CASES list.

- **#5 Voyage rerank token budget** (src/embeddings/hybrid-search.ts):
  rerank sent ALL candidates' full content with no enforcement of the
  ~600K-token cap. Realistic queries with many large condensed summaries
  hit Voyage 400 → silent RRF degradation, losing the +52.5pp paraphrastic
  recall lift. Now packs candidates into rerank input cumulatively until
  85% of MAX_TOKENS_PER_RERANK_CALL, dropping tail when over budget.
  Surfaces rerankPackTruncated + rerankPackedCount in HybridSearchResult.

- **#6 lcm_describe base content not charged**
  (src/tools/lcm-describe-tool.ts): Wave-9 P1.2 fix added
  consumeTokenBudget for expandedChildren + expandedMessages but skipped
  the base summary's s.content (which lines.push()es ALL of it). A
  sub-agent could lcm_describe a 30K-token condensed summary with NO
  expansion flags and drain context for free. Now charges base s.tokenCount
  too.

## P2 (5)

- **#3 Suppressed entity leakage** (src/tools/lcm-get-entity-tool.ts +
  src/tools/lcm-search-entities-tool.ts): when ALL mentions of an entity
  were suppressed via /lcm purge, the entity row in lcm_entities still
  leaked canonical_text + alternate_surfaces + metadata via both tools.
  The reviewer's framing: "suppression means invisible to agents, period."
  Both tools now require at least one unsuppressed mention via EXISTS
  guard. The "not found" branch now covers both "no such entity" AND
  "all mentions suppressed" indistinguishably (so an attacker can't
  infer entity existence). Updated test fixtures' insertEntity helpers
  to auto-create a default visible mention; tests that explicitly want
  the all-suppressed case opt out via noDefaultMention: true.

- **#7 Pending-extractions count** (src/extraction/entity-coreference.ts):
  countPendingExtractions filtered only on (kind, completed_at IS NULL),
  but runCoreferenceTick's selector ALSO requires (attempts < 5,
  summaries.suppressed_at IS NULL). Mismatch caused autostart to spin
  forever on rows the tick would never select. Predicate now exactly
  matches the selector.

- **#8 QA runner period coverage + exit semantics** (scripts/v41-qa-runner.mjs):
  period test cases I added in Wave-9 P1.12 omitted window_kind="period"
  (required by the tool), so they only hit schema-validation early-return
  and the regex match on 'period' made them trivially pass. Added the
  required field. Plus failedImportant had no exit branch — runner exited
  0 on any "important" failure, advisory-only. Added exit code 1 for
  important failures so the runner can act as a release gate.

- **#9 sqlite-vec install honesty** (package.json + semantic-infra-init.ts):
  sqlite-vec wasn't in any dependencies block, init log was log.info
  (low visibility), and PR_DESCRIPTION emphasized VOYAGE_API_KEY alone.
  Added to optionalDependencies; bumped log to log.warn with explicit
  install instructions + clear "what becomes unavailable" message.

- **#10 Backfill complete message lies** (src/plugin/lcm-command.ts):
  countBackfillPending excludes leaves with token_count >
  MAX_TOKENS_PER_EMBED_DOC, so an over-cap leaf was neither pending nor
  backfilled. Worker-tick output printed "✅ Backfill complete" even when
  over-cap leaves remained unembedded. Added countOverCapPendingForBackfill
  helper; completion message now distinguishes "in-range complete +
  over-cap remain" from full coverage.

## P3 (2)

- **#11 lcm_synthesize_around description** (src/tools/lcm-synthesize-around-tool.ts):
  agent-tool description still said "Two modes" (time + semantic) while
  schema declared three. Rewrote description + JSDoc to mention all three
  (period, time, semantic) and explicitly call out 'period' as the
  lcm_recent replacement / "what did we work on yesterday" surface.

- **#12 NUL byte in source** (src/tools/lcm-synthesize-around-tool.ts:331):
  fingerprintLeaves used a literal NUL byte (\x00) as a hashing separator,
  making the file binary to grep. Replaced with the escape sequence "\0"
  (functionally identical at runtime, readable in source). File is now
  searchable.

# Sub-agent test layers (4 in parallel)

## Sub-agent #1 — Concurrency / TOCTOU (test/v41-concurrency-invariants.test.ts, ~1044 LOC, 8 tests)

Worker-thread-based parallel-writer harness reproduces and pins
race-condition fixes: reconcileSessionKeys race (Wave-9 P1.5),
runSoftPurgeAtomic race (Wave-8 P1), worker-lock acquire (5-way),
heartbeat-during-LLM-call (Wave-9 Agent #8 P2), recordEmbedding
DELETE-before-INSERT atomicity. Verified regression-detection by
simulating pre-fix code. 0 new bugs found.

## Sub-agent #2 — Schema/placeholder drift (test/v41-schema-drift-invariants.test.ts, ~654 LOC, 19 tests)

Static-analysis tests via readFileSync + regex. Catches: placeholder
drift in seeded prompts vs renderer (Wave-9 P1.9 class), tier_label
CHECK constraint coverage vs TS union (Final.review.3 Bug 4.4 class),
manifest-vs-registered-tool drift (Wave-9 vapor-tools class),
parser/handler symmetry, FK ON-DELETE explicitness. **Found 3 P3 FK
drift bugs** — 3 declarations missing explicit ON DELETE clauses.
Closed in this commit (lcm_synthesis_cache.prompt_id,
lcm_synthesis_audit.prompt_id, lcm_embedding_meta.embedding_model →
all now `ON DELETE RESTRICT`).

## Sub-agent #3 — Adversarial scenarios + fixture-test circularity audit (test/v41-adversarial-scenarios.test.ts, ~1149 LOC, 37 tests)

Audit of original 25 scenarios: 16/26 strong, 9/26 weak ("only
totalMatches > 0"), 1 sentinel. Strengthened 6 weak tests in
v41-five-questions.test.ts (B1-B5, E2) to assert specific summary
IDs. **Found 1 real fixture bug**: summaries_fts insert used `rowid`
but schema declares `(summary_id UNINDEXED, content)` — original
B1-B5 tests "passed" only because they matched at the messages
layer, never actually exercising summary FTS. Fixed in fixture; the
strengthened B1-B5 tests now actually exercise summary FTS. 37 hard
adversarial scenarios spanning paraphrase, ambiguity/ranking, compound
queries, negative queries, content injection (placeholder/XML/script/
SQL-injection), ranking sensitivity, cross-tool composition,
suppression boundary.

## Sub-agent #4 — Stress fixture (test/fixtures/v41-stress-corpus.ts + test/v41-stress-fixture.test.ts, ~898 LOC, 11 tests)

Deterministic generator for 1500-2500 leaves with realistic distribution
(30% last-7-days, dense days with 100+ leaves, 5-10% suppressed, 5% CJK,
near-duplicates, 5 adversarial-content leaves). 11 stress tests cover
build smoke, determinism, distribution, dense-day query, suppression
cascade, FTS5 perf, vec0 KNN (graceful no-op when vec0 unavailable),
adversarial-content non-breaking, near-duplicate handling, recency floor.

# Wave-10 reviewer regression coverage (test/v41-wave10-reviewer-regressions.test.ts, 6 tests)

Pins fixes for #2 (cache UNIQUE index w/ tier+prompt), #3 (suppressed
entity invisibility), #7 (pending count predicate), #10 (over-cap
counting). #1 has its own dedicated v41-period-timezone.test.ts (8
tests). #4 covered by extending v41-authorization-invariants.test.ts
DESTRUCTIVE_OPERATOR_CASES.

# Verification

- **1490/1490 tests passing** (1401 pre-Wave-10 + 89 new from this commit)
- **677 TS errors** (FEWER than the 739 main baseline — type-tightening
  fixes cascaded from the source changes)
- 4 sub-agent test files all green
- 6 reviewer-regression tests all green
- Authorization invariant test now covers `eval` → catches future
  removal of the gate

# What's NOT in this commit (future work)

- Mutation testing CI integration (stryker is too slow for per-PR;
  config exists for ad-hoc invocation)
- Wave-1-9 antipattern tabulation update with Wave-10 findings
100yenadmin pushed a commit that referenced this pull request May 6, 2026
…ed 12/12 real)

Fresh re-audit at 37e2b71 found 12 issues; 11 closed in this commit, 1
documented as known limitation. Reviewer was 12-for-12 real (Wave-10
was also 12-for-12; reviewer track record: 24-for-24).

# CI blockers

- **#1 (P1)** Auth invariant test hardcoded `/tmp/lossless-claw-upstream`
  path. CI failed because that path doesn't exist on GitHub runners;
  local runs accidentally succeeded by reading whatever stale checkout
  was at that path. Now resolves via `import.meta.url` →
  `__dirname/../src/plugin/lcm-command.ts`. Works in any worktree.

- **#10 (P2)** `pnpm-lock.yaml` was stale after the Wave-10
  `optionalDependencies` addition. Regenerated via `pnpm install
  --lockfile-only`; verified `pnpm install --frozen-lockfile` succeeds.

# Security parity

- **#2 (P1)** `/lcm doctor apply` and `/lcm doctor clean apply` lacked
  `senderIsOwner` gate. Wave-9 Agent #10 had classified the doctor
  cases as READ_ONLY, but the `apply` flag inside dispatches to the
  summarizer (cost) AND mutates summaries (state) for `doctor apply`,
  and DELETEs cleaner matches for `doctor clean apply`. Mirror the
  purge / reconcile / worker-tick / eval gate pattern. Read-only
  variants (no `--apply`) stay open.

  Plus updated `test/lcm-command.test.ts`'s `createCommandContext`
  helper to default `senderIsOwner: true` so existing tests for the
  doctor mutating paths continue passing — Wave-9 negative tests
  still explicitly pass `senderIsOwner: false` via overrides.

  Plus added 4 new tests to `v41-authorization-invariants.test.ts`
  pinning the Wave-11 doctor-apply gate behavior (apply-rejected,
  read-only-allowed for both `doctor` and `doctor clean`).

- **#5 (P1)** `lcm_describe` early-budget-gate. The Wave-10 fix charged
  base summary tokens against the grant AFTER emitting `s.content`.
  For a sub-agent at zero remaining budget, the content was already
  disclosed before accounting could prevent it. Added an EARLY gate:
  if delegated session AND base summary tokens > remaining grant,
  redact `s.content` with a clear "[REDACTED — base summary content
  is N tokens but grant has only M remaining]" message and skip the
  charge. Closes the disclosure-before-accounting path.

# Correctness

- **#3 (P1)** Timezone fractional offsets + DST. Wave-10's "sample
  offset at noon" approach broke on:
    - Half-hour zones: Asia/Kolkata (UTC+5:30) → showed +5 not +5:30
    - Quarter-hour zones: Asia/Kathmandu (UTC+5:45)
    - DST transition days: LA spring-forward 2026-03-08 → noon is in
      PDT (-7) but local midnight was in PST (-8); my function used
      the noon offset for the whole day → wrong by 1 hour
  Replaced with iterative converge-to-midnight algorithm:
    1. Format `at` in target tz to get y/m/d
    2. Probe = naive `Date.UTC(y, m-1, d, 0, 0, 0)`
    3. Format probe in target tz; compute delta from target midnight
    4. Adjust probe; repeat until delta=0 (typically 1-2 iters)
  Handles all IANA timezones, DST transitions, and arbitrary offsets.

  Added 3 new regression tests:
    - Asia/Kolkata 'yesterday' (UTC+5:30) — half-hour offset
    - Asia/Kathmandu 'today' (UTC+5:45) — quarter-hour offset
    - America/Los_Angeles 2026-03-08 — spring-forward day, asserting
      'today' duration is exactly 23h

- **#6 (P1)** Hybrid rerank now skips individually oversized
  candidates instead of bailing. Pre-fix: when the FIRST candidate
  exceeded the 510K-token (85% of 600K) rerank budget, the packer
  set `rerankPacked=[]` and broke out, disabling rerank for the
  whole result set. Now: oversized candidates are individually
  skipped (counted in `rerankPackSkippedOversized`) and packing
  continues with later candidates that fit. Result: a single huge
  FTS hit no longer takes down the whole rerank.

- **#7 (P1)** Voyage `output_dimension` not forwarded. Configurable
  embedding dimensions (`LCM_EMBEDDING_DIM=2048` registers a 2048-dim
  profile in `lcm_embedding_profile`) but `embedTexts()` never sent
  `output_dimension` to Voyage, so Voyage returned its default (1024).
  vec0 INSERT then failed with dim mismatch on the per-model table.
  Added `outputDimension?: number` to `VoyageEmbedOptions`; forwarded
  via backfill (`opts.voyageOutputDimension`) and semantic-search
  query embed (`active.dim`). Default unchanged (omit → Voyage 1024).

# Documentation accuracy

- **#4 (P1)** Synthesis dispatch model claim. Tool description said
  "per-tier dispatch (haiku/sonnet/opus/thinking)" but actual LLM call
  routes through the configured summarizer chain (which ignores
  `args.model`). Source code already had honest comment in
  `buildLlmCallFromSummarizer` ("the summarizer wrapper ignores the
  dispatch-supplied model"); the tool description and PR description
  overclaimed. Updated tool description to be accurate: dispatch
  records the per-tier model name in the audit table, but the
  actual LLM call uses the operator's configured summarizer chain.

# Polish

- **#9 (P2)** Health archive filter. `readActiveProfile` selected on
  `active = 1` alone, ignoring `archive_after IS NOT NULL`. Semantic
  retrieval correctly filters archived; health was reporting a
  profile semantic search would not actually use during model cutover.
  Now matches: `WHERE active = 1 AND archive_after IS NULL`.

- **#11 (P2)** Changeset rewritten. Old changeset only mentioned
  session-family recall. New changeset documents the full v4.1
  release surface: 8 agent tools (with new modes), 2 worker autostarts,
  9 operator commands (with owner-gating), schema changes, sqlite-vec
  optionalDependency, configuration env vars, and what was cut to Martian-Engineering#616.

- **#12 (P3)** Stale entity-search docblock. The header comment said
  "entities with all-suppressed mentions can still appear here";
  Wave-10 added the EXISTS guard so they no longer can. Updated
  comment to reflect the actual filter behavior.

# Known limitation (deferred)

- **#8 (P2)** Cache key still ignores resolved model. Adding `model_used`
  to the UNIQUE index doesn't help because model resolution is dynamic
  (the summarizer chain picks at call time, not before INSERT). The
  proper fix is invalidate-on-mismatch at cache-hit time, which is a
  larger refactor. Documented in the entry above + tracked for follow-up.

# Verification

- `npx vitest run`: **1513 / 1513 tests passing** (1502 → 1513;
  +11 new regression tests for Wave-11 fixes)
- `npx tsc --noEmit`: **677 errors** (still below 739 main baseline;
  no PR-introduced TS errors)
- `pnpm install --frozen-lockfile --ignore-scripts --lockfile-only`:
  **succeeds** (was failing pre-fix with ERR_PNPM_OUTDATED_LOCKFILE)
- Authorization invariant test: now resolves the source path relative
  to test file via `__dirname` — works in any checkout location
100yenadmin pushed a commit that referenced this pull request May 7, 2026
…Summarizer

Methodology: Research → Debate → Decide → Implement.

Step 1 archeology found two LlmCall wrappers:
  - createWorkerLlmCall (worker-llm.ts:52-126) honors args.model + returns actualModel
  - buildLlmCallFromSummarizer (this file) ignored args.model + returned no actualModel

Wave-11 commit e96e03e finding #4 ("Documentation accuracy" heading)
fixed the tool description's overclaim — but did NOT adjudicate the
audit-row gap. lcm_synthesis_audit.model_used recorded the dispatched
intent (pickModel's recommendation), not the actually-resolved model.
Operators debugging a synthesis failure would see the wrong model in
audit logs.

Step 3 adversarial review verified: the original "close as won't-fix"
recommendation overclaimed Wave-11 precedent. The decision record had
already filed a P3 follow-up to do this exact 10 LOC fix — calling it
won't-fix while filing P3 was contradictory. Just do the fix.

Step 4 decision: thread a `resolveActualModel: () => string | undefined`
parameter into the wrapper. Pass `() => summarizerBuilt.model` from the
call site. This eliminates the audit/execution gap. The wrapper now
returns `actualModel` from the summarizer's resolved primary candidate
(src/summarize.ts:1688-1695).

Caveat documented in code comment: if mid-call fallback fires, the
recorded model may not match the candidate that actually succeeded.
Strictly better than recording dispatched intent. Future improvement:
have the summarizer surface the candidate that actually ran.

Tool description also updated to say "audit table records the resolved
model that actually ran" (was: "records the per-tier model name in the
audit table") — the contract is now honest end-to-end.

LOC: 10 (parameter + return field + call site + description text).

Documents: /tmp/adversarial-f8.md, /tmp/decision-phase2-final.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants