Skip to content

feat: add weekly and monthly lcm rollups#15

Closed
100yenadmin wants to merge 23 commits into
mainfrom
feat/lcm-week-month-rollups
Closed

feat: add weekly and monthly lcm rollups#15
100yenadmin wants to merge 23 commits into
mainfrom
feat/lcm-week-month-rollups

Conversation

@100yenadmin

@100yenadmin 100yenadmin commented Apr 27, 2026

Copy link
Copy Markdown
Member

Summary

Builds on #13/#14 by adding calendar weekly/monthly aggregate rollups and the operator-only lcm_rollup_debug tool. This completes the first temporal-memory stack: daily recap, sub-day windows, rolling 7d/30d recap, calendar week/month aggregates, provenance, and debug visibility.

lcm_recent remains a recap/window-entry tool, not proof. Exact commands, paths, timestamps, root cause, and shipped/decided claims still require lcm_describe or lcm_expand_query.

What Changed

  • Added weekly/monthly aggregate rollup builders from daily rollups.
  • Added lcm_rollup_debug behind explicit rollupDebugEnabled/LCM_ROLLUP_DEBUG_ENABLED operator gating, with strict period/status handling, clamped limits, and source IDs hidden unless includeSources=true.
  • Preserved provenance semantics: source_message_count counts source messages, aggregate counts sum source-message counts, and source-token counts remain separate.
  • Hardened fingerprints with content and temporal fields so same-ID/same-token content edits invalidate rollups.
  • Moved rollup maintenance outside transcript-GC-only flow, preserved retry state on daily/aggregate failures, and made final daily sweep-state failures retry-safe.
  • Avoided serving stored week/month rollups while rebuild is pending.
  • Filtered aggregate inputs and stored-rollup reads by timezone; rollup uniqueness includes timezone.
  • Deleted stale day/aggregate rollups when source inputs disappear.
  • Kept 7d/30d recap useful by combining complete prior daily rollups with live current-day fallback.
  • Inherited feat: add daily lcm_recent rollups #13/feat: add sub-day lcm_recent windows #14 hardening for strict dates, DST gaps, night/24:00, UTC+13 keys, canonical UTC timestamp outputs, expression indexes, and state-preserving upserts.

Out Of Scope

  • No fuzzy AI semantic memory layer.
  • No “first time X happened” semantics.
  • No unverified “what shipped” proof; use source expansion for proof.
  • Event-bounded questions still require anchoring the event time/window first.

Validation

  • npm test -- --run test/rollup-store-builder.test.ts test/lcm-tools.test.ts — 33 tests passed.
  • npm run build — passed.
  • git diff --check — clean.

Reviewer Notes

Known blockers addressed: source-message provenance counts, aggregate retry state, rebuild maintenance placement, pending aggregate freshness, content-change fingerprints, debug source hiding, strict status/period validation, unused retrieval removal, timezone-safe aggregate inputs, timezone-qualified uniqueness/lookups, empty-day stale rollup deletion, UTC+13 week/month aggregation, 7d/30d complete-rollup plus live-current-day behavior, complete aggregate coverage, lcm_recent includeSources/budget enforcement, direct empty-day deletion, and orphaned aggregate sweeps, quiet-day aggregate completeness, DST-skipped-midnight day bounds, sub-1000 output budgets, and corrected confidence/token reporting, canonical week-key validation, unbounded aggregate key discovery, and idempotent aggregate fingerprints, sparse inactive-day 7d coverage, and skipped-midnight 24:00 window endpoints, and monotonic daily sweep last_rollup_check_at updates, DST-safe week/month aggregate bounds, shared Unicode-aware token estimates, SQL fallback cap accounting, and truncation propagation, aggregate phase timestamp preservation, stale-row rebuild restoration, final daily sweep-state failure reporting, non-fake leaf updated_at semantics, exact fallback total counts past the SQL cap, debug limit clamping, and operator-gated debug registration, and 30-day maintenance backfill for 30d recall.

Summary by CodeRabbit

  • New Features
    • Temporal rollups (daily, weekly, monthly) with timezone-aware periods, deterministic rebuilds, token-aware content truncation, and automatic maintenance triggered by new messages
  • Tools
    • Recent-recall tool for flexible time-based queries with budgeted truncation and live-fallback
    • Rollup-debug tool to inspect rollup state and optionally reveal source IDs
  • Database
    • Migrations add rollup tables, state columns, indexes, period views, and safe column-add checks
  • Tests
    • End-to-end tests for rollup building, period parsing, DST edge cases, retrieval fallbacks, and tools

Copilot AI review requested due to automatic review settings April 27, 2026 22:02
@coderabbitai

coderabbitai Bot commented Apr 27, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

Adds temporal rollups: DB migrations and safe ALTER helper; RollupStore persistence and state; RollupBuilder for daily/weekly/monthly aggregates with timezone-aware bounds, deterministic fingerprinting, and token-budgeted rendering; engine wiring to mark and rebuild rollups on ingest/maintenance; two agent tools (lcm_recent, lcm_rollup_debug); plugin registration and tests.

Changes

Cohort / File(s) Summary
Migrations & schema
src/db/migration.ts
Formatting changes and callback reorder; new addColumnIfMissing(...); creates lcm_rollups, lcm_rollup_sources, lcm_rollup_state; adds last_weekly_build_at/last_monthly_build_at; new indexes and daily_rollups/weekly_rollups/monthly_rollups views.
Engine wiring
src/engine.ts
Constructs RollupStore/RollupBuilder, adds getRollupStore()/getRollupBuilder() getters; upserts per-conversation rollup state on message ingest (pending_rebuild, timezone, last_message_at); conditionally triggers rebuilds in maintain() with error-tolerant logging.
Rollup construction
src/rollup-builder.ts
New RollupBuilder and utilities: timezone-safe local-day bounds, deterministic fingerprinting, day/weekly/monthly build flows, hard token-cap rendering/truncation, progressive omission strategies, and transactional upserts.
Persistence layer
src/store/rollup-store.ts
New RollupStore and types; upsertRollup, read APIs (getRollup/getRollupById/listRollups/listRollupsInRange), replaceRollupSources, getRollupSources, state getState/upsertState, markStale/deleteRollup, and getLeafSummariesForDay.
Agent tools
src/tools/lcm-recent-tool.ts, src/tools/lcm-rollup-debug-tool.ts
Adds lcm_recent (period parsing, timezone-aware bounds, rollup usage with live/fallback behavior, token budgeting, test internals) and lcm_rollup_debug (inspect rollup state/rows, optional source provenance).
Plugin integration
src/plugin/index.ts
Registers lcm_recent and lcm_rollup_debug; inserts lcm_recent earlier in lossless recall routing; other edits are wiring/formatting.
Tests
test/rollup-store-builder.test.ts
New Vitest suite covering migrations, schema, day/aggregate builds (idempotency, DST/local-midnight handling), fallback semantics, cleanup, and debug tool behaviors.
Release metadata
.changeset/lcm-temporal-rollups.md
Adds changeset noting temporal rollups and two new tools.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant User
  participant AgentTool as Agent Tool
  participant Engine as LcmContextEngine
  participant Store as RollupStore/DB
  participant Builder as RollupBuilder

  User->>AgentTool: request recent/inspect
  AgentTool->>Engine: resolve engine & session
  Engine->>Store: getState / query rollups or summaries
  Store-->>Engine: rollup rows or leaf summaries
  Engine-->>AgentTool: return formatted markdown + details

  Note over Engine,Store: on message ingest
  Engine->>Store: upsertState(pending_rebuild=1, timezone, last_message_at)
  Store-->>Engine: ack

  Note over Engine,Builder: maintenance run
  Engine->>Store: getState(conversation)
  Store-->>Engine: state (pending_rebuild?)
  alt pending_rebuild true
    Engine->>Builder: buildDailyRollups(conversation)
    Builder->>Store: read leaf summaries / day rollups
    Store-->>Builder: summaries / day rollups
    Builder->>Store: upsertRollup(s) (BEGIN IMMEDIATE txn)
    Store-->>Builder: ack
    Builder-->>Engine: BuildResult(built/skipped/errors)
    Engine->>Store: upsertState(clear pending_rebuild/update timestamps)
  else no rebuild
    Engine->>Builder: skip build
  end
Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 22.76% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The PR title 'feat: add weekly and monthly lcm rollups' accurately reflects the primary change: introduction of weekly and monthly aggregate rollup builders alongside the existing daily rollup infrastructure.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/lcm-week-month-rollups

Comment @coderabbitai help to get the list of available commands and usage tips.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@100yenadmin

Copy link
Copy Markdown
Member Author

Adversarial review fixes pushed in a75513a.

Addressed merge-blocking issues:

  • sub-day windows no longer return full daily rollups when a matching day rollup exists; windowed periods force bounded leaf-summary fallback
  • 4-8pm now infers the missing start meridiem as 16:00-20:00
  • sub-day local-time conversion now uses iterative timezone resolution instead of adding minutes to midnight, so DST transition boundaries are handled correctly
  • fallback summary selection now uses interval overlap and julianday(...) rather than a single effective timestamp / raw TEXT comparisons
  • replaceRollupSources is now awaited so provenance write failures propagate
  • rollup pending state is cleared only after the full daily pass, and remains dirty when errors occur
  • maintenance builds rollups after summary-producing maintenance, and also backfills when rollup state is missing
  • rollup dirty marking during ingest is fault-isolated so rollup-state failures do not break core ingestion
  • rollups use the engine's effective timezone, not raw config timezone
  • lcm_recent description and prompt routing now match the SQL fallback / temporal-recall behavior
  • added changeset for user-facing tools/behavior

Verification:

  • npm test -- --run test/rollup-store-builder.test.ts test/lcm-tools.test.ts
  • npm run build

Note: PR #15 now contains hardening that should also be backported/split into #13/#14 before final merge if we keep the PRs stacked against main.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@100yenadmin

Copy link
Copy Markdown
Member Author

Follow-up hardening pushed in 6210175.

Additional review-driven fixes:

  • lcm_recent and lcm_rollup_debug now prefer the public getRollupStore() path instead of relying only on a private db cast
  • debug timestamps are rendered in the stored rollup-state timezone when present
  • removed the no-op markStale() transitions in aggregate/day rebuild paths
  • weekly aggregate keying now derives week starts using the originating rollup timezone rather than assuming UTC weekday math

Re-verified after the change:

  • npm test -- --run test/rollup-store-builder.test.ts test/lcm-tools.test.ts
  • npm run build

coderabbitai[bot]

This comment was marked as resolved.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@100yenadmin

Copy link
Copy Markdown
Member Author

Follow-up hardening pushed locally and about to land:

  • daily rollups now persist meaningful source_message_count totals from source_message_token_count instead of always writing 0
  • rollup fingerprints now include content hash + temporal/source metadata, so stale day/aggregate rollups rebuild when content changes even if IDs/token counts stay the same
  • lcm_recent now rejects impossible plain dates (for example 2026-02-30) instead of silently accepting them
  • lcm_rollup_debug no longer prints source summary IDs unless includeSources=true, and its period filter is strictly bounded to day/week/month
  • removed the dead since/before parse no-op from lcm_recent

Verification:

  • npm test -- --run test/rollup-store-builder.test.ts test/lcm-tools.test.ts
  • npm run build

One caveat I kept explicit: current summary rows do not store a separate mutable updated_at, so the stronger fingerprint uses content hash + created/coverage/source metadata today. If/when summary-level updated_at lands in schema, it should be threaded into the rollup source query too.

coderabbitai[bot]

This comment was marked as resolved.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

@100yenadmin

Copy link
Copy Markdown
Member Author

Follow-up hardening from PR #14 was carried forward in e20ff8a.\n\nInherited fixes:\n- Sparse inactive-day 7d/day coverage no longer forces full-window fallback when missing day rollups are genuinely empty.\n- 24:00 / night endpoints use the DST-skipped-midnight tolerant boundary helper.\n\nValidation on PR #15:\n- npm test -- --run test/rollup-store-builder.test.ts test/lcm-tools.test.ts — 25 tests passed.\n- npm run build — passed.\n- git diff --check — clean.

@100yenadmin

Copy link
Copy Markdown
Member Author

Carried the PR #13 daily-sweep state fix forward in eb4d8be.\n\nValidation on current #15 head:\n- npm test -- --run test/rollup-store-builder.test.ts test/lcm-tools.test.ts — 26 tests passed.\n- npm run build — passed.\n- git diff --check HEAD~1..HEAD — clean.

coderabbitai[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

@100yenadmin

Copy link
Copy Markdown
Member Author

Follow-up hardening landed in 3831470.\n\nAddressed current-head review items:\n- Week/month aggregate bounds now use the same skipped-midnight tolerant local-day boundary helper as daily rollups.\n- Rollup-builder now uses the shared Unicode-aware token estimator instead of a local length / 4 shadow.\n- Fallback SQL cap is surfaced in totalMatches/truncated instead of silently pretending the capped set is complete.\n- Rollup/live fallback trimming now propagates into response details.truncated.\n- Dead confidence/budget helper code was removed.\n- Inherited #14 explicit 24:00 endpoint parsing and fallback-only current-day status.\n\nValidation on current #15 head:\n- npm test -- --run test/rollup-store-builder.test.ts test/lcm-tools.test.ts — 29 tests passed.\n- npm run build — passed.\n- git diff --check — clean.

@100yenadmin

Copy link
Copy Markdown
Member Author

Follow-up hardening landed in 28af178.\n\nAddressed current-head review items:\n- Weekly/monthly phase timestamps now advance independently only when the corresponding phase succeeds.\n- Stale day/week/month rows with matching fingerprints rebuild back to ready instead of being skipped.\n- Added regression coverage for both behaviors.\n\nValidation on current #15 head:\n- npm test -- --run test/rollup-store-builder.test.ts test/lcm-tools.test.ts — 31 tests passed.\n- npm run build — passed.\n- git diff --check — clean.

coderabbitai[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

This comment was marked as resolved.

@100yenadmin

Copy link
Copy Markdown
Member Author

Added final hardening on this branch:

  • 30-day maintenance backfill so 30d recall has materialized daily coverage more reliably
  • build + targeted tests re-run clean (npm test -- --run test/rollup-store-builder.test.ts test/lcm-tools.test.ts, npm run build)
  • branch remains budget-aware / confidence-aware from the earlier fixes

I also re-verified that source-token provenance stays separate from summary-token counts in aggregate rollups.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 12 new potential issues.

Open in Devin Review

Comment thread src/db/config.ts
Comment on lines +94 to +95
/** When true, register the operator-facing lcm_rollup_debug tool. */
rollupDebugEnabled: boolean;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 AGENTS.md violation: skills/lossless-claw/references/config.md not updated with rollupDebugEnabled

The PR adds the rollupDebugEnabled config option to src/db/config.ts:94-95, openclaw.plugin.json:196-199, docs/configuration.md:115, and README.md:186, but does not update skills/lossless-claw/references/config.md. The AGENTS.md rule explicitly states: "Keep skills/lossless-claw/references/config.md consistent with docs/configuration.md, openclaw.plugin.json, and the runtime defaults in src/db/config.ts. When config keys, aliases, defaults, or precedence rules change, update that bundled skill reference in the same change."

Prompt for agents
The AGENTS.md rule requires updating skills/lossless-claw/references/config.md whenever config keys change. The new rollupDebugEnabled config option was added to src/db/config.ts, openclaw.plugin.json, docs/configuration.md, and README.md but is missing from skills/lossless-claw/references/config.md. Add a section for rollupDebugEnabled to that file, following the same structure as existing entries (e.g. the transcriptGcEnabled or proactiveThresholdCompactionMode sections). Include the key name, type (boolean), default (false), env override (LCM_ROLLUP_DEBUG_ENABLED), and a brief description of its purpose (registers the operator-facing lcm_rollup_debug inspection tool).
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread docs/configuration.md
| `timezone` | `string` | `TZ` or system timezone | `TZ` | IANA timezone used for timestamp rendering in summaries. |
| `pruneHeartbeatOk` | `boolean` | `false` | `LCM_PRUNE_HEARTBEAT_OK` | Retroactively removes `HEARTBEAT_OK` turn cycles from persisted storage. |
| `transcriptGcEnabled` | `boolean` | `false` | `LCM_TRANSCRIPT_GC_ENABLED` | Enables transcript rewrite GC during `maintain()`; disabled by default so transcript rewrites stay opt-in. |
| `rollupDebugEnabled` | `boolean` | `false` | `LCM_ROLLUP_DEBUG_ENABLED` | Registers the operator-facing `lcm_rollup_debug` inspection tool. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 AGENTS.md violation: docs/configuration.md full example block missing rollupDebugEnabled

The reference table at docs/configuration.md:115 correctly adds rollupDebugEnabled, but the "Complete plugins.entries.lossless-claw.config example" block at lines 15–65 of the same file was not updated. The AGENTS.md rule states: "update the reference tables and the full example plugins.entries.lossless-claw.config block at the top of that file in the same change." Other boolean defaults like transcriptGcEnabled: false (line 47) are shown in the example, so rollupDebugEnabled: false should appear there as well.

Prompt for agents
Add rollupDebugEnabled: false to the full JSON example block at the top of docs/configuration.md (lines 15-65). It should be placed after transcriptGcEnabled: false (line 47), consistent with the key ordering in src/db/config.ts and openclaw.plugin.json. The entry should be: "rollupDebugEnabled": false
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/db/config.ts
Comment on lines +425 to +428
rollupDebugEnabled:
env.LCM_ROLLUP_DEBUG_ENABLED !== undefined
? env.LCM_ROLLUP_DEBUG_ENABLED === "true"
: toBool(pc.rollupDebugEnabled) ?? false,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 AGENTS.md violation: no regression test for new rollupDebugEnabled config option

The PR adds rollupDebugEnabled as a new config option with env override LCM_ROLLUP_DEBUG_ENABLED and default false, but does not add or update any test in test/config.test.ts. The AGENTS.md rule states: "Add or update a regression test when changing config options so schema drift is caught before release." The existing config tests (e.g. expect(config.transcriptGcEnabled).toBe(false) at test/config.test.ts:41) follow a clear pattern: the defaults test verifies the default, the env-override test verifies env takes precedence, and the manifest test verifies the schema entry exists. None of these exist for rollupDebugEnabled.

Prompt for agents
Add regression tests for the new rollupDebugEnabled config option in test/config.test.ts. Following the existing patterns:

1. In the defaults test (around line 41), add: expect(config.rollupDebugEnabled).toBe(false)

2. In the env-override test (around line 71), add rollupDebugEnabled: true to the plugin config fixture and LCM_ROLLUP_DEBUG_ENABLED: 'true' to the env, then verify: expect(config.rollupDebugEnabled).toBe(true)

3. In the manifest schema test (around line 551), add: expect(manifest.configSchema.properties.rollupDebugEnabled).toEqual({ type: 'boolean' })
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/rollup-builder.ts
Comment on lines +1101 to +1111

function uniqueStrings(values: string[]): string[] {
return [...new Set(values)];
}

function startOfWeekKey(dayKey: string, timezone: string): string {
void timezone;
assertValidDateKey(dayKey);
const [year, month, day] = dayKey
.split("-")
.map((part) => Number.parseInt(part, 10));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: startOfWeekKey ignores its timezone parameter — correct because day-of-week is timezone-invariant for date strings

The startOfWeekKey function at src/rollup-builder.ts:1101 accepts a timezone parameter but explicitly ignores it with void timezone. This is correct: a date key like 2026-04-27 represents Monday regardless of timezone — the day-of-week calculation only needs the calendar date, not the timezone. The Date.UTC() construction and getUTCDay() call correctly determine the weekday for any YYYY-MM-DD string. The void timezone is presumably intentional to match the function signature pattern used elsewhere while documenting the deliberate omission.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +470 to +499
): { startMinutes: number; endMinutes: number; label: string } | null {
const match = /^(.+?)\s*(?:-|–|—|to)\s*(.+)$/.exec(
windowText.trim().toLowerCase()
);
if (!match) {
return null;
}

const displayStartRaw = match[1].trim();
const displayEndRaw = match[2].trim();
const { start: startRaw, end: endRaw } = inferWindowMeridiems(
displayStartRaw,
displayEndRaw
);
const startMinutes = parseClockToken(startRaw);
const endMinutes = parseClockToken(endRaw, { allowEndOfDay: true });
if (
startMinutes == null ||
endMinutes == null ||
endMinutes <= startMinutes
) {
return null;
}

return {
startMinutes,
endMinutes,
label: `${displayStartRaw}-${displayEndRaw}`,
};
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Cross-midnight sub-day windows are intentionally rejected

The parseExplicitWindow function in src/tools/lcm-recent-tool.ts:470-499 returns null when endMinutes <= startMinutes, which means windows like "9pm-2am" (crossing midnight) are not supported. This falls through to the error message listing valid period formats.

This appears deliberate — a cross-midnight window would need to span two calendar days, complicating the day-boundary logic. Users can work around this by specifying two separate queries (e.g., "yesterday 9pm-24:00" and "today 0:00-2am").

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/plugin/index.ts
Comment on lines +197 to +205
"1. `lcm_recent` — use for temporal recaps and bounded recent windows (`today`, `yesterday`, `week`, `month`, `date:YYYY-MM-DD`, `yesterday 4-8pm`, `last 3h`) before free-text search.",
"2. `lcm_grep` — search by regex or full-text across messages and summaries",
"3. `lcm_describe` — inspect a specific summary (cheap, no sub-agent)",
"4. `lcm_expand_query` — deep recall: spawns bounded sub-agent, expands DAG, and returns answer plus cited IDs in tool output for follow-up (~120s, don't ration it)",
"",
"**`lcm_recent` routing guidance:**",
"- Use `lcm_recent` for questions anchored by time rather than topic: what happened today/yesterday/this week/this month, a specific date, a local-time window, or a relative range like `last 90m`.",
"- If `lcm_recent` returns summary IDs and the user needs exact detail, follow with `lcm_expand_query` using those IDs.",
"- `lcm_recent` may return prebuilt rollups or a bounded leaf-summary fallback; both preserve source IDs for drill-down.",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Rollup recall policy prompt significantly changes tool escalation ordering

The LOSSLESS_RECALL_POLICY_PROMPT in src/plugin/index.ts:161-206 reorders the tool escalation chain. lcm_recent is now listed as step 1, with lcm_grep demoted to step 2. The prompt also adds 6 new lines of routing guidance specifically for lcm_recent. This is a user-facing behavioral change — models will now prefer temporal retrieval over free-text search for compacted history questions. This matches the PR's intent but operators should be aware the recall heuristic changed.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/rollup-builder.ts
Comment on lines +449 to +461
try {
const finishedAt = new Date();
const latestState = this.store.getState(conversationId);
const shouldClearPending =
result.errors.length === 0 &&
isTimestampAtOrBefore(latestState?.last_message_at, scannedAt);
this.store.upsertState(conversationId, {
timezone: this.config.timezone,
last_rollup_check_at: laterDate(
finishedAt,
latestState?.last_rollup_check_at
).toISOString(),
pending_rebuild:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: buildDailyRollups race condition with concurrent ingest handled via scannedAt guard

In src/rollup-builder.ts:350-461, buildDailyRollups reads state, processes days, then re-reads state before updating. A concurrent ingest could set pending_rebuild: 1 and update last_message_at between the scan start and the final state write. The code correctly guards against this: shouldClearPending at line 451-453 uses isTimestampAtOrBefore(latestState?.last_message_at, scannedAt), which only clears pending_rebuild if no new messages arrived after the scan began. The last_rollup_check_at update uses laterDate() to prevent backwards movement. This is a well-designed optimistic concurrency pattern.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +632 to +641
if (normalized.startsWith("date:")) {
const day = normalized.slice(5);
try {
assertValidPlainDate(day);
} catch {
throw new Error('period date must be in the form "date:YYYY-MM-DD" with a real calendar date.');
}
const start = getUtcDateForZonedMidnight(day, timezone);
const end = getUtcDateForZonedMidnight(addDays(day, 1), timezone);
return { label: day, kind: "day", periodKey: day, start, end };

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Unrecognized window suffix on date:YYYY-MM-DD produces a slightly misleading error

When a user passes period: 'date:2026-04-27 foobar', resolveWindowPeriod at src/tools/lcm-recent-tool.ts:565 matches group1=date:2026-04-27 and group2=foobar, but parseNamedWindow('foobar') and parseExplicitWindow('foobar') both return null, so resolveWindowPeriod returns null. Control falls to line 632 where normalized.startsWith('date:') matches, and day = 'date:2026-04-27 foobar'.slice(5) = '2026-04-27 foobar'. The assertValidPlainDate check fails, throwing 'period date must be in the form "date:YYYY-MM-DD" with a real calendar date.' — the date IS valid, it's the window that's unrecognized. The user still gets a clear error, just with a slightly imprecise message.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +1235 to +1242
const currentDayKey = getZonedDayString(new Date(), timezone);
const canUseStoredCurrentDay =
resolution.periodKey == null || resolution.periodKey !== currentDayKey;
const hasPendingRebuild =
rollupStore.getState(conversationId)?.pending_rebuild === 1;
const canUseStoredResolvedRollup =
canUseStoredCurrentDay &&
(resolution.kind === "day" || !hasPendingRebuild);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Current-day freshness guard prevents stale rollup serving

The lcm_recent tool at src/tools/lcm-recent-tool.ts:1222-1228 implements a freshness guard: canUseStoredCurrentDay is false when the resolved period's key matches today's date, preventing use of a stored (potentially stale) daily rollup for the current day. For multi-day periods like 7d, the code at lines 1302-1358 separately handles the current day via live fallback (getRecentSummaryFallback) while using stored rollups for completed prior days. For weekly/monthly stored rollups, canUseStoredResolvedRollup additionally checks !hasPendingRebuild to avoid serving stale aggregates when daily rollups need rebuilding. This three-layer freshness strategy is well-designed.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +543 to +563
): PeriodResolution | null {
const relative =
/^last\s+(\d+)\s*(h|hr|hrs|hour|hours|m|min|mins|minute|minutes)$/.exec(
normalized
);
if (relative) {
const amount = Number(relative[1]);
const unit = relative[2];
const minutes = unit.startsWith("h") ? amount * 60 : amount;
if (!Number.isFinite(minutes) || minutes <= 0) {
return null;
}
const end = new Date();
const start = new Date(end.getTime() - minutes * 60_000);
return {
label: `last ${amount}${unit.startsWith("h") ? "h" : "m"}`,
start,
end,
window: { relative: true },
};
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Relative time windows ('last Nh/Nm') always use fallback path — never stored rollups

When resolvePeriod processes "last 3h" or "last 90m", the returned PeriodResolution has kind = undefined and periodKey = undefined (src/tools/lcm-recent-tool.ts:541-546). This causes both stored-rollup branches in the execute function (lines 1200-1204 and 1221-1224) to be skipped, always falling through to the leaf-summary SQL fallback. This is correct behavior — relative windows are anchored to new Date() and cannot match pre-built period-keyed rollups.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@100yenadmin

Copy link
Copy Markdown
Member Author

Consolidation note: this public fork PR remains review provenance for the LCM temporal-memory stack. I consolidated the docs + daily + sub-day + aggregate/debug layers onto current upstream main in Martian-Engineering#516 so upstream reviewers can inspect one clean current-main diff. This PR should be read as historical/layered context for that upstream PR rather than the current merge target.

@100yenadmin

Copy link
Copy Markdown
Member Author

Closing this local/fork PR as superseded by the canonical upstream temporal-spine PR:

The consolidated architecture and sprint map is now recorded in:

  • docs/projects/lcm/spec-lcm-ultimate-architecture-plan.md

This local PR remains useful as historical source material only; future work should happen upstream.

@100yenadmin

Copy link
Copy Markdown
Member Author

Closed as part of LCM PR-stack cleanup. See the preceding comment for the canonical upstream PR or follow-up issue.

@100yenadmin

Copy link
Copy Markdown
Member Author

Fork cleanup note: weekly/monthly aggregate rollup scope is superseded by upstream Martian-Engineering/lossless-claw PR Martian-Engineering#516. Review hardening around provenance, bounded maintenance, and debug surfaces is represented in PR Martian-Engineering#526.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants