feat: add weekly and monthly lcm rollups#15
Conversation
📝 WalkthroughWalkthroughAdds temporal rollups: DB migrations and safe ALTER helper; RollupStore persistence and state; RollupBuilder for daily/weekly/monthly aggregates with timezone-aware bounds, deterministic fingerprinting, and token-budgeted rendering; engine wiring to mark and rebuild rollups on ingest/maintenance; two agent tools ( Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant User
participant AgentTool as Agent Tool
participant Engine as LcmContextEngine
participant Store as RollupStore/DB
participant Builder as RollupBuilder
User->>AgentTool: request recent/inspect
AgentTool->>Engine: resolve engine & session
Engine->>Store: getState / query rollups or summaries
Store-->>Engine: rollup rows or leaf summaries
Engine-->>AgentTool: return formatted markdown + details
Note over Engine,Store: on message ingest
Engine->>Store: upsertState(pending_rebuild=1, timezone, last_message_at)
Store-->>Engine: ack
Note over Engine,Builder: maintenance run
Engine->>Store: getState(conversation)
Store-->>Engine: state (pending_rebuild?)
alt pending_rebuild true
Engine->>Builder: buildDailyRollups(conversation)
Builder->>Store: read leaf summaries / day rollups
Store-->>Builder: summaries / day rollups
Builder->>Store: upsertRollup(s) (BEGIN IMMEDIATE txn)
Store-->>Builder: ack
Builder-->>Engine: BuildResult(built/skipped/errors)
Engine->>Store: upsertState(clear pending_rebuild/update timestamps)
else no rebuild
Engine->>Builder: skip build
end
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
|
Adversarial review fixes pushed in Addressed merge-blocking issues:
Verification:
Note: PR #15 now contains hardening that should also be backported/split into #13/#14 before final merge if we keep the PRs stacked against |
|
Follow-up hardening pushed in Additional review-driven fixes:
Re-verified after the change:
|
|
Follow-up hardening pushed locally and about to land:
Verification:
One caveat I kept explicit: current summary rows do not store a separate mutable |
|
Follow-up hardening from PR #14 was carried forward in |
|
Follow-up hardening landed in |
|
Follow-up hardening landed in |
|
Added final hardening on this branch:
I also re-verified that source-token provenance stays separate from summary-token counts in aggregate rollups. |
| /** When true, register the operator-facing lcm_rollup_debug tool. */ | ||
| rollupDebugEnabled: boolean; |
There was a problem hiding this comment.
🔴 AGENTS.md violation: skills/lossless-claw/references/config.md not updated with rollupDebugEnabled
The PR adds the rollupDebugEnabled config option to src/db/config.ts:94-95, openclaw.plugin.json:196-199, docs/configuration.md:115, and README.md:186, but does not update skills/lossless-claw/references/config.md. The AGENTS.md rule explicitly states: "Keep skills/lossless-claw/references/config.md consistent with docs/configuration.md, openclaw.plugin.json, and the runtime defaults in src/db/config.ts. When config keys, aliases, defaults, or precedence rules change, update that bundled skill reference in the same change."
Prompt for agents
The AGENTS.md rule requires updating skills/lossless-claw/references/config.md whenever config keys change. The new rollupDebugEnabled config option was added to src/db/config.ts, openclaw.plugin.json, docs/configuration.md, and README.md but is missing from skills/lossless-claw/references/config.md. Add a section for rollupDebugEnabled to that file, following the same structure as existing entries (e.g. the transcriptGcEnabled or proactiveThresholdCompactionMode sections). Include the key name, type (boolean), default (false), env override (LCM_ROLLUP_DEBUG_ENABLED), and a brief description of its purpose (registers the operator-facing lcm_rollup_debug inspection tool).
Was this helpful? React with 👍 or 👎 to provide feedback.
| | `timezone` | `string` | `TZ` or system timezone | `TZ` | IANA timezone used for timestamp rendering in summaries. | | ||
| | `pruneHeartbeatOk` | `boolean` | `false` | `LCM_PRUNE_HEARTBEAT_OK` | Retroactively removes `HEARTBEAT_OK` turn cycles from persisted storage. | | ||
| | `transcriptGcEnabled` | `boolean` | `false` | `LCM_TRANSCRIPT_GC_ENABLED` | Enables transcript rewrite GC during `maintain()`; disabled by default so transcript rewrites stay opt-in. | | ||
| | `rollupDebugEnabled` | `boolean` | `false` | `LCM_ROLLUP_DEBUG_ENABLED` | Registers the operator-facing `lcm_rollup_debug` inspection tool. | |
There was a problem hiding this comment.
🔴 AGENTS.md violation: docs/configuration.md full example block missing rollupDebugEnabled
The reference table at docs/configuration.md:115 correctly adds rollupDebugEnabled, but the "Complete plugins.entries.lossless-claw.config example" block at lines 15–65 of the same file was not updated. The AGENTS.md rule states: "update the reference tables and the full example plugins.entries.lossless-claw.config block at the top of that file in the same change." Other boolean defaults like transcriptGcEnabled: false (line 47) are shown in the example, so rollupDebugEnabled: false should appear there as well.
Prompt for agents
Add rollupDebugEnabled: false to the full JSON example block at the top of docs/configuration.md (lines 15-65). It should be placed after transcriptGcEnabled: false (line 47), consistent with the key ordering in src/db/config.ts and openclaw.plugin.json. The entry should be: "rollupDebugEnabled": false
Was this helpful? React with 👍 or 👎 to provide feedback.
| rollupDebugEnabled: | ||
| env.LCM_ROLLUP_DEBUG_ENABLED !== undefined | ||
| ? env.LCM_ROLLUP_DEBUG_ENABLED === "true" | ||
| : toBool(pc.rollupDebugEnabled) ?? false, |
There was a problem hiding this comment.
🔴 AGENTS.md violation: no regression test for new rollupDebugEnabled config option
The PR adds rollupDebugEnabled as a new config option with env override LCM_ROLLUP_DEBUG_ENABLED and default false, but does not add or update any test in test/config.test.ts. The AGENTS.md rule states: "Add or update a regression test when changing config options so schema drift is caught before release." The existing config tests (e.g. expect(config.transcriptGcEnabled).toBe(false) at test/config.test.ts:41) follow a clear pattern: the defaults test verifies the default, the env-override test verifies env takes precedence, and the manifest test verifies the schema entry exists. None of these exist for rollupDebugEnabled.
Prompt for agents
Add regression tests for the new rollupDebugEnabled config option in test/config.test.ts. Following the existing patterns:
1. In the defaults test (around line 41), add: expect(config.rollupDebugEnabled).toBe(false)
2. In the env-override test (around line 71), add rollupDebugEnabled: true to the plugin config fixture and LCM_ROLLUP_DEBUG_ENABLED: 'true' to the env, then verify: expect(config.rollupDebugEnabled).toBe(true)
3. In the manifest schema test (around line 551), add: expect(manifest.configSchema.properties.rollupDebugEnabled).toEqual({ type: 'boolean' })
Was this helpful? React with 👍 or 👎 to provide feedback.
|
|
||
| function uniqueStrings(values: string[]): string[] { | ||
| return [...new Set(values)]; | ||
| } | ||
|
|
||
| function startOfWeekKey(dayKey: string, timezone: string): string { | ||
| void timezone; | ||
| assertValidDateKey(dayKey); | ||
| const [year, month, day] = dayKey | ||
| .split("-") | ||
| .map((part) => Number.parseInt(part, 10)); |
There was a problem hiding this comment.
📝 Info: startOfWeekKey ignores its timezone parameter — correct because day-of-week is timezone-invariant for date strings
The startOfWeekKey function at src/rollup-builder.ts:1101 accepts a timezone parameter but explicitly ignores it with void timezone. This is correct: a date key like 2026-04-27 represents Monday regardless of timezone — the day-of-week calculation only needs the calendar date, not the timezone. The Date.UTC() construction and getUTCDay() call correctly determine the weekday for any YYYY-MM-DD string. The void timezone is presumably intentional to match the function signature pattern used elsewhere while documenting the deliberate omission.
Was this helpful? React with 👍 or 👎 to provide feedback.
| ): { startMinutes: number; endMinutes: number; label: string } | null { | ||
| const match = /^(.+?)\s*(?:-|–|—|to)\s*(.+)$/.exec( | ||
| windowText.trim().toLowerCase() | ||
| ); | ||
| if (!match) { | ||
| return null; | ||
| } | ||
|
|
||
| const displayStartRaw = match[1].trim(); | ||
| const displayEndRaw = match[2].trim(); | ||
| const { start: startRaw, end: endRaw } = inferWindowMeridiems( | ||
| displayStartRaw, | ||
| displayEndRaw | ||
| ); | ||
| const startMinutes = parseClockToken(startRaw); | ||
| const endMinutes = parseClockToken(endRaw, { allowEndOfDay: true }); | ||
| if ( | ||
| startMinutes == null || | ||
| endMinutes == null || | ||
| endMinutes <= startMinutes | ||
| ) { | ||
| return null; | ||
| } | ||
|
|
||
| return { | ||
| startMinutes, | ||
| endMinutes, | ||
| label: `${displayStartRaw}-${displayEndRaw}`, | ||
| }; | ||
| } |
There was a problem hiding this comment.
📝 Info: Cross-midnight sub-day windows are intentionally rejected
The parseExplicitWindow function in src/tools/lcm-recent-tool.ts:470-499 returns null when endMinutes <= startMinutes, which means windows like "9pm-2am" (crossing midnight) are not supported. This falls through to the error message listing valid period formats.
This appears deliberate — a cross-midnight window would need to span two calendar days, complicating the day-boundary logic. Users can work around this by specifying two separate queries (e.g., "yesterday 9pm-24:00" and "today 0:00-2am").
Was this helpful? React with 👍 or 👎 to provide feedback.
| "1. `lcm_recent` — use for temporal recaps and bounded recent windows (`today`, `yesterday`, `week`, `month`, `date:YYYY-MM-DD`, `yesterday 4-8pm`, `last 3h`) before free-text search.", | ||
| "2. `lcm_grep` — search by regex or full-text across messages and summaries", | ||
| "3. `lcm_describe` — inspect a specific summary (cheap, no sub-agent)", | ||
| "4. `lcm_expand_query` — deep recall: spawns bounded sub-agent, expands DAG, and returns answer plus cited IDs in tool output for follow-up (~120s, don't ration it)", | ||
| "", | ||
| "**`lcm_recent` routing guidance:**", | ||
| "- Use `lcm_recent` for questions anchored by time rather than topic: what happened today/yesterday/this week/this month, a specific date, a local-time window, or a relative range like `last 90m`.", | ||
| "- If `lcm_recent` returns summary IDs and the user needs exact detail, follow with `lcm_expand_query` using those IDs.", | ||
| "- `lcm_recent` may return prebuilt rollups or a bounded leaf-summary fallback; both preserve source IDs for drill-down.", |
There was a problem hiding this comment.
📝 Info: Rollup recall policy prompt significantly changes tool escalation ordering
The LOSSLESS_RECALL_POLICY_PROMPT in src/plugin/index.ts:161-206 reorders the tool escalation chain. lcm_recent is now listed as step 1, with lcm_grep demoted to step 2. The prompt also adds 6 new lines of routing guidance specifically for lcm_recent. This is a user-facing behavioral change — models will now prefer temporal retrieval over free-text search for compacted history questions. This matches the PR's intent but operators should be aware the recall heuristic changed.
Was this helpful? React with 👍 or 👎 to provide feedback.
| try { | ||
| const finishedAt = new Date(); | ||
| const latestState = this.store.getState(conversationId); | ||
| const shouldClearPending = | ||
| result.errors.length === 0 && | ||
| isTimestampAtOrBefore(latestState?.last_message_at, scannedAt); | ||
| this.store.upsertState(conversationId, { | ||
| timezone: this.config.timezone, | ||
| last_rollup_check_at: laterDate( | ||
| finishedAt, | ||
| latestState?.last_rollup_check_at | ||
| ).toISOString(), | ||
| pending_rebuild: |
There was a problem hiding this comment.
📝 Info: buildDailyRollups race condition with concurrent ingest handled via scannedAt guard
In src/rollup-builder.ts:350-461, buildDailyRollups reads state, processes days, then re-reads state before updating. A concurrent ingest could set pending_rebuild: 1 and update last_message_at between the scan start and the final state write. The code correctly guards against this: shouldClearPending at line 451-453 uses isTimestampAtOrBefore(latestState?.last_message_at, scannedAt), which only clears pending_rebuild if no new messages arrived after the scan began. The last_rollup_check_at update uses laterDate() to prevent backwards movement. This is a well-designed optimistic concurrency pattern.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (normalized.startsWith("date:")) { | ||
| const day = normalized.slice(5); | ||
| try { | ||
| assertValidPlainDate(day); | ||
| } catch { | ||
| throw new Error('period date must be in the form "date:YYYY-MM-DD" with a real calendar date.'); | ||
| } | ||
| const start = getUtcDateForZonedMidnight(day, timezone); | ||
| const end = getUtcDateForZonedMidnight(addDays(day, 1), timezone); | ||
| return { label: day, kind: "day", periodKey: day, start, end }; |
There was a problem hiding this comment.
📝 Info: Unrecognized window suffix on date:YYYY-MM-DD produces a slightly misleading error
When a user passes period: 'date:2026-04-27 foobar', resolveWindowPeriod at src/tools/lcm-recent-tool.ts:565 matches group1=date:2026-04-27 and group2=foobar, but parseNamedWindow('foobar') and parseExplicitWindow('foobar') both return null, so resolveWindowPeriod returns null. Control falls to line 632 where normalized.startsWith('date:') matches, and day = 'date:2026-04-27 foobar'.slice(5) = '2026-04-27 foobar'. The assertValidPlainDate check fails, throwing 'period date must be in the form "date:YYYY-MM-DD" with a real calendar date.' — the date IS valid, it's the window that's unrecognized. The user still gets a clear error, just with a slightly imprecise message.
Was this helpful? React with 👍 or 👎 to provide feedback.
| const currentDayKey = getZonedDayString(new Date(), timezone); | ||
| const canUseStoredCurrentDay = | ||
| resolution.periodKey == null || resolution.periodKey !== currentDayKey; | ||
| const hasPendingRebuild = | ||
| rollupStore.getState(conversationId)?.pending_rebuild === 1; | ||
| const canUseStoredResolvedRollup = | ||
| canUseStoredCurrentDay && | ||
| (resolution.kind === "day" || !hasPendingRebuild); |
There was a problem hiding this comment.
📝 Info: Current-day freshness guard prevents stale rollup serving
The lcm_recent tool at src/tools/lcm-recent-tool.ts:1222-1228 implements a freshness guard: canUseStoredCurrentDay is false when the resolved period's key matches today's date, preventing use of a stored (potentially stale) daily rollup for the current day. For multi-day periods like 7d, the code at lines 1302-1358 separately handles the current day via live fallback (getRecentSummaryFallback) while using stored rollups for completed prior days. For weekly/monthly stored rollups, canUseStoredResolvedRollup additionally checks !hasPendingRebuild to avoid serving stale aggregates when daily rollups need rebuilding. This three-layer freshness strategy is well-designed.
Was this helpful? React with 👍 or 👎 to provide feedback.
| ): PeriodResolution | null { | ||
| const relative = | ||
| /^last\s+(\d+)\s*(h|hr|hrs|hour|hours|m|min|mins|minute|minutes)$/.exec( | ||
| normalized | ||
| ); | ||
| if (relative) { | ||
| const amount = Number(relative[1]); | ||
| const unit = relative[2]; | ||
| const minutes = unit.startsWith("h") ? amount * 60 : amount; | ||
| if (!Number.isFinite(minutes) || minutes <= 0) { | ||
| return null; | ||
| } | ||
| const end = new Date(); | ||
| const start = new Date(end.getTime() - minutes * 60_000); | ||
| return { | ||
| label: `last ${amount}${unit.startsWith("h") ? "h" : "m"}`, | ||
| start, | ||
| end, | ||
| window: { relative: true }, | ||
| }; | ||
| } |
There was a problem hiding this comment.
📝 Info: Relative time windows ('last Nh/Nm') always use fallback path — never stored rollups
When resolvePeriod processes "last 3h" or "last 90m", the returned PeriodResolution has kind = undefined and periodKey = undefined (src/tools/lcm-recent-tool.ts:541-546). This causes both stored-rollup branches in the execute function (lines 1200-1204 and 1221-1224) to be skipped, always falling through to the leaf-summary SQL fallback. This is correct behavior — relative windows are anchored to new Date() and cannot match pre-built period-keyed rollups.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
Consolidation note: this public fork PR remains review provenance for the LCM temporal-memory stack. I consolidated the docs + daily + sub-day + aggregate/debug layers onto current upstream main in Martian-Engineering#516 so upstream reviewers can inspect one clean current-main diff. This PR should be read as historical/layered context for that upstream PR rather than the current merge target. |
|
Closing this local/fork PR as superseded by the canonical upstream temporal-spine PR: The consolidated architecture and sprint map is now recorded in:
This local PR remains useful as historical source material only; future work should happen upstream. |
|
Closed as part of LCM PR-stack cleanup. See the preceding comment for the canonical upstream PR or follow-up issue. |
|
Fork cleanup note: weekly/monthly aggregate rollup scope is superseded by upstream Martian-Engineering/lossless-claw PR Martian-Engineering#516. Review hardening around provenance, bounded maintenance, and debug surfaces is represented in PR Martian-Engineering#526. |
Summary
Builds on #13/#14 by adding calendar weekly/monthly aggregate rollups and the operator-only
lcm_rollup_debugtool. This completes the first temporal-memory stack: daily recap, sub-day windows, rolling 7d/30d recap, calendar week/month aggregates, provenance, and debug visibility.lcm_recentremains a recap/window-entry tool, not proof. Exact commands, paths, timestamps, root cause, and shipped/decided claims still requirelcm_describeorlcm_expand_query.What Changed
lcm_rollup_debugbehind explicitrollupDebugEnabled/LCM_ROLLUP_DEBUG_ENABLEDoperator gating, with strict period/status handling, clamped limits, and source IDs hidden unlessincludeSources=true.source_message_countcounts source messages, aggregate counts sum source-message counts, and source-token counts remain separate.night/24:00, UTC+13 keys, canonical UTC timestamp outputs, expression indexes, and state-preserving upserts.Out Of Scope
Validation
npm test -- --run test/rollup-store-builder.test.ts test/lcm-tools.test.ts— 33 tests passed.npm run build— passed.git diff --check— clean.Reviewer Notes
Known blockers addressed: source-message provenance counts, aggregate retry state, rebuild maintenance placement, pending aggregate freshness, content-change fingerprints, debug source hiding, strict status/period validation, unused retrieval removal, timezone-safe aggregate inputs, timezone-qualified uniqueness/lookups, empty-day stale rollup deletion, UTC+13 week/month aggregation, 7d/30d complete-rollup plus live-current-day behavior, complete aggregate coverage, lcm_recent includeSources/budget enforcement, direct empty-day deletion, and orphaned aggregate sweeps, quiet-day aggregate completeness, DST-skipped-midnight day bounds, sub-1000 output budgets, and corrected confidence/token reporting, canonical week-key validation, unbounded aggregate key discovery, and idempotent aggregate fingerprints, sparse inactive-day 7d coverage, and skipped-midnight
24:00window endpoints, and monotonic daily sweeplast_rollup_check_atupdates, DST-safe week/month aggregate bounds, shared Unicode-aware token estimates, SQL fallback cap accounting, and truncation propagation, aggregate phase timestamp preservation, stale-row rebuild restoration, final daily sweep-state failure reporting, non-fake leaf updated_at semantics, exact fallback total counts past the SQL cap, debug limit clamping, and operator-gated debug registration, and 30-day maintenance backfill for30drecall.Summary by CodeRabbit