Skip to content

feat: run LCM observed extraction in maintenance (7/10)#540

Closed
100yenadmin wants to merge 45 commits into
Martian-Engineering:mainfrom
electricsheephq:feat/lcm-extraction-maintenance
Closed

feat: run LCM observed extraction in maintenance (7/10)#540
100yenadmin wants to merge 45 commits into
Martian-Engineering:mainfrom
electricsheephq:feat/lcm-extraction-maintenance

Conversation

@100yenadmin

@100yenadmin 100yenadmin commented Apr 29, 2026

Copy link
Copy Markdown
Collaborator

PR #540: Maintenance wiring for extraction and event processing

Maintainer Quick Read

Field Value
Stack position 7/10
Logical parent for review #538
Current head 800354d63f3b2cc1c2504d4b261bf45a94e87ab3
Current PR state Ready after #538
Merge note Explicit write path; keeps read tools read-only.
Last body refresh 2026-05-02

Why Maintainers Should Care

This connects extraction/event processing to maintenance so LCM can update observed-work and event observations outside user-facing reads. The key product point is trust: asking a question does not secretly mutate memory.

Stack Map

flowchart LR
  P516["#516 Temporal spine"] --> P517["#517 Observed work read model"]
  P517 --> P518["#518 Suggestion ledger"]
  P518 --> P537["#537 Deterministic extraction"]
  P537 --> P538["#538 Event observations"]
  P518 --> P539["#539 Opt-in suggestion tools"]
  P538 --> P540["#540 Maintenance wiring"]
  P540 --> P530["#530 Tracker/open state"]
  P538 --> P531["#531 Event episodes"]
  P530 --> P532["#532 Topic/vocabulary"]
  P526["#526 Draft reference"] -. "not the merge path" .-> P516
Loading

How It Works

flowchart TD
  A["Summary-producing maintenance"] --> B["Observed-work extraction sweep"]
  B --> C["Event-observation sweep"]
  C --> D["Persist cursor/retry state"]
  D --> E["Later reads see updated evidence"]
  F["lcm_recent / lcm_work_density / lcm_event_search"] -. "read-only" .-> E
Loading

What Ships In This PR

  • maintenance hooks for observed-work extraction
  • maintenance hooks for event observation extraction
  • bounded sweeps with retry state
  • ordering after summary-producing work
  • tests that read tools do not perform maintenance writes

What This PR Intentionally Does Not Do

  • does not run extraction from every read
  • does not add background reminders/wakes
  • does not grant task/Cortex write authority

How An Agent Uses It

  • After new summaries are produced, maintenance processes only the new evidence window.
  • If a batch fails, retry state preserves the failure instead of losing coverage.
  • A read tool call observes whatever maintenance already produced; it does not kick off a write.

Engineering Boundaries

  • LCM is an evidence layer. It does not become Cortex, GBrain, OpenClaw Tasks, or a fuzzy semantic authority.
  • Recaps and observed states are entry points for investigation, not proof of exact commands, SHAs, paths, timestamps, root cause, or task completion.
  • Exact claims still require source drilldown through lcm_describe, lcm_expand, lcm_expand_query, raw messages, GitHub, logs, tickets, or the relevant authority.
  • Source identifiers stay hidden by default unless a tool has an explicit includeSources or debug affordance.
  • Reads stay read-only. Writes happen through explicit storage, maintenance, extraction, or review-ledger paths for the slice that owns them.

Primary Review Files

  • src/engine.ts
  • src/maintenance.ts
  • src/observed-work-extractor.ts
  • src/event-observation-extractor.ts
  • test/observed-work-store.test.ts

Size And Review Surface

These are intentionally large LCM changes, so the body separates the GitHub-visible diff from the intended logical review slice. Later PRs target main, which makes the GitHub Files tab include parent-stack code; reviewers should use the logical parent listed above when judging the slice.

View Compare base Files Additions Deletions
GitHub-visible PR page main 40 12172 51
Intended reviewer slice #538 12 250 39

Logical slice breakdown:

Category Files Additions Deletions
runtime code 5 77 31
tests 3 154 8
docs/release notes 3 12 0
metadata/package 1 7 0

Review Concerns Already Folded Into The Code

  • Fallback message timestamp scans keep mixed-format julianday() correctness and now use matching expression indexes for indexed range/order plans.
  • Maintenance runs after summary-producing work.
  • Retry state is preserved on failure.
  • Read-vs-write path separation is covered by tests.
  • No task/Cortex side effects are introduced.

Validation Evidence

  • npm test -- --run test/observed-work-store.test.ts test/rollup-store-builder.test.ts test/task-bridge-suggestion-store.test.ts
  • npm test
  • npm run build
  • git diff --check
  • GitHub test check: success

Suggested Maintainer Review Path

  1. Start with the Why Maintainers Should Care and How It Works sections.
  2. Use Primary Review Files for the logical slice; the GitHub Files tab can look much larger because the public PRs target main while the series is logically stacked.
  3. Check the non-goals before asking whether LCM should create tasks, close tasks, write Cortex memory, or make authoritative project claims.
  4. Review and merge in the order shown in the stack diagram. feat: implement LCM architecture follow-up (draft) #526 stays draft/reference unless maintainers explicitly choose a single large integration branch.

Copilot AI review requested due to automatic review settings April 29, 2026 09:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Wires deterministic observed-work and event extraction into LcmContextEngine.maintain() so all extraction/mutation happens only on the explicit maintenance/write path (not via read tools), and introduces/extends supporting storage + tool surfaces for temporal rollups, observed-work density, and event search.

Changes:

  • Run rollup building + observed-work/event extraction during maintain(), with logging and failure isolation/pending-rebuild preservation.
  • Add SQLite migrations + stores for rollups, observed work, event observations, and inert task-bridge suggestions.
  • Register/read tools for temporal recaps (lcm_recent), work density (lcm_work_density), event search (lcm_event_search), and operator-gated rollup inspection (lcm_rollup_debug), with updated docs/tests.

Reviewed changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
test/task-bridge-suggestion-store.test.ts Adds coverage for inert task-bridge suggestion storage semantics.
test/plugin-prompt-hook.test.ts Updates prompt-hook assertions for new recall-order guidance.
test/observed-work-store.test.ts Adds observed-work + event observation + tool behavior tests (redaction, cursoring, bounded reads).
test/engine.test.ts Adds regression test ensuring extraction runs only during maintain() (read-path remains non-mutating).
src/tools/lcm-work-density-tool.ts Implements lcm_work_density tool with period filtering, redaction, and bounded scope behavior.
src/tools/lcm-rollup-debug-tool.ts Adds operator-gated rollup inspection tool with optional source inclusion.
src/tools/lcm-event-search-tool.ts Implements lcm_event_search tool for deterministic event observations (sources hidden by default).
src/timezone-windows.ts Provides shared timezone/date-window helpers used by period resolution.
src/store/task-bridge-suggestion-store.ts Adds inert suggestion ledger store with validation and review-state updates.
src/store/rollup-store.ts Adds rollup persistence/state APIs and leaf-summary range reads for rollup building.
src/store/observed-work-store.ts Adds observed-work persistence, density queries, provenance, and processing state handling.
src/store/index.ts Re-exports EventObservationStore types for broader consumption.
src/store/event-observation-store.ts Adds event observation persistence and query APIs with source redaction.
src/rollup-builder.ts Adds deterministic daily + weekly/monthly rollup building and fingerprinting logic.
src/plugin/index.ts Registers new tools and extends system prompt guidance; gates rollup debug tool by config.
src/observed-work-extractor.ts Implements deterministic extraction from leaf summaries with cursoring + optional event upserts.
src/engine.ts Instantiates new stores/builders and runs rollup + extraction work during maintain().
src/db/migration.ts Adds idempotent migrations for rollups, observed work, task-bridge suggestions, and event observations.
src/db/config.ts Adds rollupDebugEnabled config/env surface for gating rollup debug tool.
specs/lcm-temporal-memory-plan.md Documents temporal-memory/rollup plan and tool-selection guidance.
specs/lcm-task-bridge-option-c-experimental.md Documents experimental task-bridge approach and inert-store intent.
specs/lcm-observed-work-density-option-b.md Documents observed-work density model and tool contract.
skills/lossless-claw/references/recall-tools.md Updates recall-tool reference with availability-gated temporal guidance.
skills/lossless-claw/SKILL.md Updates skill description and workflow guidance to include lcm_recent when available.
openclaw.plugin.json Adds config schema/help for rollupDebugEnabled.
docs/configuration.md Documents LCM_ROLLUP_DEBUG_ENABLED / rollupDebugEnabled.
README.md Documents LCM_ROLLUP_DEBUG_ENABLED environment variable.
.changeset/lcm-temporal-skill-docs.md Changeset for skill-doc guidance updates.
.changeset/lcm-temporal-rollups.md Changeset for temporal rollups + tools.
.changeset/lcm-task-bridge-suggestions.md Changeset for inert task-bridge suggestion storage.
.changeset/lcm-observed-work-extraction.md Changeset for observed-work extraction addition.
.changeset/lcm-observed-work-density.md Changeset for observed-work density model + tool.
.changeset/lcm-extraction-maintenance.md Changeset for maintenance-path extraction wiring.
.changeset/lcm-event-observations.md Changeset for event observations + tool.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .changeset/lcm-extraction-maintenance.md Outdated
Comment thread src/observed-work-extractor.ts Outdated
Comment thread src/plugin/index.ts
@100yenadmin

Copy link
Copy Markdown
Collaborator Author

Review-thread triage at current head a90131d:

  • Fixed: changeset frontmatter now targets @martian-engineering/lossless-claw.
  • Fixed/inherited: observed-work extraction rowid cursoring derives rowid from lastProcessedSummaryId before using the stored rowid fallback, so rowid drift cannot skip same-second summaries.
  • Safe by stack shape: the tool-registration diff is inherited from the logical parent stack because this PR targets main; this PR's intended scope is maintenance write-path wiring for observed-work/event processing.

Local focused gates passed: test/observed-work-store.test.ts, test/engine.test.ts, test/rollup-store-builder.test.ts, npm run build, and git diff --check. GitHub CI was restarted for a90131d.

@100yenadmin

Copy link
Copy Markdown
Collaborator Author

Final adversarial pass - merge order 7/10, after #538

Score: mergeability 9.4/10, functionality 9.0/10, description 9.5/10.

Functionality audit: Works as production maintenance wiring for observed-work/event extraction. Read tools do not mutate; maintenance writes are gated by config; retry failures preserve pending state. The final hardening moved event observation writes into the per-summary savepoint so failed observed-work writes do not leak event rows.

Scenario coverage: Makes the observed-work/event layers useful at runtime through explicit maintenance, while preserving read/write-path separation.

Final state: MERGEABLE, GitHub test pass, GraphQL review threads: 0 unresolved. Local focused suite, full npm test, npm run build, and git diff --check passed on head a7693aa.

Residual caveat: Writes happen through maintenance only, not user read tools.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated 7 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread README.md
Comment thread src/tools/lcm-work-density-tool.ts
Comment thread test/config.test.ts
Comment thread src/tools/lcm-event-search-tool.ts
Comment thread src/observed-work-extractor.ts Outdated
Comment thread test/observed-work-store.test.ts
Comment thread README.md
@100yenadmin

Copy link
Copy Markdown
Collaborator Author

Final May 1 hardening/status update:

  • Merge order: 7/10 — Maintenance wiring
  • Head: bef331f
  • Review state: 0 unresolved GraphQL review threads after the latest push
  • GitHub state: MERGEABLE / CLEAN, test check green
  • Validation: 144 focused tests; build; diff-check
  • Score: Functionality 10/10, mergeability 10/10, description 10/10
  • Note: Ready after feat: add LCM event observations (5/10) #538. Extraction/event writes happen through explicit maintenance, not read tools; config and README surfaces are covered.

The PR description top note has been refreshed with current head SHA, visible-vs-logical LOC breakdown, scope, caveats, and validation status. LCM remains an evidence layer: recaps, observed work, events, episodes, and suggestions are entry points, not proof or task authority.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +160 to +166
rollups: includeSources
? rollups
: rollups.map((rollup) => {
const { source_summary_ids: omitted, ...redacted } = rollup;
void omitted;
return redacted;
}),
Comment on lines +304 to +308
const limit = typeof p.limit === "number" ? Math.trunc(p.limit) : 5;
const detailLevel = typeof p.detailLevel === "number" ? Math.trunc(p.detailLevel) : 1;
const topic = typeof p.topic === "string" && p.topic.trim() ? p.topic.trim() : undefined;
const minConfidence = typeof p.minConfidence === "number" ? p.minConfidence : undefined;
const store = lcm.getObservedWorkStore();
if (query) {
const likeQuery = `%${escapeLikePattern(query)}%`;
where.push(
"(lower(coalesce(query_key, '')) = ? OR lower(title) LIKE ? ESCAPE '\\' OR lower(coalesce(description, '')) LIKE ? ESCAPE '\\')"
@100yenadmin 100yenadmin changed the title feat: run LCM observed extraction in maintenance feat: run LCM observed extraction in maintenance (7/10) May 3, 2026
@100yenadmin 100yenadmin closed this May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants