Skip to content

fix: tighten cache-aware compaction, logging, and docs#329

Merged
jalehman merged 13 commits intomainfrom
fix/prerelease-cleanup
Apr 8, 2026
Merged

fix: tighten cache-aware compaction, logging, and docs#329
jalehman merged 13 commits intomainfrom
fix/prerelease-cleanup

Conversation

@jalehman
Copy link
Copy Markdown
Contributor

@jalehman jalehman commented Apr 8, 2026

What

This branch packages the prerelease cleanup work on fix/prerelease-cleanup into one PR. It combines cache-aware incremental-compaction tuning, runtime logging cleanup, bootstrap replay hardening, command/UX polish, and the matching config/schema/docs/test updates.

Why

These commits were developed across several topics on the same branch, but they converge on making lossless-claw safer to run in production and easier to operate ahead of the next release. A single PR preserves the actual branch history while giving reviewers one place to evaluate the combined behavior changes.

Changes

  • Retune hot-cache incremental compaction with hysteresis, headroom checks, bounded cold-cache catch-up, and new config knobs
  • Route LCM operational logs through the OpenClaw runtime logger and promote key decision traces to info
  • Keep bootstrap append-only replay on the fast path after heartbeat pruning and non-message transcript sidecars
  • Keep raw summary IDs out of normal user-facing guidance and add Telegram native progress placeholder text
  • Sync manifest schema, README, config docs, and bundled skill reference with the runtime config surface
  • Ignore committed Pebbles state and harden Vitest isolation around HOME and default DB placement

Testing

  • npm test
  • Current branch result: fails with 24 tests across test/circuit-breaker.test.ts, test/engine.test.ts, and test/summarize.test.ts
File Change
src/engine.ts Hot-cache policy, bootstrap replay, afterTurn logging
src/compaction.ts Logger injection and condensed-pass gating
src/lcm-log.ts Shared runtime/file-backed logger adapter
src/plugin/index.ts Runtime logger wiring, startup logging, command guidance
src/summarize.ts Summarizer diagnostics now use runtime logger
src/db/config.ts New cache-aware knobs and dynamic-leaf defaults
openclaw.plugin.json Config schema and UI hints synced with runtime
README.md Clarify command surfaces and allowlist behavior
docs/configuration.md Document cache-aware compaction behavior
test/engine.test.ts Coverage for hot-cache policy and bootstrap replay
test/plugin-config-registration.test.ts Assert runtime logger usage and native progress text
vitest.config.ts Isolated HOME and test DB defaults

Closes #321
Closes #311
Closes #282
Closes #276
Closes #272

@jalehman jalehman marked this pull request as ready for review April 8, 2026 20:19
jalehman added 12 commits April 8, 2026 14:55
Add a repo-level Vitest config that pins discovery to the root test tree, excludes .worktrees, and gives tests a sandbox HOME so default LCM database paths resolve away from ~/.openclaw. Add a regression test that asserts resolveLcmConfig uses the sandboxed HOME for its default database path.

Regeneration-Prompt: |
  The lossless-claw test suite was vulnerable to two isolation problems. Vitest could discover duplicate test files from .worktrees when invoked with default or filtered discovery, and plugin tests that relied on the default LCM database path could touch the real ~/.openclaw/lcm.db because register() eagerly opens the configured database.

  Fix this without changing existing runtime behavior outside tests. Add repo-level Vitest configuration that constrains discovery to the root test/ tree and explicitly excludes .worktrees. Also give the Vitest process a temporary HOME so any test that relies on the default LCM database path resolves into a throwaway .openclaw directory instead of the user's live home directory. Preserve dbPath precedence so tests that explicitly pass a database path still verify that behavior unchanged.

  Add a small regression test that proves resolveLcmConfig defaults to HOME/.openclaw/lcm.db under the Vitest sandbox.
Stop injected lossless-claw prompt guidance from encouraging user-visible summary IDs in normal assistant prose. Keep cited IDs available in tool output for follow-up and update prompt coverage expectations accordingly.

Regeneration-Prompt: |
  User noticed normal responses from LCM-backed recall sometimes ended with raw citation handles like sum_abc in prose and wanted those hidden. Investigate whether the IDs were being post-processed into messages or whether injected prompt guidance was teaching the model to expose them.

  Preserve the retrieval workflow and follow-up traceability: tool outputs can still carry cited summary IDs, but normal user-facing answers should not mention them unless the user explicitly asks for sources or IDs. Make the smallest additive change by adjusting the static lossless recall policy, the dynamic compacted-context prompt guidance, and any tool descriptions that imply surfaced IDs. Update the prompt assertions to lock in the new behavior without trying to fix unrelated existing test failures in the branch.
Raise hot-cache preservation so incremental compaction defers more aggressively when prompt cache is healthy. Make dynamic leaf sizing enabled by default, add configurable hot-cache pressure and budget-headroom thresholds, apply short hot-cache hysteresis, and suppress follow-on condensed passes during hot-cache maintenance. Update config schema/docs and add regression coverage for the new policy decisions and condensed-pass gating.

Regeneration-Prompt: |
  We had already fixed the bootstrap/import-cap path, and the remaining LCM work was to make cache-aware incremental compaction behave more like a cache-preservation policy than a weak leaf-trigger tweak. The desired behavior was: configurable hot-cache pressure factor with a much higher default, dynamic leaf sizing on by default with max at 2x the static floor, a true hot-cache skip when assembled context is comfortably under the real token budget, a short hysteresis window for recent cache hits, and no follow-on condensed passes when hot-cache maintenance still has to run.

  Implement this additively in lossless-claw without changing unrelated behavior. Update the runtime config surface, manifest schema, and configuration docs together. In the engine, stretch hot-cache trigger evaluation to the dynamic max chunk size, preserve cold-cache bounded catch-up passes, and thread an allowCondensedPasses decision into incremental compaction so hot-cache maintenance can stay leaf-only. Add targeted tests covering config defaults/overrides, hot-cache budget-headroom skips, hysteresis, hot-cache leaf-only decisions, and compactLeaf suppression of condensed passes.
Document that /lcm and /lossless are plugin slash/native commands on supported OpenClaw command surfaces, not root shell CLI subcommands. Clarify that plugins.allow must contain the plugin id lossless-claw rather than command tokens like lossless or /lcm, and update the MVP spec to keep registerCli out of scope for this command surface.

Regeneration-Prompt: |
  The docs around lossless-claw command access were ambiguous enough that a user treated /lcm and /lossless as root shell CLI commands and then hit a misleading OpenClaw plugins.allow error. Update the README and the internal MVP command spec so they explicitly distinguish plugin slash/native commands from root CLI subcommands. Preserve the existing intended behavior: the plugin provides /lcm and /lossless on supported chat/native command surfaces, but it does not currently expose openclaw lcm or openclaw lossless via registerCli. Also clarify that plugins.allow accepts plugin ids like lossless-claw, not command aliases such as lossless or /lcm.
Bring the bundled lossless-claw skill config reference back in line with the canonical configuration docs and current runtime defaults. Document the new cache-aware compaction knobs and behavior, note dynamic leaf sizing as default-on, and add an explicit repo rule that this skill reference must stay synchronized with docs/configuration.md, openclaw.plugin.json, and src/db/config.ts.

Regeneration-Prompt: |
  After finishing the cache-aware compaction work, we noticed the bundled skill documentation had drifted from the real config surface. The top-level skill guidance was still fine, but skills/lossless-claw/references/config.md was missing the new hot-cache knobs, still implied older defaults, and did not explain the new hot-cache decision logs. Update that reference so an operator reading the bundled skill gets guidance consistent with docs/configuration.md, openclaw.plugin.json, and src/db/config.ts.

  Also tighten the repository instructions so future config changes must update the bundled skill reference in the same change. Keep the commit scoped to AGENTS.md and the skill reference only, because the branch already contains unrelated modified files that should not be swept into this docs sync commit.
Fix the startup/session-stream pollution behind issue #321 by routing LCM operational logs through OpenClaw's runtime child logger instead of console or session logging. This centralizes plugin, engine, compaction, and summarizer logging onto the file-backed logger path while preserving machine-readable --json stdout behavior.\n\nAdds regression coverage to assert startup banners stay off the session logger and updates logger-dependent tests to inject LCM log spies directly.\n\nRegeneration-Prompt: |\n  Overhaul LCM logging so non-error operational logs no longer appear in the\n  OpenClaw session output stream or interfere with commands that reserve stdout\n  for machine-readable JSON. The existing behavior was split between stdout and\n  stderr because earlier fixes tried to protect [plugins] qmd-memory: registered (mode: query, collections: memories, granola, maxResults: 3, minScore: 0.5)
[plugins] qmd-memory: registered (mode: query, collections: memories, granola, maxResults: 3, minScore: 0.5)
[plugins] qmd-memory: registered (mode: query, collections: memories, granola, maxResults: 3, minScore: 0.5)
[plugins] qmd-memory: registered (mode: query, collections: memories, granola, maxResults: 3, minScore: 0.5)
[plugins] qmd-memory: registered (mode: query, collections: memories, granola, maxResults: 3, minScore: 0.5), but that\n  left startup banners and other diagnostics with inconsistent UX.\n\n  Prefer OpenClaw's runtime child logger when available, because it writes to\n  the proper file-backed gateway log surface without polluting chat/TUI output.\n  Fall back to the host logger only if the runtime logger is unavailable. Remove\n  remaining direct  logging in the plugin, engine, compaction, and\n  summarizer paths so LCM has one coherent logging story. Preserve existing log\n  messages where practical, and add regression tests proving startup banners use\n  the runtime/file logger rather than the session logger while  remains\n  safe.
Add the Telegram native progress placeholder metadata to the /lossless plugin command so native Telegram command execution sends an immediate working message and then edits it in place with the final doctor-apply result. Add focused command coverage to keep the opt-in metadata from regressing.

Regeneration-Prompt: |
  User reported that /lossless doctor apply in Telegram no longer shows an immediate working message before the final result replaces it. Investigate the lossless-claw plugin command registration against the complementary OpenClaw Telegram native-command change that introduced progress-placeholder opt-in for plugin commands.

  Preserve the existing /lossless command surface and the current final text reply behavior. Do not rework Telegram delivery logic in lossless-claw; instead, wire the plugin into the host contract by setting the command metadata that Telegram checks before sending an editable placeholder. Add a targeted regression test at the command-construction layer so future edits do not drop the Telegram progress metadata again.
Restore legacy bootstrap replay compatibility for bare JSONL message envelopes
while keeping the newer typed-envelope filtering in place. Also keep
heartbeat ACK pruning on the short-circuit path even when bootstrap
checkpoint refresh fails, so afterTurn does not continue into compaction
once the heartbeat turn was removed.

Update summarize diagnostics tests to assert against the injected runtime
logger instead of console spies, matching the current implementation.

Regeneration-Prompt: |
  The current PR branch had three concrete follow-up fixes: stale summarize
  tests, a bootstrap replay regression, and a heartbeat prune regression.
  Preserve the newer runtime behavior where summarization diagnostics go
  through deps.log rather than console, so tests should inspect the mocked
  logger instead of spying on global console methods. Preserve the newer
  append-only bootstrap hardening that ignores typed non-message sidecars,
  but restore compatibility with older JSONL transcript entries that wrap
  messages as {"message": ...} without a type field. In afterTurn, keep the
  intended behavior that heartbeat ACK turns are pruned before compaction and
  stop the lifecycle there once pruning succeeded; a checkpoint refresh failure
  should be logged as a warning, not cause compaction to continue.
Repair the direct-credentials retry test after the rebase onto origin/main
so the merged expectation keeps skipModelAuth in the call options rather
than nesting it under runtimeConfig.

Regeneration-Prompt: |
  After rebasing the PR branch onto origin/main, the only remaining problem
  was a syntax error in test/summarize.test.ts inside the direct-credentials
  retry assertion. Keep the merged upstream expectation that the second
  completion call carries skipModelAuth: true, but place that field at the
  top level of the expected call options instead of under runtimeConfig so
  the test compiles and matches the actual call shape.
@jalehman jalehman force-pushed the fix/prerelease-cleanup branch from 9a059fe to 8f3a61f Compare April 8, 2026 21:56
Add a patch changeset for the current PR so release notes cover the
runtime reliability fixes and compatibility work already on this branch.

Regeneration-Prompt: |
  The current PR branch needed a changeset before merge. Add a patch
  changeset for @martian-engineering/lossless-claw that summarizes the
  user-visible runtime fixes on the branch at a high level: cache-aware
  compaction reliability, heartbeat-turn pruning, bootstrap compatibility
  for legacy JSONL message envelopes, and logging/docs alignment. Keep the
  entry concise and release-note oriented.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment