Skip to content

fix(memory-core): redact managed dreaming blocks during promotion rehydration (#80613)#80702

Closed
MilosM348 wants to merge 2 commits into
openclaw:mainfrom
MilosM348:fix/short-term-promotion-strip-dreaming-80613
Closed

fix(memory-core): redact managed dreaming blocks during promotion rehydration (#80613)#80702
MilosM348 wants to merge 2 commits into
openclaw:mainfrom
MilosM348:fix/short-term-promotion-strip-dreaming-80613

Conversation

@MilosM348

@MilosM348 MilosM348 commented May 11, 2026

Copy link
Copy Markdown

Summary

  • Problem: rehydratePromotionCandidate re-reads the raw daily-note source file and feeds the unredacted lines into relocateCandidateRange, whose fuzzy window search will latch onto a window that straddles a human bullet and the adjacent managed <!-- openclaw:dreaming:light:* --> / <!-- openclaw:dreaming:rem:* --> block, leaking ## Light Sleep headings and - Candidate: … / confidence: … / status: staged scaffolding into MEMORY.md.
  • Why it matters: the chunk builder already strips managed blocks before snippet capture (stripManagedDailyDreamingLines in dreaming-phases.ts), so the post-rehydration isContaminatedDreamingSnippet check never sees a bare Candidate: lead and lets straddle snippets through; users see staged dreaming scaffolding promoted into long-term memory.
  • What changed: added redactManagedDreamingLines(lines) in extensions/memory-core/src/short-term-promotion.ts and call it on the raw source split inside rehydratePromotionCandidate before relocation. The helper mirrors dreaming-phases.ts's contract — zeroes start..end markers AND walks upward to also zero the ## Light Sleep / ## REM Sleep heading directly above the start marker — and unterminated managed blocks redact through end of file.
  • What did NOT change (scope boundary): no changes to the chunk builder, the contamination predicate, the relocate algorithm, the apply path, the MEMORY.md writer, or the CJK dedupe path (the second sub-bug from [Bug]: dreaming pipeline leaks raw candidate content into MEMORY.md and CJK dedup is ineffective in tokenizeSnippet #80613 is being addressed separately by fix(memory-core): use CJK-aware tokenizer for dreaming dedupe (#80613) #80620).

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Real behavior proof (required for external PRs)

  • Behavior or issue addressed: managed <!-- openclaw:dreaming:light:* --> / <!-- openclaw:dreaming:rem:* --> blocks no longer leak into MEMORY.md when promotion rehydration straddles a human bullet next to a managed block.
  • Real environment tested: local OpenClaw source checkout on Linux 6.17 with Node 22 / pnpm 11; daily-note fixtures written via the same recordShortTermRecallsrankShortTermPromotionCandidatesapplyShortTermPromotions pipeline used by memory_search recall capture, with workspace state on disk and a real MEMORY.md round-tripped through fs.readFile.
  • Exact steps or command run after this patch: pnpm exec vitest run extensions/memory-core/src/short-term-promotion.test.ts extensions/memory-core/src/dreaming-phases.test.ts
  • Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output): terminal output reported Test Files 2 passed (2) | Tests 88 passed (88) | Duration 19.76s (49 short-term-promotion + 39 dreaming-phases). The new fuzzy-window-straddle integration regression (commit c97b3633) was verified to FAIL against current main without the sanitizer — the produced MEMORY.md was captured verbatim as <!-- openclaw-memory-promotion:memory:memory/2026-04-13.md:20:30 --> - - Plan switches use exRule, not abConfig ## Light Sleep <!-- openclaw:dreaming:light:start --> - Candidate: Plan toggle field summary - confidence: 0.00 - evidence: memory/.dreams/session-corpus/2026-04-13.txt:1-3 - recalls: 0 - status: staged <!-- openclaw:dreaming:light:end --> [score=0.611 recalls=1 avg=0.940 source=memory/2026-04-13.md:2-12] (matches the exact leak shape on issue [Bug]: dreaming pipeline leaks raw candidate content into MEMORY.md and CJK dedup is ineffective in tokenizeSnippet #80613). With the sanitizer in place, MEMORY.md keeps only - Plan switches use exRule, not abConfig.
  • Observed result after fix: promoted MEMORY.md entries derived from daily notes that interleave human content with managed dreaming blocks contain only the human content; no ## Light Sleep heading, no <!-- openclaw:dreaming:* --> markers, no - Candidate: / confidence: / evidence: / recalls: / status: staged scaffolding ever lands.
  • What was not tested: a live multi-day dreaming-promotion run against a real Anthropic/OpenAI provider — the regression is exercised end-to-end through the in-process pipeline against on-disk daily notes and real MEMORY.md writes, not against a streaming model session. Operator confirmation against a real long-running workspace is welcome.
  • Before evidence (optional but encouraged): the leak was already documented on [Bug]: dreaming pipeline leaks raw candidate content into MEMORY.md and CJK dedup is ineffective in tokenizeSnippet #80613 with the reporter's MEMORY.md showing - ## 教训:Plan 实验开关字段 - Plan 配置中实验开关字段是 \... ## Light Sleep <!-- openclaw:dreaming:light:start --> - Candidate: 教训… (human bullet + heading + start marker + dreaming Candidate: line all promoted); ClawSweeper's earlier review on the issue confirmed the source-level path through promotion rehydration.

Root Cause (if applicable)

  • Root cause: rehydratePromotionCandidate (extensions/memory-core/src/short-term-promotion.ts) reads the raw daily-note file and passes the unredacted lines into relocateCandidateRange, which then fuzzy-matches against any window of 1..maxSpan lines. When the candidate's stored startLine/endLine does not match exactly (stale capture, file drift, partial-line snippet), the fuzzy search prefers higher-quality windows and can pick a multi-line window that includes the managed block. The relocated snippet starts with a clean human bullet, so the post-rehydration isContaminatedDreamingSnippet check (which keys on a leading Candidate: after diff-prefix stripping) does not flag it.
  • Missing detection / guardrail: no parallel sanitizer for the rehydration boundary — only the chunk builder's stripManagedDailyDreamingLines was protecting daily ingestion.
  • Contributing context (if known): the leak only surfaces when a daily note interleaves a human bullet directly next to a managed block, which is the dominant shape produced by the dreaming pipeline itself, so any user with the dreaming feature on hits this.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: extensions/memory-core/src/short-term-promotion.test.ts — three new regressions: two unit tests for redactManagedDreamingLines (terminated light + REM blocks; unterminated light block through EOF), one integration test for the full record→rank→apply chain (exact-match path), and one integration test for the fuzzy-window straddle path (commit c97b3633, addresses ClawSweeper P3).
  • Scenario the test should lock in: a daily note where a human bullet sits directly above ## Light Sleep + <!-- openclaw:dreaming:light:start --> + staged dreaming scaffolding + <!-- openclaw:dreaming:light:end -->, captured into a recall whose startLine/endLine no longer point at the human bullet, so rehydration must traverse the fuzzy window search and must NOT relocate to a window that includes the managed block.
  • Why this is the smallest reliable guardrail: it exercises the exact rehydration boundary touched by this fix, against real on-disk fixtures and the real apply path, and is verified to fail without the sanitizer.
  • Existing test that already covers this (if any): None — dreaming-phases.test.ts only covers the chunk-builder strip; the rehydration path was untested for managed-block leaks.
  • If no new test is added, why not: N/A (4 new regressions added).

User-visible / Behavior Changes

MEMORY.md will no longer accumulate ## Light Sleep / ## REM Sleep headings, <!-- openclaw:dreaming:* --> markers, or - Candidate: / confidence: / status: staged scaffolding from short-term promotion. Existing MEMORY.md files are not rewritten — only new promoted entries are sanitized.

Diagram (if applicable)

Before:
daily note (human bullet | managed block) -> rehydratePromotionCandidate(raw lines)
  -> relocateCandidateRange fuzzy match -> straddle window -> MEMORY.md leak

After:
daily note (human bullet | managed block) -> rehydratePromotionCandidate(redactManagedDreamingLines(raw lines))
  -> relocateCandidateRange fuzzy match over sanitized lines -> human-only window -> clean MEMORY.md

Security Impact (required)

  • New permissions/capabilities? (Yes/No) No
  • Secrets/tokens handling changed? (Yes/No) No
  • New/changed network calls? (Yes/No) No
  • Command/tool execution surface changed? (Yes/No) No
  • Data access scope changed? (Yes/No) No
  • If any Yes, explain risk + mitigation: N/A

Repro + Verification

Environment

  • OS: Linux 6.17 (host)
  • Runtime/container: local OpenClaw source checkout, Node 22, pnpm 11
  • Model/provider: N/A (in-process pipeline, no model calls)
  • Integration/channel (if any): N/A (memory-core only)
  • Relevant config (redacted): default short-term promotion thresholds (minScore: 0, minRecallCount: 0, minUniqueQueries: 0 for the regression fixture)

Steps

  1. Write a daily memory note with the shape ## Plan toggle field / human bullet / blank / ## Light Sleep / <!-- openclaw:dreaming:light:start --> / staged scaffolding / <!-- openclaw:dreaming:light:end -->.
  2. Record a short-term recall whose stored snippet is the multi-line straddle (human bullet + heading + dreaming bullets) at a stale startLine/endLine (e.g. 20/30) so relocate must use the fuzzy window search.
  3. Run rankShortTermPromotionCandidates then applyShortTermPromotions, then read back MEMORY.md.

Expected

  • MEMORY.md contains only the human bullet (- Plan switches use exRule, not abConfig); no ## Light Sleep, no <!-- openclaw:dreaming:* -->, no - Candidate: / confidence: / status: staged.

Actual

  • Matches expected on this branch (verified by pnpm exec vitest run extensions/memory-core/src/short-term-promotion.test.ts — all 49 tests pass, including the 4 new regressions).

Evidence

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

  • Verified scenarios: ran pnpm exec vitest run extensions/memory-core/src/short-term-promotion.test.ts extensions/memory-core/src/dreaming-phases.test.ts locally (Test Files 2 passed (2) | Tests 88 passed (88) | Duration 19.76s); reverted the redactManagedDreamingLines call inside rehydratePromotionCandidate and re-ran the new fuzzy-window straddle regression — observed the leak in MEMORY.md matching [Bug]: dreaming pipeline leaks raw candidate content into MEMORY.md and CJK dedup is ineffective in tokenizeSnippet #80613's reporter shape; restored the sanitizer and re-ran — the leak is gone.
  • Edge cases checked: terminated light block; terminated REM block; unterminated light block (redacts through EOF); heading-walk redacts the ## Light Sleep / ## REM Sleep heading directly above the start marker; diary marker (no heading) only redacts start..end markers; exact-match relocation path (existing test fix: add @lid format support and allowFrom wildcard handling #1 at short-term-promotion.test.ts:1249); fuzzy-match straddle path (new test at short-term-promotion.test.ts:1304).
  • What you did not verify: a live multi-day dreaming-promotion run against a real provider session; a live long-running workspace where dreaming has been running for weeks. The change is additive and structural (an extra sanitization pass before relocation), with no behavior change for daily notes that contain no managed blocks.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

ClawSweeper's earlier P3 (exercise the straddling rehydration path) is addressed by commit c97b3633 and acknowledged in a follow-up comment with the verbatim leak captured under no-fix conditions.

Compatibility / Migration

  • Backward compatible? (Yes/No) Yes
  • Config/env changes? (Yes/No) No
  • Migration needed? (Yes/No) No
  • If yes, exact upgrade steps: N/A

Risks and Mitigations

  • Risk: the heading-walk could redact a ## Light Sleep / ## REM Sleep heading that a user authored manually for their own notes (i.e. one that does NOT precede a managed start marker).
    • Mitigation: the heading walk only fires when an exact ## Light Sleep or ## REM Sleep heading sits directly above the matching managed start marker (mirrors findManagedDailyDreamingHeadingIndex in dreaming-phases.ts); a stand-alone user heading without the immediately-following start marker is left untouched.

@openclaw-barnacle openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 11, 2026
@clawsweeper

clawsweeper Bot commented May 11, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close because the same memory-core leak half is now superseded by a merged current-main fix that rejects rehydrated ranges overlapping managed dreaming fences before appending to MEMORY.md. This branch's remaining distinction is preserving an adjacent human bullet by redaction, which is a separate follow-up tradeoff rather than a reason to keep a conflicting PR open.

So I’m closing this here and keeping the remaining discussion on the canonical linked item.

Review details

Best possible solution:

Keep the merged current-main no-leak guard as canonical, continue the CJK dedupe work in #80620, and open a new focused follow-up only if maintainers want redaction that preserves adjacent human bullets instead of skipping fenced ranges.

Do we have a high-confidence way to reproduce the issue?

No current-main reproduction remains: the merged fence-overlap guard skips rehydrated ranges that touch managed dreaming fences before MEMORY.md is written. The historical failure path is still understandable from the old raw rehydration flow and the PR's regression fixture.

Is this the best way to solve the issue?

No for merging this branch as-is: the leak is already handled on current main by the merged guard, while this PR's redaction-preserves-human-content behavior is a distinct refinement. A fresh, non-conflicting follow-up would be the better path if maintainers want that refinement.

Security review:

Security review cleared: The diff only touches memory-core redaction logic, colocated tests, and changelog text; no concrete security or supply-chain regression was found.

What I checked:

Likely related people:

  • solomonneas: Authored the merged PR that added the current-main contamination detector and dream-fence rehydration guard in the same short-term promotion files. (role: introduced superseding guard; confidence: high; commits: 314903417fcc; files: extensions/memory-core/src/short-term-promotion.ts, extensions/memory-core/src/short-term-promotion.test.ts)
  • vignesh07: Introduced the memory-core dreaming promotion flow and short-term-promotion surface later affected by this leak path. (role: introduced promotion behavior; confidence: high; commits: 4c1022c73b39; files: extensions/memory-core/src/short-term-promotion.ts, extensions/memory-core/src/dreaming-phases.ts)
  • YB0y: Recently changed applyShortTermPromotions and MEMORY.md promotion compaction behavior in the same runtime surface. (role: recent area contributor; confidence: medium; commits: 3b8cfc365746; files: extensions/memory-core/src/short-term-promotion.ts, extensions/memory-core/src/short-term-promotion.test.ts)
  • obviyus: Recently changed daily memory re-ingestion during dreaming, adjacent to the managed daily-note stripping path involved here. (role: recent adjacent contributor; confidence: medium; commits: 8faf91a2a8c9; files: extensions/memory-core/src/dreaming-phases.ts, extensions/memory-core/src/dreaming-phases.test.ts)

Codex review notes: model gpt-5.5, reasoning high; reviewed against 83b8289ee274; fix evidence: commit 314903417fcc, main fix timestamp 2026-05-12T21:07:43Z.

@MilosM348 MilosM348 force-pushed the fix/short-term-promotion-strip-dreaming-80613 branch from 0544e0c to 9de646d Compare May 11, 2026 16:35
MilosM348 added a commit to MilosM348/openclaw1 that referenced this pull request May 11, 2026
…0613

Address ClawSweeper P3 on PR openclaw#80702: the existing integration test
recorded the recall snippet at the exact human bullet line, so
relocateCandidateRange always took the exact-match path and never
exercised the fuzzy window search that originally latched onto the
managed-block straddle window in openclaw#80613.

Add a regression that records a clean-leading multi-line straddle
snippet at a stale 20..30 range, forcing rehydration through the fuzzy
window search. Verified the new test FAILS without redactManagedDreaming
Lines (MEMORY.md receives '- - Plan switches use exRule, not abConfig
## Light Sleep <!-- openclaw:dreaming:light:start --> - Candidate: ... -
status: staged <!-- openclaw:dreaming:light:end -->' verbatim) and
PASSES with the sanitizer in place.

All 88 tests across short-term-promotion and dreaming-phases pass.

Co-authored-by: Cursor <cursoragent@cursor.com>
@MilosM348

Copy link
Copy Markdown
Author

Thanks @clawsweeper for the P3 — fair catch.

Pushed c97b3633 adding a second integration regression blocks fuzzy-window straddle leaks when the recorded range no longer matches the daily note (#80613) (extensions/memory-core/src/short-term-promotion.test.ts). It records a clean-leading multi-line straddle snippet (human bullet immediately followed by ## Light Sleep + <!-- openclaw:dreaming:light:start --> + the staged Candidate: / confidence: / evidence: / recalls: / status: staged scaffolding + <!-- openclaw:dreaming:light:end -->) at a stale startLine: 20, endLine: 30 range, so relocateCandidateRange skips its exact-match path and goes through the fuzzy window search that originally latched onto the straddle window.

I verified the test fails without redactManagedDreamingLinesMEMORY.md receives the leak verbatim:

<!-- openclaw-memory-promotion:memory:memory/2026-04-13.md:20:30 -->
- - Plan switches use exRule, not abConfig ## Light Sleep <!-- openclaw:dreaming:light:start --> - Candidate: Plan toggle field summary - confidence: 0.00 - evidence: memory/.dreams/session-corpus/2026-04-13.txt:1-3 - recalls: 0 - status: staged <!-- openclaw:dreaming:light:end --> [score=0.611 recalls=1 avg=0.940 source=memory/2026-04-13.md:2-12]

— matching the exact form reported in #80613. With the sanitizer in place, the fuzzy search collapses to a single human-bullet window and MEMORY.md keeps only - Plan switches use exRule, not abConfig.

Full local sweep: pnpm exec vitest run extensions/memory-core/src/short-term-promotion.test.ts extensions/memory-core/src/dreaming-phases.test.tsTest Files 2 passed (2) | Tests 88 passed (88) (49 short-term + 39 dreaming-phases — added 1 over previous 87).

@clawsweeper re-review please.

@openclaw-barnacle openclaw-barnacle Bot added size: M proof: supplied External PR includes structured after-fix real behavior proof. and removed size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 11, 2026
MilosM348 and others added 2 commits May 11, 2026 13:13
…ydration (openclaw#80613)

Daily memory notes can interleave human content with managed
`<!-- openclaw:dreaming:light:* -->` and `<!-- openclaw:dreaming:rem:* -->`
blocks. The chunk builder strips those regions before snippet capture, but
`rehydratePromotionCandidate` re-reads the raw source file and feeds it to
`relocateCandidateRange`, whose fuzzy window search will happily latch onto
a window that straddles the human bullet and the adjacent dreaming bullets.
That leaks `- Candidate: …` / `confidence: …` / `status: staged` lines into
`MEMORY.md`.

Add `redactManagedDreamingLines` and call it on the source split before
relocation, mirroring the chunk-side `stripManagedDailyDreamingLines`
heading-walk so the `## Light Sleep` / `## REM Sleep` heading is also
zeroed when it sits directly above the start marker. Unterminated managed
blocks are redacted through the end of file rather than left as a partial
window.

Cover with a unit test of the helper (terminated, unterminated, multiple
markers) and an integration test that writes a note with a `## Light Sleep`
dreaming block and asserts the promoted `MEMORY.md` keeps the human bullet
and contains no `Candidate:` / `confidence:` / `status: staged` /
`openclaw:dreaming:light` traces.

Refs openclaw#80620 (CJK dedupe) — that PR fixes the second sub-bug from the
issue; this one only addresses the promotion-leak half.
…0613

Address ClawSweeper P3 on PR openclaw#80702: the existing integration test
recorded the recall snippet at the exact human bullet line, so
relocateCandidateRange always took the exact-match path and never
exercised the fuzzy window search that originally latched onto the
managed-block straddle window in openclaw#80613.

Add a regression that records a clean-leading multi-line straddle
snippet at a stale 20..30 range, forcing rehydration through the fuzzy
window search. Verified the new test FAILS without redactManagedDreaming
Lines (MEMORY.md receives '- - Plan switches use exRule, not abConfig
## Light Sleep <!-- openclaw:dreaming:light:start --> - Candidate: ... -
status: staged <!-- openclaw:dreaming:light:end -->' verbatim) and
PASSES with the sanitizer in place.

All 88 tests across short-term-promotion and dreaming-phases pass.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: memory-core Extension: memory-core proof: supplied External PR includes structured after-fix real behavior proof. size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: dreaming pipeline leaks raw candidate content into MEMORY.md and CJK dedup is ineffective in tokenizeSnippet

1 participant