fix(memory-core): keep rotated cron transcripts out of dreaming corpus + add operator filters#72913
fix(memory-core): keep rotated cron transcripts out of dreaming corpus + add operator filters#72913zqchris wants to merge 6 commits intoopenclaw:mainfrom
Conversation
Greptile SummaryThis PR fixes a regression (#72611) where isolated cron transcripts leaked into the dreaming session corpus after rotation by augmenting Confidence Score: 4/5Safe to merge after correcting the One P1 finding: the
|
| }, | ||
| "excludeSourcePathRegex": { | ||
| "type": "array", | ||
| "description": "Skip transcripts whose normalized source path (e.g. `main/sessions/<uuid>.jsonl`) matches any of these regular expressions. Invalid patterns are ignored at runtime with a warning. Use for ad-hoc one-off exclusions where neither cron-id nor agent-id captures the target.", |
There was a problem hiding this comment.
Wrong path format in
excludeSourcePathRegex schema example
The description shows main/sessions/<uuid>.jsonl but sessionPathForFile(absPath) (called in collectSessionIngestionBatches) actually returns "sessions/" + path.basename(absPath) — no agent prefix, just "sessions/<uuid>.jsonl". Any operator who copies the documented example and writes "^main/sessions/.*" will get a silently-failing filter.
Additionally, for rotated artifacts the basename retains the rotation suffix (e.g. "sessions/<uuid>.jsonl.deleted.2026-04-25T06-33-10.801Z"), so a strict \.jsonl$-anchored pattern would also miss them. Both facts belong in the description.
| "description": "Skip transcripts whose normalized source path (e.g. `main/sessions/<uuid>.jsonl`) matches any of these regular expressions. Invalid patterns are ignored at runtime with a warning. Use for ad-hoc one-off exclusions where neither cron-id nor agent-id captures the target.", | |
| "description": "Skip transcripts whose normalized source path (e.g. `sessions/<uuid>.jsonl` for a live transcript or `sessions/<uuid>.jsonl.deleted.<ts>` for a rotated artifact) matches any of these regular expressions. Invalid patterns are ignored at runtime with a warning. Use for ad-hoc one-off exclusions where neither cron-id nor agent-id captures the target.", |
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/memory-core/openclaw.plugin.json
Line: 219
Comment:
**Wrong path format in `excludeSourcePathRegex` schema example**
The description shows `main/sessions/<uuid>.jsonl` but `sessionPathForFile(absPath)` (called in `collectSessionIngestionBatches`) actually returns `"sessions/" + path.basename(absPath)` — no agent prefix, just `"sessions/<uuid>.jsonl"`. Any operator who copies the documented example and writes `"^main/sessions/.*"` will get a silently-failing filter.
Additionally, for rotated artifacts the basename retains the rotation suffix (e.g. `"sessions/<uuid>.jsonl.deleted.2026-04-25T06-33-10.801Z"`), so a strict `\.jsonl$`-anchored pattern would also miss them. Both facts belong in the description.
```suggestion
"description": "Skip transcripts whose normalized source path (e.g. `sessions/<uuid>.jsonl` for a live transcript or `sessions/<uuid>.jsonl.deleted.<ts>` for a rotated artifact) matches any of these regular expressions. Invalid patterns are ignored at runtime with a warning. Use for ad-hoc one-off exclusions where neither cron-id nor agent-id captures the target.",
```
How can I resolve this? If you propose a fix, please make it concise.35176b2 to
e821864
Compare
35d6cd3 to
676df8f
Compare
|
Codex review: needs changes before merge. Summary Reproducibility: yes. A sessions store with an explicit non-cron entry using sessionId shared-id and a cron entry whose :run: id is also shared-id makes lookupSessionKeyForTranscriptPath resolve the explicit session while isCronRunTranscriptPath still returns true from cronRunSessionIds. Next step before merge Security Review findings
Review detailsBest possible solution: Keep the PR open, make cron transcript classification honor explicit session ownership when runIds collide, then rerun the focused memory-host/memory-core tests and changed gate before merge. Do we have a high-confidence way to reproduce the issue? Yes. A sessions store with an explicit non-cron entry using sessionId shared-id and a cron entry whose :run: id is also shared-id makes lookupSessionKeyForTranscriptPath resolve the explicit session while isCronRunTranscriptPath still returns true from cronRunSessionIds. Is this the best way to solve the issue? No, not as currently implemented. The session-id-aware helper direction is maintainable, but cron classification should follow the resolved owning session key or avoid adding colliding runIds to the cron set. Full review comments:
Overall correctness: patch is incorrect Acceptance criteria:
What I checked:
Likely related people:
Remaining risk / open question:
Codex review notes: model gpt-5.5, reasoning high; reviewed against 95cee64ca6e8. |
5e61679 to
90e6c11
Compare
67a7aa9 to
177756f
Compare
d9307e5 to
b885a67
Compare
…g corpus cron runs sometimes leave a second transcript whose basename equals the runId embedded in the sessionKey, distinct from `entry.sessionId`. With classification only indexing `entry.sessionId`, that mirror file looks like an unowned orphan: `lookupSessionKeyForTranscriptPath` returns null, `isCronRunTranscriptPath` returns false, and the file's content slips into the dreaming session corpus despite the operator's `agent:<id>:cron:` prefix exclusion. Add `extractCronRunIdFromSessionKey` in `src/sessions/session-key-utils.ts` and let `loadSessionTranscriptClassificationForSessionsDir` register the extracted runId in both `cronRunSessionIds` and (when not already taken) `sessionIdToSessionKey`. This makes the existing sessionId-fallback in `extractSessionIdFromTranscriptFileName` recognize mirror files — including their `.deleted.<ts>` / `.reset.<ts>` rotated variants — as cron-owned, so dreaming auto-skips them via `generatedByCronRun` even without operator-supplied `excludeSessionKeyPrefixes`. Tests: `session-key-utils.test.ts` covers the extractor's accept/reject boundary and asserts agreement with `isCronRunSessionKey`. `session-files.test.ts` adds two regression cases: - a mirror transcript named after the runId (live + rotated) resolves back to the cron sessionKey and is reported as cron-owned; - a registered explicit-session entry whose sessionId collides with some cron's runId is not overwritten by the runId fallback. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Closing to stay under the active-PR quota — will resubmit if there's reviewer interest. Patch lives on |
…penclaw#73241, openclaw#73400 Second hardening pass after the first round triggered new findings on Aisle's re-scan. PR openclaw#73406 (auto-reply voice silent) was meanwhile landed by the maintainer in upstream commit 28bf71d — that one will drop out at the next rebase via patch-id match, no source change here. PR openclaw#73241 (BB http hardening) — 1 HIGH + 2 MEDIUM: - monitor-processing.sanitizeForLog: redact common secret-bearing patterns (`?password=`, `?token=`, `?api_key=`, `?secret=`, `Authorization: Bearer / Basic …`) before the value reaches the log sink. BlueBubbles uses query-string auth by default, so attachment download failures and similar errors can carry the API password in the captured request URL (CWE-532). - history.fetchBlueBubblesMessageByGuid: validate the derived bareGuid (length ≤128, charset `[A-Za-z0-9._:-]+`, non-empty) before issuing the request. A trailing `/` would otherwise produce an empty bareGuid and turn the call into an unintended `/api/v1/message/` collection query (CWE-20). - monitor-processing reply-context verbose logs: log only `bodyLen=` metadata, not the quoted message text itself. Verbose logs are retained / shipped to aggregators and would otherwise leak private chat content (CWE-532). PR openclaw#73235 (BB routing guards) — 1 MEDIUM + 2 LOW: - monitor-reply-cache.resolveBlueBubblesMessageId: when `requireKnownShortId=true` and `chatContext` lacks any identifier, throw "requires a chat scope" instead of resolving the short id. Short ids are allocated from a single global counter across every account and chat, so an action call without chat scope could silently apply to the wrong conversation (CWE-285). Test updated to expect fail-closed (was previously fail-open with a comment that acknowledged the risk). - monitor-reply-cache.buildCrossChatError: replace raw inputId in the thrown error message with `<short:N-digit>` or `<uuid:prefix…>` shape descriptors. Combined with the earlier chatGuid redaction, cross-chat errors now leak no concrete identifier (CWE-117 / CWE-200). The Low PII finding on monitor-processing verbose logs is resolved automatically by the sanitizeForLog redaction extension added for PR openclaw#73241 openclaw#1. PR openclaw#73400 (silent-reply 🧵) — 1 MEDIUM: - silent-reply-policy.classifySilentReplyConversationType: classify a session key as `internal` only when `parseThreadSessionSuffix` returns a real `🧵<id>` suffix, not on a free-form `🧵` substring match. Caller-supplied session keys can no longer embed the marker mid-string to force the conversation type to `internal` and bypass silent-reply rewrites (CWE-840).
, openclaw#73241 Bundled hardening for the three open PRs: - session-files: use path.posix.join + strip CR/LF/TAB instead of rewriting backslashes to forward slashes. POSIX filenames may legally contain `\`, and the prior translation would synthesize fake path segments that bypass `excludeSourcePathRegex` (CWE-20). NUL is already rejected by Node's fs path layer. - monitor-reply-cache: redact chat identifiers in cross-chat error messages (CWE-200). Phone numbers / email addresses / chat GUIDs must not leak into agent transcripts, tool results, or remote channel deliveries. - monitor-reply-cache: rewrite isCrossChatMismatch so chatIdentifier and chatId comparisons run independently. Earlier version gated fallback comparisons on `!ctxChatGuid`, which let any non-empty ctx.chatGuid suppress the fallback checks when cached entry lacked chatGuid — letting a short id from chat A be reused while acting in chat B (CWE-697). - monitor-processing: drop group reactions where chatGuid / chatIdentifier is whitespace-only, not just empty. A webhook sender supplying " " or "\t" must not satisfy the guard and degrade peerId to the literal "group". - monitor-processing: sanitizeForLog around webhook senderId / messageId / action / attachment guid / err in verbose log lines and attachment download error logs (CWE-117). - monitor-processing: validate replyToId shape (length ≤128, charset alnum + `._:-`) before issuing the API fallback request, so a webhook with a pathological replyToId cannot drive arbitrary outbound load (cheap CWE-400 mitigation). Tests updated: monitor-reply-cache.test.ts cross-chat error assertions now expect `chatGuid=<redacted>` instead of raw values. Also fix unrelated typecheck regression: delivery-dispatch.mirror- bluebubbles.test.ts makeBaseParams missing the `runSessionKey` field introduced in DispatchCronDeliveryParams as part of v2026.4.26.
, openclaw#73241 Bundled hardening for the three open PRs: - session-files: use path.posix.join + strip CR/LF/TAB instead of rewriting backslashes to forward slashes. POSIX filenames may legally contain `\`, and the prior translation would synthesize fake path segments that bypass `excludeSourcePathRegex` (CWE-20). NUL is already rejected by Node's fs path layer. - monitor-reply-cache: redact chat identifiers in cross-chat error messages (CWE-200). Phone numbers / email addresses / chat GUIDs must not leak into agent transcripts, tool results, or remote channel deliveries. - monitor-reply-cache: rewrite isCrossChatMismatch so chatIdentifier and chatId comparisons run independently. Earlier version gated fallback comparisons on `!ctxChatGuid`, which let any non-empty ctx.chatGuid suppress the fallback checks when cached entry lacked chatGuid — letting a short id from chat A be reused while acting in chat B (CWE-697). - monitor-processing: drop group reactions where chatGuid / chatIdentifier is whitespace-only, not just empty. A webhook sender supplying " " or "\t" must not satisfy the guard and degrade peerId to the literal "group". - monitor-processing: sanitizeForLog around webhook senderId / messageId / action / attachment guid / err in verbose log lines and attachment download error logs (CWE-117). - monitor-processing: validate replyToId shape (length ≤128, charset alnum + `._:-`) before issuing the API fallback request, so a webhook with a pathological replyToId cannot drive arbitrary outbound load (cheap CWE-400 mitigation). Tests updated: monitor-reply-cache.test.ts cross-chat error assertions now expect `chatGuid=<redacted>` instead of raw values. Also fix unrelated typecheck regression: delivery-dispatch.mirror- bluebubbles.test.ts makeBaseParams missing the `runSessionKey` field introduced in DispatchCronDeliveryParams as part of v2026.4.26.
…penclaw#73241, openclaw#73400 Second hardening pass after the first round triggered new findings on Aisle's re-scan. PR openclaw#73406 (auto-reply voice silent) was meanwhile landed by the maintainer in upstream commit 28bf71d — that one will drop out at the next rebase via patch-id match, no source change here. PR openclaw#73241 (BB http hardening) — 1 HIGH + 2 MEDIUM: - monitor-processing.sanitizeForLog: redact common secret-bearing patterns (`?password=`, `?token=`, `?api_key=`, `?secret=`, `Authorization: Bearer / Basic …`) before the value reaches the log sink. BlueBubbles uses query-string auth by default, so attachment download failures and similar errors can carry the API password in the captured request URL (CWE-532). - history.fetchBlueBubblesMessageByGuid: validate the derived bareGuid (length ≤128, charset `[A-Za-z0-9._:-]+`, non-empty) before issuing the request. A trailing `/` would otherwise produce an empty bareGuid and turn the call into an unintended `/api/v1/message/` collection query (CWE-20). - monitor-processing reply-context verbose logs: log only `bodyLen=` metadata, not the quoted message text itself. Verbose logs are retained / shipped to aggregators and would otherwise leak private chat content (CWE-532). PR openclaw#73235 (BB routing guards) — 1 MEDIUM + 2 LOW: - monitor-reply-cache.resolveBlueBubblesMessageId: when `requireKnownShortId=true` and `chatContext` lacks any identifier, throw "requires a chat scope" instead of resolving the short id. Short ids are allocated from a single global counter across every account and chat, so an action call without chat scope could silently apply to the wrong conversation (CWE-285). Test updated to expect fail-closed (was previously fail-open with a comment that acknowledged the risk). - monitor-reply-cache.buildCrossChatError: replace raw inputId in the thrown error message with `<short:N-digit>` or `<uuid:prefix…>` shape descriptors. Combined with the earlier chatGuid redaction, cross-chat errors now leak no concrete identifier (CWE-117 / CWE-200). The Low PII finding on monitor-processing verbose logs is resolved automatically by the sanitizeForLog redaction extension added for PR openclaw#73241 openclaw#1. PR openclaw#73400 (silent-reply 🧵) — 1 MEDIUM: - silent-reply-policy.classifySilentReplyConversationType: classify a session key as `internal` only when `parseThreadSessionSuffix` returns a real `🧵<id>` suffix, not on a free-form `🧵` substring match. Caller-supplied session keys can no longer embed the marker mid-string to force the conversation type to `internal` and bypass silent-reply rewrites (CWE-840).
, openclaw#73241 Bundled hardening for the three open PRs: - session-files: use path.posix.join + strip CR/LF/TAB instead of rewriting backslashes to forward slashes. POSIX filenames may legally contain `\`, and the prior translation would synthesize fake path segments that bypass `excludeSourcePathRegex` (CWE-20). NUL is already rejected by Node's fs path layer. - monitor-reply-cache: redact chat identifiers in cross-chat error messages (CWE-200). Phone numbers / email addresses / chat GUIDs must not leak into agent transcripts, tool results, or remote channel deliveries. - monitor-reply-cache: rewrite isCrossChatMismatch so chatIdentifier and chatId comparisons run independently. Earlier version gated fallback comparisons on `!ctxChatGuid`, which let any non-empty ctx.chatGuid suppress the fallback checks when cached entry lacked chatGuid — letting a short id from chat A be reused while acting in chat B (CWE-697). - monitor-processing: drop group reactions where chatGuid / chatIdentifier is whitespace-only, not just empty. A webhook sender supplying " " or "\t" must not satisfy the guard and degrade peerId to the literal "group". - monitor-processing: sanitizeForLog around webhook senderId / messageId / action / attachment guid / err in verbose log lines and attachment download error logs (CWE-117). - monitor-processing: validate replyToId shape (length ≤128, charset alnum + `._:-`) before issuing the API fallback request, so a webhook with a pathological replyToId cannot drive arbitrary outbound load (cheap CWE-400 mitigation). Tests updated: monitor-reply-cache.test.ts cross-chat error assertions now expect `chatGuid=<redacted>` instead of raw values. Also fix unrelated typecheck regression: delivery-dispatch.mirror- bluebubbles.test.ts makeBaseParams missing the `runSessionKey` field introduced in DispatchCronDeliveryParams as part of v2026.4.26.
…penclaw#73241, openclaw#73400 Second hardening pass after the first round triggered new findings on Aisle's re-scan. PR openclaw#73406 (auto-reply voice silent) was meanwhile landed by the maintainer in upstream commit 28bf71d — that one will drop out at the next rebase via patch-id match, no source change here. PR openclaw#73241 (BB http hardening) — 1 HIGH + 2 MEDIUM: - monitor-processing.sanitizeForLog: redact common secret-bearing patterns (`?password=`, `?token=`, `?api_key=`, `?secret=`, `Authorization: Bearer / Basic …`) before the value reaches the log sink. BlueBubbles uses query-string auth by default, so attachment download failures and similar errors can carry the API password in the captured request URL (CWE-532). - history.fetchBlueBubblesMessageByGuid: validate the derived bareGuid (length ≤128, charset `[A-Za-z0-9._:-]+`, non-empty) before issuing the request. A trailing `/` would otherwise produce an empty bareGuid and turn the call into an unintended `/api/v1/message/` collection query (CWE-20). - monitor-processing reply-context verbose logs: log only `bodyLen=` metadata, not the quoted message text itself. Verbose logs are retained / shipped to aggregators and would otherwise leak private chat content (CWE-532). PR openclaw#73235 (BB routing guards) — 1 MEDIUM + 2 LOW: - monitor-reply-cache.resolveBlueBubblesMessageId: when `requireKnownShortId=true` and `chatContext` lacks any identifier, throw "requires a chat scope" instead of resolving the short id. Short ids are allocated from a single global counter across every account and chat, so an action call without chat scope could silently apply to the wrong conversation (CWE-285). Test updated to expect fail-closed (was previously fail-open with a comment that acknowledged the risk). - monitor-reply-cache.buildCrossChatError: replace raw inputId in the thrown error message with `<short:N-digit>` or `<uuid:prefix…>` shape descriptors. Combined with the earlier chatGuid redaction, cross-chat errors now leak no concrete identifier (CWE-117 / CWE-200). The Low PII finding on monitor-processing verbose logs is resolved automatically by the sanitizeForLog redaction extension added for PR openclaw#73241 openclaw#1. PR openclaw#73400 (silent-reply 🧵) — 1 MEDIUM: - silent-reply-policy.classifySilentReplyConversationType: classify a session key as `internal` only when `parseThreadSessionSuffix` returns a real `🧵<id>` suffix, not on a free-form `🧵` substring match. Caller-supplied session keys can no longer embed the marker mid-string to force the conversation type to `internal` and bypass silent-reply rewrites (CWE-840).
Closes #72611.
Problem
Two related leaks let isolated cron transcripts flow into the Dreaming session corpus (
memory/.dreams/session-corpus/<day>.txt) on real deployments, even withsessionTarget: isolatedand existinggeneratedByCronRun/DIRECT_CRON_PROMPT_REfiltering:Path-comparison miss after rotation.
loadSessionTranscriptClassificationForSessionsDirindexes cron / dreaming-narrative transcripts by the livesessionFileabsolute path read fromsessions.json. As soon as that transcript is rotated to*.jsonl.deleted.<ts>(or*.trajectory.jsonl[.deleted.<ts>]) the on-disk path no longer matches the live entry, so the rotated artifact is no longer attributed to the cron session.dreaming-phases.collectSessionIngestionBatchesthen reads it as a normal transcript and ingests its[cron:<id> ...]content lines. Real-world recurrence in Dreaming needs configurable session/cron exclusions; isolated cron transcripts still enter session corpus #72611 with paths likemain/sessions/<uuid>.jsonl.deleted.2026-04-25T06-33-10.801Z.No operator-facing exclusion knob.
dreaming.*only exposes phase-level settings. There is no way to say "skip every cron run for the main agent" or "skip session keys starting withagent:ops:" without forking memory-core. Cron runs whose key is the broadercron:<id>shape (without:run:<runId>) are not even matched byisCronRunSessionKeyso the built-in classifier silently lets them through.Fix
Two complementary changes in one commit:
1. Session-id-aware classification (rotated artifacts)
SessionTranscriptClassificationnow also exposes session-id sets and reverse-lookup maps:New helpers in
session-files.ts(also re-exported fromopenclaw/plugin-sdk/memory-core-host-engine-qmd):extractSessionIdFromTranscriptFileName(fileName)— handles<id>.jsonl,<id>.jsonl.deleted.<ts>,<id>.jsonl.reset.<ts>,<id>.trajectory.jsonl[.deleted.<ts> | .reset.<ts>]. Returnsnullfor non-transcript file shapes.isCronRunTranscriptPath(classification, absPath)/isDreamingNarrativeTranscriptPath(...)— try direct path lookup first, fall back toextractSessionIdFromTranscriptFileName(...)plus the new sessionId set.lookupSessionKeyForTranscriptPath(classification, absPath)— resolves the owning session key for a (possibly rotated) transcript; needed by the operator filter below.dreaming-phases.collectSessionIngestionBatchesis wired to use the new helpers in place ofSet.has(normalizedPath).2. Operator-facing dreaming session filters
New
dreaming.sessionFilterconfig block inextensions/memory-core/openclaw.plugin.json:{ "plugins": { "entries": { "memory-core": { "config": { "dreaming": { "sessionFilter": { "excludeCronJobIds": ["job-1", "job-2"], "excludeSessionKeyPrefixes": ["agent:main:cron:", "agent:ops:"], "excludeAgentIds": ["batch-runner"], "excludeSourcePathRegex": ["^ops/sessions/.*\\.jsonl$"] } } } } } } }resolveSessionIngestionExcludePredicate(cfg, logger)builds a single predicate per ingestion sweep:() => falsewhen no filter is configured (zero overhead in the common case).excludeSourcePathRegexonce; invalid patterns are logged as warnings and skipped, never throw.cron:<id>without:run:).excludeSessionKeyPrefixesis the most flexible knob since it covers bothcron:<id>:run:<runId>andcron:<id>shapes via a singleagent:<id>:cron:prefix.Tests
src/memory-host-sdk/host/session-files.test.ts(+10 tests, 34 total):extractSessionIdFromTranscriptFileName— primary jsonl,.jsonl.deleted.<ts>,.jsonl.reset.<ts>,.trajectory.jsonl,.trajectory.jsonl.deleted.<ts>, non-transcript / null shapes.isCronRunTranscriptPath/isDreamingNarrativeTranscriptPath— live path classification, rotated.deleted.<ts>recovery, rotated.trajectory.jsonl.deleted.<ts>recovery, live.trajectory.jsonlrecovery, unrelated transcripts not matched.lookupSessionKeyForTranscriptPath— live path, rotated path, unknown path.extensions/memory-core/src/dreaming-phases.test.ts(+2 tests, 32 total):.jsonl.deleted.<ts>) via session id (Dreaming needs configurable session/cron exclusions; isolated cron transcripts still enter session corpus #72611)" — full ingestion harness, assertssession-corpus/2026-04-05.txtis not created.dreaming.sessionFilter.excludeSessionKeyPrefixesfor operator-driven exclusion" — uses acron:<id>(non-run) key the built-in classifier wouldn't catch; only the operator filter drops it.Out of scope
isCronRunSessionKeyregex — kept strict for back-compat. Operators who want to dropcron:<id>(non-run) shapes use the newexcludeSessionKeyPrefixesknob.session-file-repair.ts) — only the in-memory ingestion path.openclaw doctor --fixextension) could prune historical leaks.Verified
pnpm tsgo:core✓pnpm tsgo:extensions✓pnpm test src/memory-host-sdk/host/session-files.test.ts→ 34/34 passpnpm test extensions/memory-core/src/dreaming-phases.test.ts→ 32/32 passpnpm check:changed→ all gates pass (lint / typecheck / 0 import cycles / policy guards)