Skip to content

fix(memory-core): dreaming daily ingestion across calendar days#76359

Merged
obviyus merged 1 commit into
openclaw:mainfrom
neeravmakwana:fix/memory-dreaming-daily-ingestion-76225
May 5, 2026
Merged

fix(memory-core): dreaming daily ingestion across calendar days#76359
obviyus merged 1 commit into
openclaw:mainfrom
neeravmakwana:fix/memory-dreaming-daily-ingestion-76225

Conversation

@neeravmakwana

@neeravmakwana neeravmakwana commented May 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Deep dreaming promotion could stay at “Promoted 0” when daily notes were unchanged on disk: daily-ingestion.json only stored mtimeMs/size, so later dreaming days skipped re-reading those files. Short-term entries then failed default deep gates (minRecallCount / minUniqueQueries against combined signal counts) even though the underlying content was still eligible.

Root cause

collectDailyIngestionBatches treated “unchanged since last run” as “never ingest again,” which is correct for same-day duplicate runs but wrong across calendar dreaming days when the file content is stable.

Linked issue

Fixes #76225 (related context: #67061 ingestion-day dedupe, #64068).

Why this is safe

  • Still reads the same daily memory files and applies the same recordShortTermRecalls / dedupe rules; we only decide when to open a file again.
  • Same-day runs still skip after a successful ingest for that calendar bucket (no extra I/O or duplicate batches for identical mtime/size on the same day).
  • State is backward compatible: older checkpoints without lastDreamingDayIngested re-ingest once on upgrade, then persist the new field.

Security / runtime controls

Unchanged: memory file read policy, workspace scoping, dreaming triggers, and LLM-facing prompts. This is checkpoint bookkeeping only.

Testing

  • git diff --check origin/main...HEAD
  • pnpm exec vitest run extensions/memory-core/src/dreaming-phases.test.ts -t "increments dailyCount|checkpoints daily ingestion"
  • pnpm check:changed

Out of scope

  • Session transcript checkpoint semantics (separate state machine).
  • Further tuning of default deep promotion thresholds (config feature requests remain separate).
  • AI-assisted PR (implementation and validation run in an agent environment; human review before merge is expected).

@neeravmakwana neeravmakwana mentioned this pull request May 3, 2026
Closed
@clawsweeper

clawsweeper Bot commented May 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

Summary
The PR adds per-file lastDreamingDayIngested checkpointing for memory-core daily ingestion, strengthens the cross-day regression test, and adds a changelog entry.

Reproducibility: yes. from source inspection. Current main skips unchanged daily notes using only mtimeMs/size, and the existing test reaches the second-day path only after deleting the daily-ingestion checkpoint.

Real behavior proof
Needs real behavior proof before merge: The PR body lists tests/checks but no after-fix live output, logs, screenshot, recording, or linked artifact showing the changed behavior in a real setup.

Next step before merge
Contributor action is needed to add after-fix real behavior proof; ClawSweeper automation cannot supply evidence from the contributor's real setup.

Security
Cleared: The diff only changes memory-core checkpoint bookkeeping, a focused regression test, and changelog text; it adds no dependency, workflow, permission, secret, or code-execution surface.

Review details

Best possible solution:

Land the per-dreaming-day checkpoint fix after contributor-provided real behavior proof is added and the memory-core changed checks are green.

Do we have a high-confidence way to reproduce the issue?

Yes, from source inspection. Current main skips unchanged daily notes using only mtimeMs/size, and the existing test reaches the second-day path only after deleting the daily-ingestion checkpoint.

Is this the best way to solve the issue?

Yes for the functional code direction. Tracking the last dreaming calendar day per daily note preserves same-day dedupe while allowing cross-day promotion signals; merge should wait for real behavior proof.

What I checked:

  • Current main skips unchanged daily notes: collectDailyIngestionBatches records only mtimeMs and size; when those match, it continues without re-reading the daily note, so a stable file cannot add later calendar-day daily signals. (extensions/memory-core/src/dreaming-phases.ts:1117, a3870686947b)
  • Existing test masked persisted checkpoint behavior: The current test deletes memory/.dreams/daily-ingestion.json before the second-day run, so it does not cover the real unchanged-checkpoint path reported by the linked issue. (extensions/memory-core/src/dreaming-phases.test.ts:2589, a3870686947b)
  • Promotion gates use combined daily signals: recordShortTermRecalls dedupes per query and day bucket, while candidate ranking gates on combined signal count and context diversity, supporting once-per-dreaming-day re-ingestion as the narrow fix. (extensions/memory-core/src/short-term-promotion.ts:970, a3870686947b)
  • PR implements per-day checkpointing: The patch adds lastDreamingDayIngested, normalizes the stored day, passes the current dreaming day into collection, skips only same-day unchanged files, and writes the day after collecting results. (extensions/memory-core/src/dreaming-phases.ts:387, 59d65298a45a)
  • Changelog credit is present after force-push: The PR head includes the memory/dreaming fix entry with Fixes #76225 and Thanks @neeravmakwana, addressing the earlier ClawSweeper changelog-credit finding. (CHANGELOG.md:548, 59d65298a45a)
  • Real behavior proof is still missing: The PR body lists validation commands but does not include terminal output, logs, screenshot, recording, or another after-fix artifact showing the changed behavior in a real setup; the PR is labeled triage: needs-real-behavior-proof. (59d65298a45a)

Likely related people:

  • Bartok9: Authored the merged fix(dreaming): use ingestion date for dayBucket instead of file date #67091 fix that changed dreaming daily recall dedupe to use the ingestion date and added the test this PR now strengthens. (role: introduced adjacent behavior; confidence: high; commits: 4de56b18bada; files: extensions/memory-core/src/dreaming-phases.ts, extensions/memory-core/src/dreaming-phases.test.ts, CHANGELOG.md)
  • vignesh07: Authored the original memory-core Dreaming promotion flow with weighted thresholds, short-term promotion state, docs, and UI surface. (role: introduced feature; confidence: high; commits: 4c1022c73b39; files: extensions/memory-core/src/dreaming-phases.ts, extensions/memory-core/src/short-term-promotion.ts, docs/cli/memory.md)
  • steipete: Recent merged work touched memory-core dreaming sweep behavior and tests shortly before this PR, including primary workspace inclusion in dreaming runs. (role: recent maintainer; confidence: medium; commits: 99f1db33bfe2; files: extensions/memory-core/src/dreaming-phases.ts, extensions/memory-core/src/dreaming-phases.test.ts, docs/concepts/dreaming.md)

Remaining risk / open question:

  • The external-PR real behavior proof gate is unresolved: the body lists tests/checks but no observed after-fix runtime output or artifact.

Codex review notes: model gpt-5.5, reasoning high; reviewed against a3870686947b.

@neeravmakwana neeravmakwana marked this pull request as ready for review May 3, 2026 01:42
@obviyus obviyus force-pushed the fix/memory-dreaming-daily-ingestion-76225 branch from 590a519 to 59d6529 Compare May 5, 2026 15:34
@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 5, 2026
@obviyus obviyus self-assigned this May 5, 2026
@obviyus obviyus added proof: override Maintainer override for the external PR real behavior proof gate. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 5, 2026
@obviyus obviyus force-pushed the fix/memory-dreaming-daily-ingestion-76225 branch from 59d6529 to 271ca6e Compare May 5, 2026 15:44
obviyus
obviyus approved these changes May 5, 2026
View reviewed changes

@obviyus obviyus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the dreaming daily-ingestion regression: current main skipped unchanged daily files by mtime/size only, and this head records the last dreaming day so same-day dedupe stays intact while later-day signals are counted.

Maintainer follow-up: rebased on current main, moved the changelog entry to the active Fixes tail with #76359 credit, trimmed noisy comments, and kept the branch to one signed commit.

Local gate: OPENCLAW_TEST_HEAVY_CHECK_LOCK_HELD=1 OPENCLAW_VITEST_FS_MODULE_CACHE_PATH=.artifacts/vitest-cache-pr76359-land OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test extensions/memory-core/src/dreaming-phases.test.ts -t 'checkpoints daily ingestion|increments dailyCount'.

@obviyus obviyus merged commit 8faf91a into openclaw:main May 5, 2026
80 checks passed
@obviyus

obviyus commented May 5, 2026

Copy link
Copy Markdown
Contributor

Landed on main.

Thanks @neeravmakwana.

@github-actions github-actions Bot mentioned this pull request May 5, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
@obviyus
fix: re-ingest daily memory during dreaming (openclaw#76359) (thanks @…
df3eae6 
…neeravmakwana)
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
@obviyus
fix: re-ingest daily memory during dreaming (openclaw#76359) (thanks @…
5b52876 
…neeravmakwana)
@C1-BA-B1-F3 C1-BA-B1-F3 mentioned this pull request Jun 2, 2026
Open
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
@obviyus
fix: re-ingest daily memory during dreaming (openclaw#76359) (thanks @…
b4fbcaa 
…neeravmakwana)
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
@obviyus
fix: re-ingest daily memory during dreaming (openclaw#76359) (thanks @…
1a11b68 
…neeravmakwana)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@obviyus obviyus obviyus approved these changes

Assignees

@obviyus obviyus

Labels

extensions: memory-core Extension: memory-core proof: override Maintainer override for the external PR real behavior proof gate. size: XS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Dreaming promotion regression — Promoted 0 on v2026.4.29 (reproduction of #64068)

2 participants

@neeravmakwana @obviyus