feat(memory-core): add dreaming promotion with weighted recall thresholds#60569
Conversation
Greptile SummaryThis PR adds a short-term memory dreaming/promotion system to The overall architecture is solid — best-effort tracking, atomic writes, and clear threshold defaults. A few issues worth addressing before merge:
Confidence Score: 3/5Safe to merge with minor fixes; the dreaming flow is best-effort and isolated, but a few correctness issues in logging and temp-file uniqueness should be addressed. The feature is well-scoped and opt-in with sensible defaults. No security or data-loss risks in the main flow. However, the optimistic removed-counter logic produces misleading logs, the compareOptionalStrings no-op could hide future null-handling bugs, and the same-millisecond temp-file collision is a real (if rare) data-loss path for a best-effort store. extensions/memory-core/src/dreaming.ts (counter logic, no-op coalescing, limit=0 handling) and extensions/memory-core/src/short-term-promotion.ts (temp-file uniqueness). Prompt To Fix All With AIThis is a comment left during a code review.
Path: extensions/memory-core/src/dreaming.ts
Line: 178-180
Comment:
**`compareOptionalStrings` no-op coalescing**
The `?? undefined` on each operand is completely redundant — the type is `string | undefined` (not nullable), so `undefined ?? undefined` is still `undefined`. The function is always equivalent to a plain strict-equality check (`a === b`). While this doesn't cause a functional bug today, it could hide a real error if the intent was to normalise `null` to `undefined` (e.g. values coming from an external API that returns `null`). Consider removing the no-op coalescing or widening the type to `string | null | undefined` if null inputs are genuinely possible:
```suggestion
function compareOptionalStrings(a: string | undefined, b: string | undefined): boolean {
return a === b;
}
```
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: extensions/memory-core/src/dreaming.ts
Line: 318-323
Comment:
**Optimistic removed-count can mislead log messages**
Both the "disabled" cleanup loop (lines 318–323) and the duplicate-pruning loop (lines 339–344) count a removal as successful when `result.removed !== false`. Because `CronServiceLike.remove` returns `Promise<{ removed?: boolean }>` — the field is _optional_ — a service that returns `{}` (field absent) produces `undefined !== false === true`, so the counter increments even without a confirmed removal. The resulting `logger.info` message then reports a higher-than-real removal count.
Use `=== true` to require an explicit positive acknowledgment:
```suggestion
const result = await cron.remove(job.id);
if (result.removed === true) {
removed += 1;
}
```
Apply the same fix to the duplicate-pruning loop at lines 339–344.
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: extensions/memory-core/src/dreaming.ts
Line: 101-107
Comment:
**`limit: 0` silently falls back to the default via `normalizePositiveInt`**
`normalizePositiveInt` treats `0` as invalid (`floored > 0` is false) and returns the fallback (`DEFAULT_DREAMING_LIMIT = 10`). This means a user who sets `shortTermPromotion.dreaming.limit: 0` in config silently gets 10 instead.
The CLI path (`--limit 0`) does correctly produce 0 candidates (via `slice(0, 0)` in `rankShortTermPromotionCandidates`), so there is a behavioural inconsistency between config and CLI for `limit=0`.
If `limit=0` should be a supported "disable" value, use `normalizeNonNegativeInt` instead. If it should be rejected, add a note to the docs so users aren't surprised.
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: extensions/memory-core/src/short-term-promotion.ts
Line: 304-310
Comment:
**Temp-file name collision is possible within the same process**
```ts
const tmpPath = `${storePath}.${process.pid}.${Date.now()}.tmp`;
```
Two concurrent calls (e.g. overlapping `recordShortTermRecalls` invocations from two parallel searches in the same process) that land in the same millisecond will generate identical `tmpPath` values. The second `writeFile` call would silently overwrite the first's temp file before either `rename` runs, discarding one write.
Since this is a best-effort store you could widen the uniqueness window cheaply with a random suffix:
```suggestion
const tmpPath = `${storePath}.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}.tmp`;
```
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "docs(memory): mark dreaming as experimen..." | Re-trigger Greptile |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 58f9f0d613
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Stoked for this! |
|
Addressed the Greptile + Codex review points in follow-up commit
Added/updated tests for:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2993b27730
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7bd3c6d42a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2a88540f80
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
e3510bc to
fd50591
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd50591153
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| await recordShortTermRecalls({ | ||
| workspaceDir, | ||
| query, | ||
| results, | ||
| }).catch(() => { |
There was a problem hiding this comment.
Do not await best-effort recall persistence in CLI search
runMemorySearch still awaits recordShortTermRecalls, so CLI memory search blocks on lock acquisition/disk I/O even though the comment says this path is best-effort. Under concurrent promote/dreaming activity, withShortTermLock can wait up to its timeout before rejecting, which adds avoidable latency to normal search output and can make the command feel hung.
Useful? React with 👍 / 👎.
| if (ageMs > SHORT_TERM_LOCK_STALE_MS) { | ||
| await fs.unlink(lockPath).catch(() => undefined); | ||
| continue; |
There was a problem hiding this comment.
Avoid stealing live short-term lock after fixed stale window
The lock is treated as stale solely by file mtime and is forcibly deleted after 60s, but the holder never refreshes mtime while running the critical section. If a promotion/write operation runs longer than that (large MEMORY.md, slow filesystem, paused process), another process can delete the live lock and enter concurrently, reintroducing the lost-update/clobber races this lock is meant to prevent.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a739a75725
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const nextConfig = updateDreamingModeInConfig(currentConfig, requestedMode); | ||
| await api.runtime.config.writeConfigFile(nextConfig); | ||
|
|
There was a problem hiding this comment.
Reconcile dreaming cron when /dreaming updates config
The /dreaming command writes plugins.entries.memory-core.config.dreaming.mode and immediately reports success, but it never triggers cron reconciliation itself. In this change, reconciliation is only wired to the gateway:startup hook in dreaming.ts, so existing managed dreaming jobs can continue running with old settings until a gateway restart/reload cycle completes (and in environments where restart is skipped, the command change may not take effect at all).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 42fb5722a9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| void recordShortTermRecalls({ | ||
| workspaceDir, | ||
| query, | ||
| results, | ||
| }).catch(() => { |
There was a problem hiding this comment.
Detach CLI recall writes from process lifetime
runMemorySearch fire-and-forgets recordShortTermRecalls, but that promise still executes inside the same one-shot CLI process; when the short-term lock is contended, withShortTermLock can keep retrying until timeout (up to 10s), and those pending timers/I/O keep Node’s event loop alive. In practice openclaw memory search can still feel hung before returning to the shell, despite recall persistence being marked best-effort.
Useful? React with 👍 / 👎.
42fb572 to
3f78160
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3f78160231
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| `${BUNDLED_PLUGIN_ROOT_DIR}/**`, | ||
| "src/browser/**", | ||
| "src/line/**", | ||
| "src/acp/**", | ||
| "src/agents/**", |
There was a problem hiding this comment.
Restore ACP exclusion from default unit test selection
Because unitTestIncludePatterns still matches src/**/*.test.ts, dropping "src/acp/**" from unitTestAdditionalExcludePatterns makes ACP tests eligible for the generic vitest.unit.config.ts lane. Those tests already have a dedicated vitest.acp.config.ts path, so this change broadens/duplicates the default pnpm test gate and can materially increase runtime and memory pressure on routine checks.
Useful? React with 👍 / 👎.
2f2f9c1 to
0c3a313
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 40e206f68d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| shortTermCount: 0, | ||
| longTermCount: 0, | ||
| promotedCount: 0, | ||
| dreamingOf: null, | ||
| nextCycle: null, |
There was a problem hiding this comment.
Wire dreams view to live promotion metrics
The new Dreams tab always renders hardcoded placeholders (shortTermCount, longTermCount, promotedCount all 0, and nextCycle as null), so it can never reflect actual dreaming/promotion activity. In practice, users will see "0 promoted" and no upcoming cycle even after recalls are promoted, which makes the page inaccurate and misleading rather than a status view.
Useful? React with 👍 / 👎.
…olds (openclaw#60569) * memory-core: add dreaming promotion flow with weighted thresholds * docs(memory): mark dreaming as experimental * memory-core: address dreaming promotion review feedback * memory-core: harden short-term promotion concurrency * acpx: make abort-process test timer-independent * memory-core: simplify dreaming config with mode presets * memory-core: add /dreaming command and tighten recall tracking * ui: add Dreams tab with sleeping lobster animation Adds a new Dreams tab to the gateway UI under the Agent group. The tab is gated behind the memory-core dreaming config — it only appears in the sidebar when dreaming.mode is not 'off'. Features: - Sleeping vector lobster with breathing animation - Floating Z's, twinkling starfield, moon glow - Rotating dream phrase bubble (17 whimsical phrases) - Memory stats bar (short-term, long-term, promoted) - Active/idle visual states - 14 unit tests * plugins: fix --json stdout pollution from hook runner log The hook runner initialization message was using log.info() which writes to stdout via console.log, breaking JSON.parse() in the Docker smoke test for 'openclaw plugins list --json'. Downgrade to log.debug() so it only appears when debugging is enabled. * ui: keep Dreams tab visible when dreaming is off * tests: fix contracts and stabilize extension shards * memory-core: harden dreaming recall persistence and locking * fix: stabilize dreaming PR gates (openclaw#60569) (thanks @vignesh07) * test: fix rebase drift in telegram and plugin guards
…olds (openclaw#60569) * memory-core: add dreaming promotion flow with weighted thresholds * docs(memory): mark dreaming as experimental * memory-core: address dreaming promotion review feedback * memory-core: harden short-term promotion concurrency * acpx: make abort-process test timer-independent * memory-core: simplify dreaming config with mode presets * memory-core: add /dreaming command and tighten recall tracking * ui: add Dreams tab with sleeping lobster animation Adds a new Dreams tab to the gateway UI under the Agent group. The tab is gated behind the memory-core dreaming config — it only appears in the sidebar when dreaming.mode is not 'off'. Features: - Sleeping vector lobster with breathing animation - Floating Z's, twinkling starfield, moon glow - Rotating dream phrase bubble (17 whimsical phrases) - Memory stats bar (short-term, long-term, promoted) - Active/idle visual states - 14 unit tests * plugins: fix --json stdout pollution from hook runner log The hook runner initialization message was using log.info() which writes to stdout via console.log, breaking JSON.parse() in the Docker smoke test for 'openclaw plugins list --json'. Downgrade to log.debug() so it only appears when debugging is enabled. * ui: keep Dreams tab visible when dreaming is off * tests: fix contracts and stabilize extension shards * memory-core: harden dreaming recall persistence and locking * fix: stabilize dreaming PR gates (openclaw#60569) (thanks @vignesh07) * test: fix rebase drift in telegram and plugin guards
…olds (openclaw#60569) * memory-core: add dreaming promotion flow with weighted thresholds * docs(memory): mark dreaming as experimental * memory-core: address dreaming promotion review feedback * memory-core: harden short-term promotion concurrency * acpx: make abort-process test timer-independent * memory-core: simplify dreaming config with mode presets * memory-core: add /dreaming command and tighten recall tracking * ui: add Dreams tab with sleeping lobster animation Adds a new Dreams tab to the gateway UI under the Agent group. The tab is gated behind the memory-core dreaming config — it only appears in the sidebar when dreaming.mode is not 'off'. Features: - Sleeping vector lobster with breathing animation - Floating Z's, twinkling starfield, moon glow - Rotating dream phrase bubble (17 whimsical phrases) - Memory stats bar (short-term, long-term, promoted) - Active/idle visual states - 14 unit tests * plugins: fix --json stdout pollution from hook runner log The hook runner initialization message was using log.info() which writes to stdout via console.log, breaking JSON.parse() in the Docker smoke test for 'openclaw plugins list --json'. Downgrade to log.debug() so it only appears when debugging is enabled. * ui: keep Dreams tab visible when dreaming is off * tests: fix contracts and stabilize extension shards * memory-core: harden dreaming recall persistence and locking * fix: stabilize dreaming PR gates (openclaw#60569) (thanks @vignesh07) * test: fix rebase drift in telegram and plugin guards
…olds (openclaw#60569) * memory-core: add dreaming promotion flow with weighted thresholds * docs(memory): mark dreaming as experimental * memory-core: address dreaming promotion review feedback * memory-core: harden short-term promotion concurrency * acpx: make abort-process test timer-independent * memory-core: simplify dreaming config with mode presets * memory-core: add /dreaming command and tighten recall tracking * ui: add Dreams tab with sleeping lobster animation Adds a new Dreams tab to the gateway UI under the Agent group. The tab is gated behind the memory-core dreaming config — it only appears in the sidebar when dreaming.mode is not 'off'. Features: - Sleeping vector lobster with breathing animation - Floating Z's, twinkling starfield, moon glow - Rotating dream phrase bubble (17 whimsical phrases) - Memory stats bar (short-term, long-term, promoted) - Active/idle visual states - 14 unit tests * plugins: fix --json stdout pollution from hook runner log The hook runner initialization message was using log.info() which writes to stdout via console.log, breaking JSON.parse() in the Docker smoke test for 'openclaw plugins list --json'. Downgrade to log.debug() so it only appears when debugging is enabled. * ui: keep Dreams tab visible when dreaming is off * tests: fix contracts and stabilize extension shards * memory-core: harden dreaming recall persistence and locking * fix: stabilize dreaming PR gates (openclaw#60569) (thanks @vignesh07) * test: fix rebase drift in telegram and plugin guards
…olds (openclaw#60569) * memory-core: add dreaming promotion flow with weighted thresholds * docs(memory): mark dreaming as experimental * memory-core: address dreaming promotion review feedback * memory-core: harden short-term promotion concurrency * acpx: make abort-process test timer-independent * memory-core: simplify dreaming config with mode presets * memory-core: add /dreaming command and tighten recall tracking * ui: add Dreams tab with sleeping lobster animation Adds a new Dreams tab to the gateway UI under the Agent group. The tab is gated behind the memory-core dreaming config — it only appears in the sidebar when dreaming.mode is not 'off'. Features: - Sleeping vector lobster with breathing animation - Floating Z's, twinkling starfield, moon glow - Rotating dream phrase bubble (17 whimsical phrases) - Memory stats bar (short-term, long-term, promoted) - Active/idle visual states - 14 unit tests * plugins: fix --json stdout pollution from hook runner log The hook runner initialization message was using log.info() which writes to stdout via console.log, breaking JSON.parse() in the Docker smoke test for 'openclaw plugins list --json'. Downgrade to log.debug() so it only appears when debugging is enabled. * ui: keep Dreams tab visible when dreaming is off * tests: fix contracts and stabilize extension shards * memory-core: harden dreaming recall persistence and locking * fix: stabilize dreaming PR gates (openclaw#60569) (thanks @vignesh07) * test: fix rebase drift in telegram and plugin guards
Summary
memory_searchtool andopenclaw memory searchCLI pathsopenclaw memory promoteranking/apply flow backed bymemory/.dreams/short-term-recall.jsonminScore=0.75,minRecallCount=3,minUniqueQueries=2Details
memory/YYYY-MM-DD.mdand basename variant)memory promote) while dreaming is opt-in via configTests
pnpm checkpnpm test -- extensions/memory-core/src/short-term-promotion.test.ts extensions/memory-core/src/dreaming.test.ts extensions/memory-core/src/tools.citations.test.tspnpm test -- extensions/memory-core/src/cli.test.tspnpm test -- extensions/memory-core/src/tools.test.ts