Codex review: needs real behavior proof before merge. Reviewed June 9, 2026, 6:38 AM ET / 10:38 UTC.

Summary
This PR adds source-wide memory embedding batching behind a provider runtime opt-in, enables it for OpenAI-compatible memory embeddings, adapts OpenAI batch polling/upload splitting, updates memory sync/index tests, and documents the new provider contract.

PR surface: Source +658, Tests +658, Docs +10. Total +1326 across 13 files.

Reproducibility: Do we have a high-confidence way to reproduce the issue? Source inspection and reporter logs show current main batches provider work at file boundaries, but I did not run the CLI path in this read-only review.

Review metrics: 2 noteworthy metrics.

• Plugin runtime API surface: 1 optional flag added (sourceWideBatchEmbed). External memory providers can treat runtime fields as compatibility contracts once the PR ships.
• Provider batch limits encoded: 50,000 requests, 190 MiB JSONL cap, plus 128-session staging flush. These limits determine when OpenClaw creates additional provider jobs and need to be visible before merge.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦐 gold shrimp
Patch quality: 🦐 gold shrimp
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• Refresh real behavior proof on head 66d362a56dfb1b40585f927e95971f31e4989c89, including a large memory+session run that shows the current staging behavior.
• Update the provider-contract docs or implementation so the 128-session staging limit is visible and intentional.
• Have a maintainer confirm the sourceWideBatchEmbed API and OpenAI-compatible upload-cap behavior before landing.

Proof guidance:

• [P1] Needs stronger real behavior proof before merge: The PR has useful CLI logs and reporter validation, but the strongest proof predates the latest source-wide session staging and OpenAI upload-sizing commits, so current-head real behavior proof is still needed before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

• [P1] The latest head still needs refreshed real-behavior proof because source-wide session staging and OpenAI-compatible upload sizing changed after the strongest CLI/log proof and reporter validation.
• [P1] The new sourceWideBatchEmbed runtime flag becomes a plugin-facing provider contract once merged, so docs and tests need to describe the actual grouping and ordering guarantees precisely.
• [P1] Large memory+session runs now have a hidden 128-session staging flush, which is probably a reasonable memory cap but currently conflicts with the one-call-until-host-limits wording and older proof examples.
• [P1] OpenAI-compatible providers with smaller upload caps may still first see a large default JSONL upload before adaptive splitting handles the rejection, so maintainers should confirm that behavior is acceptable without a user-tunable cap.

Maintainer options:

1. Align contract and proof before merge (recommended)
   Update the provider-contract docs and current-head proof so they show the 128-session staging behavior, OpenAI upload splitting, and the expected large memory+session CLI output from the reviewed head.
2. Accept staged batching as a maintainer choice
   Maintainers can explicitly accept the 128-session flush as the source-wide host limit, but the PR body and docs should still say that this is intentional before merge.
3. Pause if the API direction is unsettled
   If the runtime flag or upload-cap behavior is not the desired long-term plugin contract, pause this branch and settle the provider API shape before more implementation churn.

Next step before merge

• [P1] The remaining blocker is maintainer review of the plugin API/session staging contract plus contributor-supplied current-head proof, not a safe automated repair lane.

Security
Cleared: The diff changes guarded remote batch HTTP usage but keeps uploads under withRemoteHttpResponse with SSRF policy/fetch plumbing, and it does not add dependencies, workflows, secrets, or package scripts.

Review findings

• [P2] Align source-wide docs with staged session flushes — docs/plugins/sdk-overview.md:118-124

@hartmark This is the focused respin for #88009. Could you try this branch against your OpenAI-compatible batch provider when you have a chance?

The expected verbose signal is that dirty memory files and sessions are prepared together and submitted as one source-wide provider batch until the file/request caps require a split, for example:

[memory] embeddings: source-wide batch prepare files=1328 sources=memory=833,sessions=495 maxFiles=2048 maxRequests=50000
[memory] embeddings: source-wide batch submit group=1 source=memory+sessions files=1328 chunks=1328 sources=memory=833,sessions=495 reason=end

This version also keeps oversized-file chunks in the same provider batch, reports OpenAI request_counts progress, uses exponential status-poll backoff, and treats transient status-poll failures such as 429/5xx/network errors as retryable until the configured batch timeout.

I also ran the reporter path gateway:watch:raw with isolated local gateway state; it reached http server listening and gateway ready without the prior launch-failure markers.

Cool, l'll take a look as soon as possible.

Thanks for implementing this annoying blocker.

Still no full reindex.

memory status says batch is disabled, that's strange

Memory Search (main)
Provider: openai (requested: openai)
Model: mistral/mistral-embed
Sources: memory, sessions
Indexed: 1731/1731 files · 6706 chunks
Dirty: no
Store: ~/.openclaw/memory/main.sqlite
Workspace: ~/.openclaw/workspace
Dreaming: off
By source:
  memory · 1395/1395 files · 5241 chunks
  sessions · 336/336 files · 1465 chunks
Vector store: unknown
Vector dims: 1024
FTS: ready
Embedding cache: enabled (7354 entries)
Cache cap: 50000
Batch: disabled (failures 0/2)

pnpm openclaw memory --help
...
  openclaw memory index --force
    Force a full reindex.

Running full reindex.

$ pnpm openclaw memory index --force
....
◒  Indexing memory sources (batch)... 0/1731 · elapsed 0:04.

The output hangs on this output for several minutes and then prints out:

Memory index updated (main).

Then when running memory status yields this response:

Memory Search (main)
Provider: openai (requested: openai)
Model: mistral/mistral-embed
Sources: memory, sessions
Indexed: 1731/1731 files · 6706 chunks
Dirty: yes
Store: ~/.openclaw/memory/main.sqlite
Workspace: ~/.openclaw/workspace
Dreaming: off
Index identity: index metadata is missing
Vector search: paused until memory is rebuilt
Fix: Run: openclaw memory status --index --agent main
By source:
  memory · 1395/1395 files · 5241 chunks
  sessions · 336/336 files · 1465 chunks
Vector store: unknown
FTS: ready
Embedding cache: enabled (7354 entries)
Cache cap: 50000
Batch: disabled (failures 0/2)

Note the message that the index metadata is missing and indexing is paused until rebuilt

@clawsweeper re-review

Current head dd4a3ff558e1dbd27ae73fcb19375262095f20c8 now has current-head safe memory index --force proof in the PR body.

The new proof covers the reporter's latest failure mode directly: two consecutive forced reindexes on the same memory source both report Memory index updated (main), and the follow-up memory status shows Dirty: no without index metadata is missing or paused until memory is rebuilt.

The Real behavior proof workflow is passing again: https://github.com/openclaw/openclaw/actions/runs/27048357559/job/79838839784

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Complete
• Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/27048423634
• Updated: 2026-06-06T01:23:47.725Z

@mushuiyu886 I can do the --force reindex and openclaw memory search foo works now.

Only thing now is that --force should do a complete rerun of the embeddings, or is it just that the index is rebuilt using the previously saved embeddings?

@hartmark Yes — --force here means “force a full safe reindex/resync of the memory index,” not “discard all saved embeddings and regenerate every embedding from the provider.”

During the safe reindex path, the existing embedding cache is carried forward, so unchanged chunks with the same provider/model/cache key and content hash can reuse previously saved embeddings. Changed, missing, or cache-mismatched chunks are embedded again.

So your current result is expected: --force reruns indexing over the sources, but it is intentionally cache-preserving. A mode that wipes cached embeddings and recomputes everything would be a separate stronger option/flag.

@hartmark Yes — --force here means “force a full safe reindex/resync of the memory index,” not “discard all saved embeddings and regenerate every embedding from the provider.”

During the safe reindex path, the existing embedding cache is carried forward, so unchanged chunks with the same provider/model/cache key and content hash can reuse previously saved embeddings. Changed, missing, or cache-mismatched chunks are embedded again.

So your current result is expected: --force reruns indexing over the sources, but it is intentionally cache-preserving. A mode that wipes cached embeddings and recomputes everything would be a separate stronger option/flag.

Cool, let's hope we get this merged soon

@clawsweeper re-review

@jalehman When will this PR be merged? I noticed that many new commits have been added. Is there anyone else who needs to be involved in the merge process?

Merged via squash.

• Prepared head SHA: 66d362a56dfb1b40585f927e95971f31e4989c89
• Merge commit: a36e05050a9de2515675fd89b94857f0abfb9dc0

Thanks @mushuiyu886!