fix(memory-core): yield event loop during fallback vector search (#81172)#83758
Conversation
|
Codex review: passed. Workflow note: Future ClawSweeper reviews update this same comment in place. How this review workflow works
Summary Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector rows when sqlite-vec is unavailable, and the PR body shows the before/after heartbeat behavior through the actual PR rating What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. PR egg Rarity: 🥚 common. What is this egg doing here?
Real behavior proof Risk before merge
Maintainer options:
Next step before merge Security Review detailsBest possible solution: Land the bounded fallback batching/yield fix after exact-head checks finish, then let it close #81172 without broadening this patch into worker-thread or timeout policy work. Do we have a high-confidence way to reproduce the issue? Yes from source and supplied live output. Current main synchronously scans fallback vector rows when sqlite-vec is unavailable, and the PR body shows the before/after heartbeat behavior through the actual Is this the best way to solve the issue? Yes. Bounded rowid batches with an event-loop yield are a narrow maintainable fix for the fallback starvation path while preserving scoring and avoiding broader worker-thread or timeout policy changes. Label justifications:
What I checked:
Likely related people:
Codex review notes: model gpt-5.5, reasoning high; reviewed against d124c5aa2005. |
|
@clawsweeper automerge |
|
🦞✅ Source: What merged:
Automerge notes:
The automerge loop is complete. Automerge progress:
|
…ge for openclaw#81172 fallback search
8272c9f to
0ede3d7
Compare
|
@clawsweeper automerge |
* fix(gateway): clear CLI bindings on session reset * fix(gateway): preserve spawned sessions in configured lists * fix(channels): clear canonical stale routes * fix(telegram): preserve forum topic origin targets * fix(agents): skip fallback for session coordination errors * fix(agents): persist subagent registry before returning accepted (openclaw#83132) (openclaw#83238) * fix(memory): catch up stale sessions on startup (openclaw#82341) * fix(memory): preserve qmd lexical search for hyphenated queries (openclaw#81423) * fix(anthropic): preserve Claude image capability (openclaw#83756) * fix(agents): exclude tool result details from guard budget (openclaw#75525) * fix(provider): use Together video API endpoint * fix(telegram): preserve implicit default account (openclaw#82794) * fix(gateway): allow trusted-proxy local-direct password fallback (openclaw#82953) * fix(discord): return subagent thread delivery origin * fix: add missing prerequisites for upstream-ported fixes Add SessionWriteLockTimeoutError class and hasSessionWriteLockTimeout helper needed by the ported fix(agents) skip-fallback commit. Remove route property references from session-delivery.ts that don't exist in gemmaclaw's SessionEntry type. Add authorizePasswordAuth helper that was present in upstream but missing from gemmaclaw's auth.ts. * fix: remove route assertions incompatible with gemmaclaw SessionEntry Remove test assertions using .route property that exists in upstream's SessionEntry type but not in gemmaclaw's, restoring typecheck green. * fix(memory-core): yield event loop during fallback vector search (openclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com> * fix(subagents): collect unresolved announce batches (openclaw#83701) Summary: - The PR changes collect-mode follow-up queue routing so unresolved-origin items can batch with a single resolved route and later compatible items can resume batching after a true cross-channel drain. - Reproducibility: yes. at source level: current main treats unkeyed-plus-same-keyed queue items as cross-chan ... failing path is directly visible in `src/utils/queue-helpers.ts` and `src/auto-reply/reply/queue/drain.ts`. Automerge notes: - PR branch already contained follow-up commit before automerge: Merge remote-tracking branch 'origin/main' into maint-83701-20260518 Validation: - ClawSweeper review passed for head e6ad029. - Required merge gates passed before the squash merge. Prepared head SHA: e6ad029 Review: openclaw#83701 (comment) Co-authored-by: Andy Ye <35905412+TurboTheTurtle@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com> * fix(config): accept gateway remote port * fix: restore Array<{}> closing bracket in manager-search.ts Cherry-pick 68b3729 accidentally dropped the '>' from '}>', producing a syntax error. Restore '}>;' as it was in origin/main. * fix: add remotePort to GatewayRemoteConfig and GatewayRemoteConfigSchema * fix(agents): prioritize manual session turns (openclaw#82765) * fix(agents): prioritize manual session turns * docs: update changelog for session priority --------- Co-authored-by: Galin Iliev <Galin.Iliev@microsoft.com> * revert: fix(agents): prioritize manual session turns (openclaw#82765) - upstream deps not in gemmaclaw * fix: resolve undefined variable errors in cherry-picked extension code * fix(tui): preserve draft while chat is busy * fix(tui): add pendingChatRunId to TuiStateAccess for cherry-picked tui commit * fix(memory-wiki): make wiki_lint tool output path-safe (openclaw#83687) * fix(ui): render session-scoped tool events (openclaw#83734) * chore: regenerate base config schema after upstream cherry-picks * fix(agents): add persistSubagentRunsToDiskOrThrow to subagent-registry test mock New export added to subagent-registry-state.ts was missing from the vi.mock definition, causing all tests in the suite to skip and the module to fail to load. * fix(telegram): wire buildTelegramInboundOriginTarget into session context Cherry-pick 675e053 added the helper and the test assertion but did not update bot-message-context.session.ts to use it. OriginatingTo now correctly includes :topic:<id> for forum groups. * fix(memory): correct session path format in startup-catchup test sessionPathForFile returns sessions/<basename> (no agent dir), but the cherry-picked test used sessions/main/<basename>. The clean-file test always failed because the path mismatch made every file look unindexed. * fix(together): update video generation test URL from v1 to v2 The source uses TOGETHER_VIDEO_BASE_URL = https://api.together.xyz/v2 but the cherry-picked test still asserted the old v1 URL. --------- Co-authored-by: nitinjwadhawan <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com> Co-authored-by: Andy Ye <35905412+TurboTheTurtle@users.noreply.github.com> Co-authored-by: Peter Steinberger <steipete@gmail.com> Co-authored-by: Galin Iliev <iliev@galcho.com> Co-authored-by: Galin Iliev <Galin.Iliev@microsoft.com> Co-authored-by: Harry Xie <harryhsieh963@yahoo.com>
…172) (#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (#81… Validation: - ClawSweeper review passed for head 0ede3d716805e7d2ced8df37c6666af510dc9e19. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d716805e7d2ced8df37c6666af510dc9e19 Review: openclaw/openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…172) (#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (#81… Validation: - ClawSweeper review passed for head 0ede3d716805e7d2ced8df37c6666af510dc9e19. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d716805e7d2ced8df37c6666af510dc9e19 Review: openclaw/openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…nclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (openclaw#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…nclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (openclaw#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…nclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (openclaw#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…nclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (openclaw#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…nclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (openclaw#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…nclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (openclaw#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…nclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (openclaw#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…nclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (openclaw#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…nclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (openclaw#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…nclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (openclaw#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
…nclaw#81172) (openclaw#83758) Summary: - The branch changes memory-core fallback vector search to scan chunks in 256-row rowid batches with `setImmediate` yields, updates regression tests, and adds a changelog entry. - Reproducibility: yes. from source and supplied live output. Current main synchronously scans fallback vector ... and the PR body shows the before/after heartbeat behavior through the actual `searchVector` fallback path. Automerge notes: - PR branch already contained follow-up commit before automerge: test(memory-core): add boundary, parity, and concurrent-insert covera… - PR branch already contained follow-up commit before automerge: fix(memory-core): yield event loop during fallback vector search (openclaw#81… Validation: - ClawSweeper review passed for head 0ede3d7. - Required merge gates passed before the squash merge. Prepared head SHA: 0ede3d7 Review: openclaw#83758 (comment) Co-authored-by: NW <nitinwadhawan66@gmail.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com> Approved-by: takhoffman Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
Summary
memory_search's fallback vector path scans every row of thechunkstable with a synchronous JS-side cosine-similarity loop. With no usable sqlite-vec index (extension missing, dimension mismatch with the active embedding model, or build in progress) and a large memory corpus, that loop pins the Node.js main thread for tens of seconds. The Discord/Telegram gateway misses heartbeats, the gateway connection closes, and the agent appears hung — exactly the symptom reported in memory-core: memory_search blocks event loop for 60+ seconds — Discord gateway closes, agents hang #81172 (event loop delay reaching 62,746 ms, gateway timeout, agent unresponsive).searchChunksByEmbeddingnow scans rows in boundedLIMIT 256batches using arowid > lastRowidcursor, andawaitssetImmediate(...)between batches so timers, gateway sockets, and other event-loop work can interleave. Also keeps the prepared statement rooted in a local so node:sqlite cannot finalize it mid-scan (a latent race that yielding would otherwise surface deterministically).Change Type
Scope
Linked Issue/PR
setImmediate-based yielding during session indexing, merged 5/10), [Bug]: Active-memory embedded sub-agent run blocks event loop, starving Telegram polling — agent goes permanently unresponsive #65517, [Bug]: Embedded run timeout leaves zombie handle blocking heartbeat delivery #52231 (closed — same event-loop-starvation family)Root Cause
extensions/memory-core/src/memory/manager-search.ts:searchChunksByEmbeddingperformed a single anonymousdb.prepare(...).iterate(...)chain followed byfor (const row of rows) { cosineSimilarity(...); parseEmbedding(...) }. The loop was synchronous: every row was parsed and scored in one un-yielded JS tick. For a corpus with thousands of chunks × 768/1536-dim embeddings, the per-row work (JSON.parseof a ~6 KB embedding string + 2N float multiplies) compounds to multi-second windows where no other event-loop task can run. Discord/Telegram gateway WebSocket heartbeats miss their deadline → connection closes → agent appears hung.db.prepareto throw on.all()and used.iterate(), which made the test green even though the production path could pin the main thread for arbitrary durations.manager-sync-ops.ts:createSessionSyncYield/SESSION_SYNC_YIELD_EVERY. This PR ports that pattern to the search fallback path.Regression Test Plan
extensions/memory-core/src/memory/manager-search.test.ts):batches fallback chunk scoring without materializing all candidates— existing test rewritten to assert the boundedLIMIT 256cadence (batchSizes === [256, 256, 1]for 513 rows) and theSELECT rowid, id, path, ...shape.yields to the event loop during large fallback scans (issue #81172)— inserts 1024 chunks (>4× the yield batch), runssearchVectorwithensureVectorReady: async () => false, and asserts a backgroundsetIntervalheartbeat fires at least once during the search (would be 0 with the pre-fix sync loop).returns an empty result set when no chunks match the provider model— N=0 model-match branch.handles a single matching row (below the yield batch size)— N=1 fast path; single batch ends before any yield.handles an exact batch-size boundary (FALLBACK_VECTOR_BATCH_SIZE rows)— N=256 boundary; one full batch then a harmless empty-batch step, no row dropped or double-counted, top-K scores strictly decreasing.preserves top-K ordering vs. a naive reference cosine implementation— N=200 chunks with deterministic synthetic vectors; compares our patched fallback against a straight-line JS reference (independent cosine) and asserts the top-K matches by id and order. Guards against algorithmic regressions slipping in via the control-flow refactor.picks up rows inserted during the inter-batch event-loop yield (rowid cursor)— schedules asetImmediate-drivenINSERTto land during the search's first inter-batch yield; asserts the new rows (rowid >lastRowid) appear in batch 2 and dominate the top-K. Proves rowid pagination is concurrency-safe under writes-during-search.User-visible / Behavior Changes
memory_searchon large corpora no longer freeze the gateway connection. Discord DMs continue working under memory load; Telegram polling does not stall.Security Impact
WHERE model = ?filter, same sources.Real Behavior Proof
Behavior or issue addressed: Per #81172,
memory_searchon real production corpora blocks the Node.js event loop for 60+ seconds (reporter measuredeventLoopDelayMaxMs=62,746.8), causing Discord gateway connection drop. Reproducible on Ubuntu 24.04 with Node v22.x on OpenClaw 2026.5.7. This PR scans the fallback vector path in bounded batches that yield between iterations, keeping the event loop responsive at production scale without changing search result correctness.Real environment tested: Node v22.14.0 +
node:sqlite:memory:databases primed withensureMemoryIndexSchemafromopenclaw/plugin-sdk/memory-core-host-engine-storage(same helper the production manager uses). Three corpus sizes — 20k, 50k, and 100k chunks × 1536-dim embeddings (matches OpenAItext-embedding-3-small). Driven by a focused Node harness that imports the actual exportedsearchVectorfrom this PR'smanager-search.ts, withensureVectorReady: async () => falseto force the changed fallback code path. Event-loop responsiveness measured by an independentsetInterval(50)heartbeat counter run in parallel with the search.Exact steps or command run after this patch:
origin/main, applied fix inmanager-search.tsand updatedmanager-search.test.ts.:memory:chunks viaensureMemoryIndexSchema+INSERT, started a 50 mssetIntervalheartbeat, ranawait searchVector({...ensureVectorReady: async()=>false}), then reported wall time + heartbeat count + max gap.EXPLAIN QUERY PLANon the batched select to verify the rowid cursor uses the implicitINTEGER PRIMARY KEYindex.pnpm test extensions/memory-core/src/memory/manager-search.test.tsfor the unit suite.pnpm test extensions/memory-corefor the full extension suite.pnpm check:changedfor lint, typecheck, and import-cycle gates.Evidence after fix:
EXPLAIN QUERY PLANfor the batched select used by the patched code (realnode:sqlite:memory:db with the production schema):The
rowid > lastRowidcursor uses the implicitINTEGER PRIMARY KEYindex. No full table scan, no temporary B-tree, noOFFSETrescan cost. Themodel = ?filter is applied during the indexed scan.Bucketed as an event-loop-delay histogram (gap between consecutive 50 ms heartbeats during the 20k × 1536-dim search):
For reference, the same harness on
origin/main(pre-fix, sync loop) at 20000 chunks × 1536-dim:Wall-time scales linearly with
Nas expected (the actual cosine work is unchanged). Event-loop responsiveness is consistent ~58–65% across all scales. Max gap stays under 1 second even at 100k chunks — well under the Discord/Telegram gateway heartbeat deadlines that the reporter observed missing.The 10 skipped tests are the sqlite-vec native-extension-dependent tests that don't load on this macOS dev host; unrelated to this change. The 8 passing search tests cover (a) batching cadence, (b) event-loop yielding, (c) N=0/N=1/N=256 row-count boundaries, (d) top-K parity vs. an independent reference cosine implementation, and (e) rowid-cursor robustness against rows inserted during the inter-batch yield.
memory_searchtool call chain):searchKeyworduses FTS5 native index, the vec0 fast path usesvec_distance_cosinenatively,embedQueryWithTimeoutalready runs async with explicitAbortControllertimeouts (60 s remote / 5 min local),mergeHybridResults/applyTemporalDecayToHybridResults/applyMMRToHybridResultsall operate on the boundedcandidates = Math.min(200, ...)set. The fallback cosine scan was the only unbounded synchronous JS loop in the chain.Observed result after fix: With identical CPU work, the event loop now gets ~58–65% of its scheduled
setInterval(50)heartbeats and never goes more than ~500 ms without firing across 20k–100k corpora at 1536-dim. At the reporter's implied production scale (extrapolation from their 62-second symptom: ~340k chunks), the projected max gap stays well below the Discord gateway heartbeat deadline, so the connection no longer drops. Search results are unchanged (same algorithm, same top-K, same ordering) — only the scheduling shape differs.What was not tested:
runEmbeddingOperationWithTimeout. End-to-end coverage relies on the existing tool-level integration tests inextensions/memory-core/src/tools.test.tsand on the search-path audit above.Compatibility / Migration
searchChunksByEmbeddingis a module-private function only called from the already-asyncsearchVector. No exported API surface changes.Risks and Mitigations
model = ?rows to concurrent writes (e.g., an indexing pass adding new chunks during a search).rowidascending. New inserts get higher rowids than the cursor, so a concurrent insert produces at-worst one extra row in the next batch, never a missed row or a duplicate. The previous.iterate()approach had the same concurrent-read semantics in WAL mode; pagination does not regress it.LIMIT 256per batch is arbitrary.setImmediate-based yielding (fix(memory): yield during session indexing #76978 in the same file family). This PR matches that pattern. If maintainers prefer the worker-thread approach as a future refactor, this fix is forward-compatible — the cosine loop becomes the obvious unit to push off-thread later.AI-assisted: yes (Cursor). I read the full
memory_searchcall chain (tools.ts,manager.ts,manager-search.ts,hybrid.ts,temporal-decay.ts,manager-embedding-ops.ts), verified the only unbounded JS hot loop is the fallback cosine scan, ported thesetImmediate-yield pattern from the recent #76978 fix (same file family, same bug class), restructured to bounded rowid-paginated batches to avoid holding a sqlite iterator across the yield, and confirmed the query plan uses the implicit rowid index. Built the live repro harness from scratch and removed it before commit per the repo's "no PR-only assets in the repo" rule.