fix(reply-queue): remove the drained item by reference instead of front index#91450
Conversation
|
Codex review: needs maintainer review before merge. Reviewed June 8, 2026, 11:55 AM ET / 15:55 UTC. Summary PR surface: Source +10, Tests +53. Total +63 across 3 files. Reproducibility: yes. Source inspection shows the current-main race between awaited drain removal and front-splicing overflow, and the PR body provides before/after helper output that loses m6 on current main and preserves it after the patch. Review metrics: none identified. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Next step before merge
Security Review detailsBest possible solution: Land this focused identity-removal fix after normal CI and maintainer review, keeping any broader overflow-accounting cleanup separate. Do we have a high-confidence way to reproduce the issue? Yes. Source inspection shows the current-main race between awaited drain removal and front-splicing overflow, and the PR body provides before/after helper output that loses m6 on current main and preserves it after the patch. Is this the best way to solve the issue? Yes. Removing the processed object reference after the awaited run is the narrowest maintainable fix because it preserves retry-on-throw behavior while avoiding a stale front-index assumption. AGENTS.md: found and applied where relevant. Codex review notes: model gpt-5.5, reasoning high; reviewed against cfeaf6897fd8. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +10, Tests +53. Total +63 across 3 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
…nt index drainNextQueueItem captured items[0], awaited the run, then shift()-ed index 0 assuming it still held the item it ran. Concurrent inbound messages mutate the same shared items array, and at or over cap applyQueueDropPolicy splices items off the front, so a burst arriving while item[0] is in flight can shift a different, still-undelivered survivor into index 0. shift() then deletes that survivor: it is never run and is not counted in the overflow summary, so the agent silently ignores a message it should have answered. Remove the item that actually ran by identity via a new removeQueuedItemsByRef helper, and apply the same reference-based removal to the collect path in drain.ts, which had the same positional splice(0, groupItems.length) assumption after an awaited group run.
974858b to
ede2b5d
Compare
|
Landed via rebase onto main.
Thanks @yetval! |
…26.6.6) (#1040) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/openclaw/openclaw](https://openclaw.ai) ([source](https://github.com/openclaw/openclaw)) | patch | `2026.6.5` → `2026.6.6` | --- ### Release Notes <details> <summary>openclaw/openclaw (ghcr.io/openclaw/openclaw)</summary> ### [`v2026.6.6`](https://github.com/openclaw/openclaw/blob/HEAD/CHANGELOG.md#202666) [Compare Source](openclaw/openclaw@v2026.6.5...v2026.6.6) ##### Highlights - Security boundaries are substantially tighter across transcripts, sandbox binds, host environment inheritance, MCP stdio, Codex HTTP access, native search policy, elevated sender checks, deleted-agent ACP bypasses, loopback tools, Discord moderation, and Teams group actions; exec approvals now fail closed on timeout. ([#​91529](openclaw/openclaw#91529), [#​91618](openclaw/openclaw#91618), [#​91615](openclaw/openclaw#91615), [#​91619](openclaw/openclaw#91619), [#​91741](openclaw/openclaw#91741), [#​91745](openclaw/openclaw#91745), [#​91746](openclaw/openclaw#91746), [#​91748](openclaw/openclaw#91748), [#​91749](openclaw/openclaw#91749), [#​91750](openclaw/openclaw#91750), [#​91751](openclaw/openclaw#91751), [#​91752](openclaw/openclaw#91752), [#​91763](openclaw/openclaw#91763), [#​89938](openclaw/openclaw#89938)) Thanks [@​joshavant](https://github.com/joshavant), [@​pgondhi987](https://github.com/pgondhi987), [@​mmaps](https://github.com/mmaps), [@​eleqtrizit](https://github.com/eleqtrizit), [@​shakkernerd](https://github.com/shakkernerd), and [@​drobison00](https://github.com/drobison00). - Telegram delivery is safer and more coherent: account-scoped topics route to the right agent, streamed text survives tool calls, `/compact` works on generic ingress, callback handling uses concrete APIs, draft chunking is shared, durable dispatch dedupe moved into the SDK, and unauthorized DM text stays out of cache and prompt context. ([#​91189](openclaw/openclaw#91189), [#​88682](openclaw/openclaw#88682), [#​89588](openclaw/openclaw#89588), [#​90212](openclaw/openclaw#90212), [#​91876](openclaw/openclaw#91876), [#​91874](openclaw/openclaw#91874), [#​91904](openclaw/openclaw#91904), [#​91478](openclaw/openclaw#91478), [#​91915](openclaw/openclaw#91915)) Thanks [@​codysai001](https://github.com/codysai001), [@​alexzhu0](https://github.com/alexzhu0), [@​joelnishanth](https://github.com/joelnishanth), [@​snowzlm](https://github.com/snowzlm), [@​obviyus](https://github.com/obviyus), and [@​sallyom](https://github.com/sallyom). - iMessage recovery and delivery now cover always-on inbound restart, durable echo markers, block streaming, idle approval discovery, hardened outbound transport, and actionable inbound startup diagnostics. ([#​91335](openclaw/openclaw#91335), [#​91449](openclaw/openclaw#91449), [#​88969](openclaw/openclaw#88969), [#​88530](openclaw/openclaw#88530), [#​91783](openclaw/openclaw#91783), [#​91785](openclaw/openclaw#91785)) Thanks [@​omarshahine](https://github.com/omarshahine), [@​jmissig](https://github.com/jmissig), and [@​colmbrogan](https://github.com/colmbrogan). - Browser and MCP connectivity gained existing-session CDP support, discovered WebSocket validation, default-profile `cdpUrl` handling, safer browser-output boundaries, Streamable HTTP loopback transport, corrected OAuth/SSE authorization handling, and broader schema compatibility. ([#​91422](openclaw/openclaw#91422), [#​89851](openclaw/openclaw#89851), [#​91736](openclaw/openclaw#91736), [#​91747](openclaw/openclaw#91747), [#​91451](openclaw/openclaw#91451), [#​80143](openclaw/openclaw#80143)) Thanks [@​pgondhi987](https://github.com/pgondhi987), [@​anagnorisis2peripeteia](https://github.com/anagnorisis2peripeteia), [@​lifuyue](https://github.com/lifuyue), [@​eleqtrizit](https://github.com/eleqtrizit), [@​LiuwqGit](https://github.com/LiuwqGit), and [@​HemantSudarshan](https://github.com/HemantSudarshan). - Control UI startup and first-reply latency are lower through cached model metadata, removal of the startup catalog wait, lazy slash-command loading, and first-event tracing with slow-reply diagnostics. ([#​91531](openclaw/openclaw#91531), [#​91538](openclaw/openclaw#91538), [#​91568](openclaw/openclaw#91568), [#​91583](openclaw/openclaw#91583), [#​91598](openclaw/openclaw#91598)) - Provider support expands with OpenRouter OAuth onboarding and Claude Fable 5 adaptive thinking, while Codex sessions keep correct compaction ownership, local models skip guardian review, dynamic tool progress normalizes cleanly, and Gemma 4 reasoning replay is preserved. ([#​91830](openclaw/openclaw#91830), [#​91882](openclaw/openclaw#91882), [#​91590](openclaw/openclaw#91590), [#​88630](openclaw/openclaw#88630), [#​88768](openclaw/openclaw#88768), [#​91696](openclaw/openclaw#91696)) Thanks [@​Patrick-Erichsen](https://github.com/Patrick-Erichsen), [@​joshavant](https://github.com/joshavant), [@​bdjben](https://github.com/bdjben), and [@​Coder-Wangyankun](https://github.com/Coder-Wangyankun). ##### Changes - CLI progress: emit Claude CLI commentary progress events and bridge inter-tool commentary into channel progress without exposing internal protocol scaffolding. ([#​89834](openclaw/openclaw#89834), [#​90883](openclaw/openclaw#90883)) Thanks [@​anagnorisis2peripeteia](https://github.com/anagnorisis2peripeteia). - Observability: allow trusted diagnostics channels to capture tool input/output content, add first-assistant-event traces, and warn on slow initial replies. ([#​91256](openclaw/openclaw#91256), [#​91568](openclaw/openclaw#91568), [#​91583](openclaw/openclaw#91583)) Thanks [@​amknight](https://github.com/amknight). - Plugins/ClawHub: dogfood reusable package publishing, let dry runs skip publish approval, allow declared installed trusted hooks, report managed plugin version drift, and warn instead of failing on retired Skill Workshop configuration. ([#​91574](openclaw/openclaw#91574), [#​91591](openclaw/openclaw#91591), [#​90004](openclaw/openclaw#90004), [#​90927](openclaw/openclaw#90927), [#​90838](openclaw/openclaw#90838)) Thanks [@​Patrick-Erichsen](https://github.com/Patrick-Erichsen), [@​brokemac79](https://github.com/brokemac79), and [@​lonexreb](https://github.com/lonexreb). - Memory/providers: move the local llama.cpp runtime into its provider plugin, batch embeddings across files, persist the agent model catalog cache, and keep QMD JSON search one-shot while filtering stale REM recall previews. ([#​91324](openclaw/openclaw#91324), [#​89138](openclaw/openclaw#89138), [#​90457](openclaw/openclaw#90457), [#​91837](openclaw/openclaw#91837), [#​91851](openclaw/openclaw#91851)) Thanks [@​osolmaz](https://github.com/osolmaz), [@​mushuiyu886](https://github.com/mushuiyu886), [@​ai-hpc](https://github.com/ai-hpc), and [@​TurboTheTurtle](https://github.com/TurboTheTurtle). - Channels/mobile: add the QQBot group mention toggle, improve iPad and iPhone control surfaces, and expose the active connection host in the TUI footer. ([#​91423](openclaw/openclaw#91423), [#​91557](openclaw/openclaw#91557), [#​89909](openclaw/openclaw#89909)) Thanks [@​cxyhhhhh](https://github.com/cxyhhhhh), [@​Solvely-Colin](https://github.com/Solvely-Colin), and [@​baskduf](https://github.com/baskduf). - Performance: prewarm TUI runtime plugins, deduplicate plugin auto-enable fanout, trim dense text-delta snapshots, and reuse prepared startup model metadata. ([#​90782](openclaw/openclaw#90782), [#​89978](openclaw/openclaw#89978), [#​91580](openclaw/openclaw#91580), [#​91531](openclaw/openclaw#91531)) Thanks [@​RomneyDa](https://github.com/RomneyDa) and [@​ai-hpc](https://github.com/ai-hpc). ##### Fixes - Agent/session recovery: drop stale approval follow-ups after session rebind, remove drained reply-queue items by identity, recover stale main and visible replies, preserve Codex context-engine compaction ownership, lower the default compaction timeout to 180 seconds while respecting explicit configuration, and keep provider-failure terminal lifecycle state correct. ([#​85679](openclaw/openclaw#85679), [#​91450](openclaw/openclaw#91450), [#​91566](openclaw/openclaw#91566), [#​91840](openclaw/openclaw#91840), [#​91590](openclaw/openclaw#91590), [#​91361](openclaw/openclaw#91361), [#​91895](openclaw/openclaw#91895)) Thanks [@​openperf](https://github.com/openperf), [@​yetval](https://github.com/yetval), [@​joshavant](https://github.com/joshavant), [@​wangmiao0668000666](https://github.com/wangmiao0668000666), and [@​TurboTheTurtle](https://github.com/TurboTheTurtle). - User-visible content boundaries: suppress Codex/Harmony protocol artifacts, neutralize browser and LanceDB memory media directives, redact transcript images, and preserve native `/compact` replies through source suppression. ([#​89151](openclaw/openclaw#89151), [#​91422](openclaw/openclaw#91422), [#​91425](openclaw/openclaw#91425), [#​91529](openclaw/openclaw#91529), [#​90212](openclaw/openclaw#90212)) Thanks [@​joelnishanth](https://github.com/joelnishanth), [@​pgondhi987](https://github.com/pgondhi987), [@​joshavant](https://github.com/joshavant), and [@​snowzlm](https://github.com/snowzlm). - Channel delivery: keep WhatsApp captured replies attached to the successor controller after restart, retry Feishu rate limits, preserve Mattermost thread replies, canonicalize LINE webhook paths, restore Discord reply hydration and runtime timeout exports, and show OpenAI Realtime WebRTC assistant transcripts. ([#​85823](openclaw/openclaw#85823), [#​89659](openclaw/openclaw#89659), [#​91684](openclaw/openclaw#91684), [#​91649](openclaw/openclaw#91649), [#​90263](openclaw/openclaw#90263), [#​91686](openclaw/openclaw#91686), [#​90426](openclaw/openclaw#90426)) Thanks [@​itsuzef](https://github.com/itsuzef), [@​ladygege](https://github.com/ladygege), [@​jacobtomlinson](https://github.com/jacobtomlinson), [@​fuller-stack-dev](https://github.com/fuller-stack-dev), and [@​shushushv](https://github.com/shushushv). - Cron: cancel active task runs cleanly, preserve terminal timeout/cancel state, and recover no-deliver tool warnings instead of silently losing the outcome. ([#​90666](openclaw/openclaw#90666), [#​90678](openclaw/openclaw#90678)) Thanks [@​ai-hpc](https://github.com/ai-hpc). - Gateway/config/auth: share the approval runtime socket token, replace arrays explicitly in `config.patch`, skip the deleted-agent guard only for valid ACP harness sessions, surface headless LaunchAgent state, verify SQLite auth migration before cleanup, and arm QMD startup maintenance. ([#​87105](openclaw/openclaw#87105), [#​91551](openclaw/openclaw#91551), [#​91219](openclaw/openclaw#91219), [#​91614](openclaw/openclaw#91614), [#​91740](openclaw/openclaw#91740), [#​91978](openclaw/openclaw#91978)) Thanks [@​fuller-stack-dev](https://github.com/fuller-stack-dev) and [@​scotthuang](https://github.com/scotthuang). - Providers/Codex: clarify quota errors, restore the Codex synthetic usage line, canonicalize Codex protocol assets, require API-key auth for realtime voice, normalize ACP model refs, preserve Gemma 4 `reasoning_content`, and avoid guardian review for local models. ([#​91390](openclaw/openclaw#91390), [#​91709](openclaw/openclaw#91709), [#​91507](openclaw/openclaw#91507), [#​91567](openclaw/openclaw#91567), [#​88630](openclaw/openclaw#88630), [#​91696](openclaw/openclaw#91696)) Thanks [@​hxy91819](https://github.com/hxy91819), [@​brokemac79](https://github.com/brokemac79), [@​RomneyDa](https://github.com/RomneyDa), [@​joshavant](https://github.com/joshavant), and [@​Coder-Wangyankun](https://github.com/Coder-Wangyankun). - Updates/builds: recover package Gateway restarts after refresh failure, expose plugin convergence repair, fall back to Corepack in PATH-less pnpm environments, seed the correct Docker store packages, and keep ClawHub dry-run and publish paths reusable. ([#​91581](openclaw/openclaw#91581), [#​91599](openclaw/openclaw#91599), [#​91547](openclaw/openclaw#91547), [#​91591](openclaw/openclaw#91591)) Thanks [@​fuller-stack-dev](https://github.com/fuller-stack-dev), [@​sallyom](https://github.com/sallyom), and [@​Patrick-Erichsen](https://github.com/Patrick-Erichsen). - UI: require explicit user intent before opening chat sessions and drain restored chat queues after session switches. ([#​91480](openclaw/openclaw#91480)) Thanks [@​TurboTheTurtle](https://github.com/TurboTheTurtle). - Android: avoid the `dataSync` foreground-service type for persistent nodes. ([#​80082](openclaw/openclaw#80082)) Thanks [@​davelutztx](https://github.com/davelutztx). - Native hooks: bound relay lifetimes so abandoned native hook connections cannot linger indefinitely. ([#​91550](openclaw/openclaw#91550)) Thanks [@​joshavant](https://github.com/joshavant). </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMDEuMSIsInVwZGF0ZWRJblZlciI6IjQzLjEwMS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL3BhdGNoIl19--> Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/1040
Summary
During a burst of inbound messages to a busy session, the reply queue can silently drop a queued message: it is never answered and is not even counted in the "Dropped N messages due to cap" overflow summary.
The drain helper removes the wrong item.
drainNextQueueItem(src/utils/queue-helpers.ts) capturesnext = items[0],awaits the run (a real agent turn, seconds long), then callsitems.shift()on the assumption that index 0 still holds the item it just ran:But
enqueueFollowupRunmutates the same shareditemsarray for every concurrent inbound message, and at or over capapplyQueueDropPolicydoesitems.splice(0, dropCount), removing from the front. If enough messages arrive whileitems[0]is in flight, the in-flight item is itself spliced off the front (and recorded as dropped), shifting a different, still-undelivered survivor into index 0. When theawaitreturns,items.shift()deletes that survivor: it was never run, and it is not in the overflow summary either.The same positional-front assumption breaks the collect path in
src/auto-reply/reply/queue/drain.ts: it snapshotsconst items = queue.items.slice(), runs each authorization group, then doesqueue.items.splice(0, groupItems.length)after the awaited run, which can splice off survivors that arrived during the run rather than the group it actually processed.The fix removes the items that were actually processed by identity instead of by front position. A small
removeQueuedItemsByRefhelper does anindexOf+spliceper processed item;drainNextQueueItemremoves the one item it ran, and the collect path removes the group it ran.Diff size
Verification
Local (Node 22.19.0), against this checkout:
node scripts/run-vitest.mjs src/utils/queue-helpers.test.ts src/auto-reply/reply/queue/drain.identity-guard.test.ts- pass (11 tests).node scripts/run-vitest.mjs src/auto-reply/reply/reply-flow.test.ts- pass (12 tests).tsgo:core- exit 0.scripts/run-oxlint.mjs --tsconfig config/tsconfig/oxlint.core.jsonon both changed files - clean.drainNextQueueItem,applyQueueDropPolicy,buildQueueSummaryPrompt) and recreates a cap-overflow burst during an in-flight drain fails onorigin/main(a survivor is lost) and passes on this branch. The harness is kept local and is not part of this single-commit fix; its captured before/after output is in Real behavior proof below.Real behavior proof
src/utils/queue-helpers.tsdrainNextQueueItem,applyQueueDropPolicy,buildQueueSummaryPrompt), composed exactly as the drain loop composes them: seed one item, begin draining it, and while that drain is awaiting, push a 7-message burst throughapplyQueueDropPolicyat cap 3 (the same shared array the drain holds), then drain the remainder and read back what was delivered, what stayed queued, what the drop policy dropped, and the emitted overflow summary. No network or provider credentials are involved; the defect is pure in-memory queue bookkeeping.m1, began draining it, and during the awaited run pushedm2..m8throughapplyQueueDropPolicyon the shared array (cap 3, summarize policy), then drained to completion and printedDELIVERED,REMAINING,DROPPED_BY_POLICY, theSUMMARY_PROMPT, andLOST(any id that is in none of delivered, remaining, or dropped). Ran the identical harness once on this branch and once with the two production files reverted toorigin/main.origin/main; after is this branch:m6survived the drop policy but, onorigin/main, appeared in none of delivered, remaining, or the overflow summary (LOST: ["m6"]) - silent loss beyond the cap accounting. On this branch the identical burst deliversm6(DELIVEREDincludesm6,LOSTis empty); the documented overflow accounting form1..m5is unchanged.