Skip to content

fix(codex): release raw assistant app-server completions [AI-assisted]#82403

Merged
steipete merged 4 commits into
openclaw:mainfrom
IWhatsskill:fix/codex-raw-assistant-completion-82343
May 16, 2026
Merged

fix(codex): release raw assistant app-server completions [AI-assisted]#82403
steipete merged 4 commits into
openclaw:mainfrom
IWhatsskill:fix/codex-raw-assistant-completion-82343

Conversation

@IWhatsskill

@IWhatsskill IWhatsskill commented May 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Problem: Codex app-server turns can emit a completed raw assistant rawResponseItem/completed with final text but never send turn/completed, leaving OpenClaw waiting for the terminal idle timeout.
  • Why it matters: channel delivery stays blocked even though the model output is already available, matching the stalled Discord/Telegram delivery shape in Agent sessions using Codex app-server backend timeout - model completes but response never delivered to channels #82343.
  • What changed: pre-tool raw assistant message completions now arm the existing completed-assistant idle release path, so OpenClaw best-effort interrupts the native Codex turn and releases the session lane with the captured assistant text.
  • What did NOT change (scope boundary): raw tool-output watchdog behavior, active request/item guards, abort-marker handling, channel delivery, and SecretRef delivery resolution are unchanged.

Change Type

  • Bug fix
  • Docs

Scope

  • Gateway / orchestration
  • Integrations

Linked Issue/PR

Real behavior proof

  • Behavior or issue addressed: a pre-tool completed raw assistant response item without turn/completed releases the Codex app-server run, while post-tool raw assistant progress does not release the turn early.
  • Real environment tested: local Windows OpenClaw source checkout, local codex CLI logged in with ChatGPT, real Codex app-server using gpt-5.5, and the real runCodexAppServerAttempt path. Workspace and CODEX_HOME are redacted below.
  • Exact steps or command run after this patch: node --import tsx C:\tmp\openclaw-real-run-attempt-proof.mjs
  • Evidence after fix: the probe used a redacting local proxy around real app-server notifications. It dropped agentMessage notifications to isolate the raw fallback, dropped turn/completed only for the final-raw scenario, and delayed turn/completed by 1000ms after post-tool raw assistant progress.
{
  "ok": true,
  "model": "gpt-5.5",
  "workspace": "<redacted-temp-workspace>",
  "codexHome": "<redacted-codex-home>",
  "finalRawWithoutTurnCompleted": {
    "passed": true,
    "proof": {
      "sawRawAssistant": true,
      "droppedTurnCompleted": 1,
      "interruptRequests": 1
    },
    "result": {
      "aborted": false,
      "timedOut": false,
      "promptError": null,
      "assistantTexts": ["REAL_OPENCLAW_RAW_FINAL_PROOF"]
    }
  },
  "postToolRawAssistantProgress": {
    "passed": true,
    "proof": {
      "sawToolRequest": true,
      "sawPostToolRawAssistant": true,
      "delayedAfterPostToolRawAssistantMs": 1000,
      "interruptRequests": 0
    },
    "result": {
      "aborted": false,
      "timedOut": false,
      "promptError": null,
      "assistantTexts": ["REAL_OPENCLAW_TOOL_DONE_PROOF"]
    }
  }
}
  • Observed result after fix: the final raw assistant path releases with captured assistant text after the terminal event is withheld, and the post-tool raw assistant path waits for turn/completed without issuing turn/interrupt.
  • What was not tested: live Discord/Telegram credentials or real channel delivery; this proof exercises the local Codex app-server/OpenClaw run-attempt boundary only.

Root Cause

  • Root cause: event-projector already captured assistant text from raw assistant rawResponseItem/completed notifications, but run-attempt only released completion on turn/completed, abort markers, or completed agentMessage items.
  • Missing detection / guardrail: the completed-assistant idle release did not recognize pre-tool raw assistant message completions as equivalent final assistant output, while post-tool raw assistant progress still needed to stay behind the dynamic-tool handoff guard.
  • Contributing context: adjacent watchdog fixes covered raw tool-output silence and completed agentMessage silence, but left the raw assistant final-output path waiting for the terminal watchdog.

Regression Test Plan

  • Coverage level that should have caught this:
    • Seam / integration test
  • Target test or file: extensions/codex/src/app-server/run-attempt.test.ts
  • Scenario the test should lock in: a pre-tool raw assistant rawResponseItem/completed with final text and no turn/completed releases the run with the captured assistant text; post-tool raw assistant progress does not release before turn/completed.
  • Why this is the smallest reliable guardrail: the bug is in the app-server attempt completion/watchdog boundary, not in channel delivery or the event projector.
  • Existing test that already covers this: extensions/codex/src/app-server/event-projector.test.ts already proves raw assistant text extraction; this PR adds the missing run-attempt release coverage.

User-visible / Behavior Changes

Codex app-server turns that have already produced pre-tool final raw assistant text can deliver instead of waiting for the terminal idle timeout when turn/completed never arrives.

Diagram

Before:
raw assistant completed -> text captured -> wait for turn/completed -> terminal idle timeout

After:
pre-tool raw assistant completed -> text captured -> completed-assistant release -> channel delivery can proceed
post-tool raw assistant progress -> wait for turn/completed -> channel delivery can proceed

Security Impact

  • New permissions/capabilities? No
  • Secrets/tokens handling changed? No
  • New/changed network calls? No
  • Command/tool execution surface changed? No
  • Data access scope changed? No

Repro + Verification

Environment

  • OS: Windows local development checkout
  • Runtime/container: local Node with tsx
  • Model/provider: real local Codex app-server, gpt-5.5
  • Integration/channel: real Codex app-server embedded-run path through runCodexAppServerAttempt
  • Relevant config (redacted): workspace path and CODEX_HOME

Steps

  1. Start a real local Codex app-server through the OpenClaw app-server attempt path.
  2. In the final-raw scenario, observe current-turn rawResponseItem/completed for assistant output and drop turn/completed in the local redacting proxy.
  3. In the post-tool scenario, force a real message tool call, observe post-tool raw assistant text, and delay turn/completed for 1000ms.

Expected

  • OpenClaw releases the pre-tool final raw assistant attempt with captured assistant text and does not mark the turn timed out.
  • OpenClaw does not release post-tool raw assistant progress through the raw assistant fallback; it waits for turn/completed.

Actual

  • Before this patch, OpenClaw waited for turn/completed until the terminal idle timeout in the final-raw scenario.
  • ClawSweeper also found that the first patch could let post-tool raw assistant progress trigger the completed-assistant release. This revision blocks that path while dynamic tool calls have occurred in the turn.

Evidence

  • Failing test/log before + passing after
  • Real local app-server behavior proof after the fix

Human Verification

  • Verified scenarios:
    • pre-tool raw assistant completion without turn/completed releases with assistant text
    • raw assistant progress after a dynamic tool response still waits for turn/completed
    • real local Codex app-server path preserves both behaviors through runCodexAppServerAttempt
    • raw tool-output watchdog behavior remains covered
    • active current-turn item guard still blocks release
    • interrupted turn marker behavior remains covered
  • Edge cases checked:
    • event projector still extracts raw assistant response items
    • formatting, lint, and whitespace checks pass
  • What you did not verify:
    • live channel delivery with real Discord/Telegram credentials

Review Conversations

  • ClawSweeper's P2 blocker is addressed in code, tests, and this PR body.
  • I left the bot review unresolved pending ClawSweeper re-review.

Compatibility / Migration

  • Backward compatible? Yes
  • Config/env changes? No
  • Migration needed? No

Risks and Mitigations

  • Risk: treating an interim raw assistant status message as final too early.
    • Mitigation: only arm the raw assistant completion release before any OpenClaw dynamic-tool handoff in the turn, reuse the completed-assistant idle release instead of immediate completion, and keep active app-server request/current-turn item guards in place. New coverage confirms raw assistant progress after a dynamic tool response still waits for turn/completed.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation extensions: codex size: XS triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. labels May 16, 2026
@clawsweeper

clawsweeper Bot commented May 16, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

Summary
The PR changes Codex app-server completion handling so pre-tool raw assistant completions can release a quiet turn, updates raw assistant phase projection, adds regression coverage, and documents the timeout/release behavior.

Reproducibility: yes. source-reproducible: current main captures raw assistant text but the focused raw-only test still expects a timeout when turn/completed never arrives. The PR body also supplies after-fix live output from a real local Codex app-server run.

Real behavior proof
Sufficient (live_output): The PR body includes redacted after-fix live output from a real local Codex app-server run through runCodexAppServerAttempt, covering final raw release and post-tool non-release.

Next step before merge
No repair-lane defect was found; the next action is maintainer CI/merge handling because the PR is open and required checks are not yet green.

Security
Cleared: No concrete security or supply-chain concern was found; the diff changes Codex runtime completion logic, tests, and docs without new permissions, dependencies, scripts, or secret handling.

Review details

Best possible solution:

Land the guarded raw assistant completion fallback once required CI and merge gates are green, then close #82343 if the merged behavior resolves the delivery deadlock.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible: current main captures raw assistant text but the focused raw-only test still expects a timeout when turn/completed never arrives. The PR body also supplies after-fix live output from a real local Codex app-server run.

Is this the best way to solve the issue?

Yes. Reusing the existing assistant-output idle release with tool-handoff and commentary guards is the narrow maintainable fix for the observed Codex app-server completion gap.

Acceptance criteria:

  • node scripts/run-vitest.mjs extensions/codex/src/app-server/run-attempt.test.ts
  • node scripts/run-vitest.mjs extensions/codex/src/app-server/event-projector.test.ts
  • node scripts/crabbox-wrapper.mjs run ... --shell -- "pnpm check:changed"

What I checked:

Likely related people:

  • mbelinky: Commit d728ff8 added the quiet completed app-server turn release path that this PR builds on. (role: introduced adjacent behavior; confidence: high; commits: d728ff818333; files: extensions/codex/src/app-server/run-attempt.ts, extensions/codex/src/app-server/run-attempt.test.ts, docs/plugins/codex-harness.md)
  • joshavant: Recent current-main commits changed the same Codex app-server watchdog, raw tool-output, and native tool policy surfaces around this completion boundary. (role: recent adjacent contributor; confidence: high; commits: 44a3301e5067, e57b137aef41; files: extensions/codex/src/app-server/run-attempt.ts, extensions/codex/src/app-server/run-attempt.test.ts, extensions/codex/src/app-server/event-projector.ts)
  • steipete: Recent app-server timeout/watchdog commits and the final PR-branch commentary guard touch the same release and projection path. (role: recent adjacent contributor and committer; confidence: high; commits: 44a3301e5067, 5e057f40981b; files: extensions/codex/src/app-server/run-attempt.ts, extensions/codex/src/app-server/event-projector.ts, extensions/codex/src/app-server/run-attempt.test.ts)

Remaining risk / open question:

  • Required CI is not green yet; the observed lint failure is outside the PR's changed files, but merge should still wait for the required checks to settle.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 863069e2c66a.

@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. labels May 16, 2026
@IWhatsskill IWhatsskill force-pushed the fix/codex-raw-assistant-completion-82343 branch from 82ccade to ee3c5bb Compare May 16, 2026 03:40
@IWhatsskill IWhatsskill marked this pull request as ready for review May 16, 2026 03:42
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026
@Kaspre

Kaspre commented May 16, 2026

Copy link
Copy Markdown
Contributor

Real-world hit confirmation. Diagnosing recurring agent-process hangs in production and the shape lines up exactly.

Setup: OC 2026.5.12 stable, agent fleet with primary openai/gpt-5.5 via codex. Several agent invocations end with the OS process hung indefinitely, well after the model output is logically complete.

Two reproducing PIDs captured this morning (4h and 24min old):

  • Both openclaw agent --local --agent <id> spawned by cron heartbeat / preflight scripts
  • wchan = do_epoll_wait, worker threads in futex_wait_queue → idle event loop pinned by ref'd handles
  • Identical FD signature: LCM SQLite (lcm.db/-wal/-shm), the OTel exporter HTTP keep-alive socket to localhost:4318, and codex stdio pipe pairs from already-exited codex children

The codex children had exited cleanly, but OC's run-attempt was apparently still awaiting turn/completed — exactly the case this PR addresses. Channel delivery never happened.

Looking forward to the merge — this is likely the dominant cause of the agent-process accumulation we've been tracking, and it complements #71710 (which focuses on the SIGTERM-doesn't-bridge angle).

@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026
@obviyus obviyus self-assigned this May 16, 2026
@obviyus obviyus force-pushed the fix/codex-raw-assistant-completion-82343 branch from f043b5c to 9c96e93 Compare May 16, 2026 10:03
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026
@amknight

amknight commented May 16, 2026

Copy link
Copy Markdown
Member

[P1] Treat all post-request handoffs as raw-assistant release blockers

item/tool/requestUserInput, mcpServer/elicitation/request, and approval requests all re-arm the post-request completion watch, but they do not mark the turn as having crossed a handoff the way the dynamic tool path does. That means a post-request raw assistant rawResponseItem/completed can still arm the completed-assistant idle release and send turn/interrupt early, potentially delivering interim text before the real final answer.

A focused regression for the item/tool/requestUserInput case fails because turn/interrupt is sent after the post-user-input raw assistant message. Please mark all current-turn request handoffs before allowing raw assistant release, and add coverage for user input plus one approval/MCP path.

@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026
@steipete steipete force-pushed the fix/codex-raw-assistant-completion-82343 branch from 3fe9dd6 to 38e06be Compare May 16, 2026 11:08
@steipete steipete merged commit f50c65f into openclaw:main May 16, 2026
107 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Landed via rebase onto current main.

  • Local proof: git diff --check; pnpm lint --threads=8; node scripts/run-vitest.mjs extensions/codex/src/app-server/run-attempt.test.ts -t "releases the session after a raw assistant response item without turn completion|does not release or return commentary raw assistant response items|does not release post-native-tool raw assistant progress after the assistant idle timeout"; earlier full run-attempt.test.ts pass and Codex review clean before landing fixups
  • CI: green on head 38e06be1918cb742dd872dea2e7e2b0d1e6662de
  • Land commit: 38e06be1918cb742dd872dea2e7e2b0d1e6662de
  • Merge commit: f50c65f12454be1d2ea76b1e786b85a65ae85a20

Thanks @IWhatsskill!

galiniliev pushed a commit to galiniliev/openclaw that referenced this pull request May 20, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 24, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 24, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 24, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
galiniliev pushed a commit to galiniliev/openclaw that referenced this pull request May 25, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 26, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 26, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 26, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
qiaokuan1992 pushed a commit to qiaokuan1992/openclaw that referenced this pull request Jun 2, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
SYU8384 pushed a commit to SYU8384/openclaw that referenced this pull request Jun 3, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
openclaw#82403)

* fix(codex): release raw assistant app-server completions

* refactor(codex): simplify raw assistant release guard

* fix(codex): ignore commentary raw assistant completions

* docs: add codex app-server completion changelog

---------

Co-authored-by: JARVIS-Glasses <284122573+JARVIS-Glasses@users.noreply.github.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation extensions: codex proof: supplied External PR includes structured after-fix real behavior proof. size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent sessions using Codex app-server backend timeout - model completes but response never delivered to channels

5 participants