Release embedded session write lock before model I/O by amknight · Pull Request #82891 · openclaw/openclaw

amknight · 2026-05-17T04:12:23Z

Summary

Fixes #13744 by narrowing the embedded run's session transcript write-lock lifecycle:

release the coarse embedded attempt lock before model/provider I/O
reacquire/fence short critical sections for transcript persistence, event processing, live-output boundaries, tool-call boundaries, Pi extension hooks, prompt cache persistence, and cleanup
run context-engine afterTurn/ingest/maintenance outside the post-prompt session lock, with only the actual SessionManager rewrite helper reacquiring the lock
add configurable session.writeLock.staleMs and session.writeLock.maxHoldMs policy with env overrides, and propagate those options through transcript writers, doctor, gateway startup cleanup, SDK runtime, Codex transcript mirroring, compaction, and rewrite/truncation paths
add regression coverage for prompt release/reacquire, takeover fencing, event queue draining, tool-call/live-output fences, context-engine maintenance rewrite locking, cleanup no-op after takeover, and stale/max-hold config

Reproduction

I reproduced the issue on unmodified main in Blacksmith Testbox through Crabbox.

provider: blacksmith-testbox
Testbox id: tbx_01krsajemdgtkm2288545hcsfe
Actions: https://github.com/openclaw/openclaw/actions/runs/25973229414
observed embedded run lock spans: embedded-lock-spans-prompt acquire=68758 prompt=113525 cleanup=180285
observed contention timeout: same-process-contention timeoutElapsedMs=26 message=session file locked (timeout 25ms): pid=4306 /tmp/openclaw-lock-repro-yYhC6d/session.jsonl.lock

That proved the embedded run still held the session write lock across prompt/model I/O and caused another same-process transcript writer to time out while the model turn was in flight.

Verification

Current head: a53710bba15daf53e8e2dd19c2a9b23e5f7431f6.

Local live provider/channel E2E on the user's local machine:

pnpm build
OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 \
OPENCLAW_LIVE_OPENAI_KEY=<redacted from local OPENAI_API_KEY> \
OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS=3000 \
OPENCLAW_QA_MATRIX_CANARY_TIMEOUT_MS=120000 \
OPENCLAW_QA_MATRIX_TIMEOUT_MS=2400000 \
OPENCLAW_QA_MATRIX_PROGRESS=1 \
node --import tsx --input-type=module -e 'import { runQaMatrixCommand } from "./extensions/qa-matrix/src/cli.runtime.ts"; await runQaMatrixCommand({ repoRoot: ".", outputDir: ".artifacts/qa-e2e/matrix-local-live-20260517T064732Z", providerMode: "live-frontier", primaryModel: "openai/gpt-5.4", alternateModel: "openai/gpt-5.4", fastMode: true, failFast: true, profile: "fast" });'

Result:

pnpm build passed after the final runner type cleanup.
Matrix QA output: .artifacts/qa-e2e/matrix-local-live-20260517T064732Z/matrix-qa-output.log
Matrix QA report: .artifacts/qa-e2e/matrix-local-live-20260517T064732Z/matrix-qa-report.md
Matrix QA summary: .artifacts/qa-e2e/matrix-local-live-20260517T064732Z/matrix-qa-summary.json
disposable channel: Tuwunel Matrix homeserver image ghcr.io/matrix-construct/tuwunel:v1.5.1, server matrix-qa.test
live provider/model: live-frontier, openai/gpt-5.4
canary: pass in 12016ms
suite: 15/15 passed, 0 failed, total 163055ms
observed Matrix events: 108
fast-profile scenarios passed: matrix-thread-follow-up, matrix-thread-isolation, matrix-top-level-reply-shape, matrix-reaction-notification, matrix-approval-exec-metadata-single-event, matrix-approval-exec-metadata-chunked, matrix-restart-resume, matrix-mention-gating, matrix-allowbots-default-block, matrix-allowbots-mentions-mentioned-room, matrix-allowlist-block, matrix-e2ee-basic-reply

Focused local regression proof on the same head:

node scripts/run-vitest.mjs src/agents/session-write-lock.test.ts src/commands/doctor-session-locks.test.ts src/config/zod-schema.session-maintenance-extensions.test.ts src/config/schema.help.quality.test.ts src/agents/pi-embedded-runner/context-engine-maintenance.test.ts src/agents/pi-embedded-runner/run/attempt.session-lock.test.ts src/agents/pi-embedded-runner/google-prompt-cache.test.ts src/agents/pi-embedded-runner/transcript-rewrite.test.ts src/agents/pi-embedded-runner/run/attempt.subscription-cleanup.test.ts src/agents/pi-embedded-runner/run/attempt.spawn-workspace.context-engine.test.ts src/agents/pi-embedded-runner/run/runtime-context-prompt.test.ts extensions/codex/src/app-server/transcript-mirror.test.ts src/plugin-sdk/agent-harness-runtime.test.ts
git diff --check

Result: Test Files 21 passed (21), Tests 310 passed (310), diff check clean.

Earlier local baseline proof before the final live E2E type cleanup:

node --import tsx scripts/generate-config-doc-baseline.ts --check
node --max-old-space-size=8192 --import tsx scripts/generate-plugin-sdk-api-baseline.ts --check

Result: both generated baseline checks OK.

Final Blacksmith Testbox through Crabbox proof:

provider: blacksmith-testbox
Testbox id: tbx_01krt8tz8kf8ddzktzj01yy63g
Actions: https://github.com/openclaw/openclaw/actions/runs/25983142307
command: CI=1 NODE_OPTIONS=--max-old-space-size=4096 OPENCLAW_TEST_PROJECTS_PARALLEL=6 OPENCLAW_VITEST_MAX_WORKERS=1 OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=900000 OPENCLAW_TESTBOX=1 OPENCLAW_TESTBOX_REMOTE_RUN=1 pnpm test ...focused session-lock/context-engine suite...
result: passed 6 Vitest shards in 9.13s, exit 0
summary: {"provider":"blacksmith-testbox","leaseId":"tbx_01krt8tz8kf8ddzktzj01yy63g","syncDelegated":true,"commandMs":51926,"totalMs":54468,"exitCode":0}

Additional review/proof completed during iteration:

Earlier final proof before the context-engine PR feedback fix: provider blacksmith-testbox, id tbx_01krt2k7k76gtfn65725vdj4kn, Actions https://github.com/openclaw/openclaw/actions/runs/25981164105, Test Files 19 passed (19), Tests 272 passed (272), exit 0.
ClawSweeper found that context-engine finalizers were still inside the post-prompt lock; the current head moves those hooks outside the lock and narrows locking to the SessionManager rewrite helper.

Real behavior proof

Behavior addressed: embedded PI runs no longer hold the session transcript write lock across model/provider I/O; context-engine post-turn hooks no longer run under the post-prompt transcript lock; stale attempts are fenced before live output, tool execution, transcript writes, and cleanup after a takeover.

Real environment tested: local macOS source checkout with Docker Desktop and a disposable Tuwunel Matrix homeserver, using the user's local OpenAI provider token for live-frontier / openai/gpt-5.4; Blacksmith Testbox through Crabbox for remote Linux focused regression proof.

Exact steps or command run after this patch: pnpm build; local Matrix QA fast profile with OPENCLAW_LIVE_OPENAI_KEY=<redacted from local OPENAI_API_KEY>, providerMode: "live-frontier", primaryModel: "openai/gpt-5.4", alternateModel: "openai/gpt-5.4"; the focused node scripts/run-vitest.mjs ... command above; git diff --check; and the focused suite in Blacksmith Testbox through Crabbox.

Evidence after fix: live Matrix provider/channel E2E passed with 15/15 checks/scenarios, 0 failed, canary pass in 12016ms, 108 observed Matrix events, report at .artifacts/qa-e2e/matrix-local-live-20260517T064732Z/matrix-qa-report.md; local regression proof passed with 21 test files and 310 tests; Testbox proof passed with provider blacksmith-testbox, id tbx_01krt8tz8kf8ddzktzj01yy63g, 6 Vitest shards passed, exit 0.

Observed result after fix: a real Matrix channel and live OpenAI provider turn completed end to end, followed by the Matrix fast-profile contract scenarios for thread replies, top-level reply shape, reaction observation, approval metadata, restart/resume, mention gating, allow-bot policy, sender allowlist blocking, and encrypted-room E2EE reply. Lock lifecycle tests verify prompt-time release, fenced reacquire, takeover no-op cleanup, live-output fencing, tool-call fencing, external-hook locking, context-engine rewrite-only locking, and stale/max-hold policy propagation.

What was not tested: full repository gates beyond the targeted session-lock, embedded-run, context-engine maintenance, config/docs baseline, SDK runtime, Codex transcript-mirror surface, and the Matrix live provider/channel E2E described above.

clawsweeper · 2026-05-17T04:13:19Z

Codex review: passed.

Summary
The PR narrows embedded PI session transcript write-lock scope, adds stale/max-hold config plumbing, and updates affected transcript, doctor, gateway, SDK, Codex mirroring, docs, and regression-test surfaces.

Reproducibility: yes. Current main source still holds the embedded session write lock from early attempt setup until cleanup, and the PR body includes Blacksmith Testbox contention proof on unmodified main; I did not rerun the live repro in this read-only pass.

Real behavior proof
Not applicable: The contributor proof gate is not applicable because this is a MEMBER-authored, maintainer-labeled PR; the body still includes live Matrix/OpenAI and Blacksmith Testbox proof.

Next step before merge
No ClawSweeper repair job remains; the existing clawsweeper:automerge flow can proceed through exact-head review and checks.

Security
Cleared: No concrete security or supply-chain regression was found in the exact-head diff.

Review details

Best possible solution:

Land the repaired PR through the existing automerge/check gates, preserving the controller-based release/fence design and the new skipped-prompt/context-engine regression coverage.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main source still holds the embedded session write lock from early attempt setup until cleanup, and the PR body includes Blacksmith Testbox contention proof on unmodified main; I did not rerun the live repro in this read-only pass.

Is this the best way to solve the issue?

Yes. The repaired head releases before provider/cache I/O, fences short transcript-write and cleanup sections, and adds regression coverage for the skipped-prompt path that the prior review flagged.

What I checked:

Protected PR context: Provided PR context shows author association MEMBER, protected maintainer label, clawsweeper:automerge, base 3dd8bcb419698c78b3371c35e7d8c55514db14ee, and repaired head 4c6dd7ed6e560aa8149aaa00c39f45f3f36253d4, so cleanup-close is not allowed and exact-head review is the right action. (4c6dd7ed6e56)
Current-main source repro: Current main acquires the embedded attempt session write lock before session setup/model work and passes that same lock into cleanup near the end of the attempt, matching the broad lock lifecycle reported in the linked issue and PR body. (src/agents/pi-embedded-runner/run/attempt.ts:2016, 3dd8bcb41969)
Prompt-time release and fallback release: PR head builds a session lock controller, wraps stream submission to drain events and release before provider/cache I/O, and also calls releaseForPrompt() before post-turn finalizers so skipped-prompt paths are covered. (src/agents/pi-embedded-runner/run/attempt.ts:3762, 4c6dd7ed6e56)
Fenced critical sections: The new controller releases the coarse lock, records a session-file fence, reacquires only for short transcript writes, returns a no-op cleanup lock after takeover, and wraps stream submission with event-drain before release. (src/agents/pi-embedded-runner/run/attempt.session-lock.ts:300, 4c6dd7ed6e56)
Context-engine finalizers moved outside coarse lock: Post-turn context-engine lifecycle runs outside the post-prompt transcript lock, while only active session-manager rewrites receive withSessionManagerRewriteLock. (src/agents/pi-embedded-runner/run/attempt.ts:4273, 4c6dd7ed6e56)
Skipped-prompt regression coverage: Regression tests now assert blank-prompt and before_agent_run block paths release the initial lock before post-turn writes, directly covering the earlier ClawSweeper finding. (src/agents/pi-embedded-runner/run/attempt.spawn-workspace.context-engine.test.ts:795, 4c6dd7ed6e56)

Likely related people:

steipete: Peter Steinberger appears repeatedly in session-lock history, including single-writer session files, embedded session lock hardening, lock budget alignment, and contention cleanup. (role: session lock and embedded runner feature-history owner; confidence: high; commits: 6668805aca17, cf8b3ed988ac, fb6e415d0c52; files: src/agents/session-write-lock.ts, src/agents/pi-embedded-runner/run/attempt.ts)
vincentkoc: Vincent Koc owns recent current-main blame for the broad embedded lock site in this checkout and has nearby session-lock follow-up commits for exit listeners, recycled PID recovery, and active-turn queued prompts. (role: recent session lock and embedded-run area contributor; confidence: medium; commits: 540a4a73d556, f00f0a95968f, 5a2200b28003; files: src/agents/session-write-lock.ts, src/agents/pi-embedded-runner/run/attempt.ts)
Josh Lehman: Josh Lehman introduced context-engine transcript maintenance, which is one of the post-turn surfaces this PR moves outside the coarse session lock except for rewrite helpers. (role: context-engine transcript maintenance feature-history contributor; confidence: medium; commits: 751d5b7849ca; files: src/agents/pi-embedded-runner/context-engine-maintenance.ts)

Remaining risk / open question:

I did not rerun the targeted suites in this read-only pass; exact-head CI/automerge checks should remain the merge gate for this broad session/runtime change.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 3dd8bcb41969.

amknight · 2026-05-17T11:06:31Z

/clawsweeper automerge

clawsweeper · 2026-05-17T11:08:29Z

ClawSweeper 🐠 automerge status

This repair pass finished without changing the PR. ClawSweeper checked the branch and found no safe patch to push this time.

Executor outcome: no planned fix actions.
Worker summary: No repair, close, or merge action is needed from this worker. The hydrated artifact shows canonical this PR already merged at 2026-05-17T11:54:04Z with merge commit 8a060b2, and linked issue #13744 was already closed at 2026-05-17T11:54:05Z. Job policy also blocks direct close and merge actions.

Worker actions:

keep_closed on this PR: skipped - The canonical PR has already landed; closure/merge mutations are both blocked by the job and unnecessary.
keep_closed on #13744: skipped - The linked issue is already closed after the canonical fix landed; no close action is valid for an already-closed target.

Nothing moved downstream from this pass: no branch update, replacement PR, merge, or re-review.

fish notes: model gpt-5.5, reasoning high.

Automerge progress:

2026-05-17 11:06:31 UTC review queued a53710bba15d (queued)

2026-05-17 11:08:28 UTC repair queued a53710bba15d (autonomous) Run: https://github.com/openclaw/clawsweeper/actions/runs/25989151126

2026-05-17 11:10:56 UTC repair started (running) in 0s Run: https://github.com/openclaw/clawsweeper/actions/runs/25989151126 automerge-openclaw-openclaw-82891

2026-05-17 11:11:11 UTC validation plan (passed) in 15s Run: https://github.com/openclaw/clawsweeper/actions/runs/25989151126 pnpm check:changed; pnpm lint; pnpm check:test-types

2026-05-17 11:11:24 UTC Codex write preflight (passed) in 29s Run: https://github.com/openclaw/clawsweeper/actions/runs/25989151126 danger-full-access

2026-05-17 11:20:52 UTC Codex edit 1 133d6be2683d (complete) in 9m 57s Run: https://github.com/openclaw/clawsweeper/actions/runs/25989151126 exit 0

2026-05-17 11:46:33 UTC validation and review 1 4c6dd7ed6e56 (base moved) in 35m 37s Run: https://github.com/openclaw/clawsweeper/actions/runs/25989151126 rebased

2026-05-17 11:46:37 UTC repair completed 4c6dd7ed6e56 (branch updated) in 35m 41s Run: https://github.com/openclaw/clawsweeper/actions/runs/25989151126 initial automerge rebase is delegated to Codex repair

2026-05-17 11:46:37 UTC review queued 4c6dd7ed6e56 (after repair)

2026-05-17 11:46:38 UTC automerge wait 4c6dd7ed6e56 (waiting) in 35m 42s Run: https://github.com/openclaw/clawsweeper/actions/runs/25989151126 up to 600s

2026-05-17 11:53:53 UTC review passed 4c6dd7ed6e56 (structured ClawSweeper verdict: pass (sha=4c6dd7ed6e560aa8149aaa00c39f45f3f3625...)

2026-05-17 11:54:06 UTC merged 4c6dd7ed6e56 (merged by ClawSweeper automerge)

2026-05-17 12:09:58 UTC repair started (running) in 0s Run: https://github.com/openclaw/clawsweeper/actions/runs/25990391254 automerge-openclaw-openclaw-82891

2026-05-17 12:10:00 UTC repair completed (no branch change) in 2s Run: https://github.com/openclaw/clawsweeper/actions/runs/25990391254 no planned fix actions

…1 (1)

…1 (2)

Summary: - The PR narrows embedded PI session transcript write-lock scope, adds stale/max-hold config plumbing, and updates affected transcript, doctor, gateway, SDK, Codex mirroring, docs, and regression-test surfaces. - Reproducibility: yes. Current main source still holds the embedded session write lock from early attempt set ... cksmith Testbox contention proof on unmodified main; I did not rerun the live repro in this read-only pass. Automerge notes: - PR branch already contained follow-up commit before automerge: fix(agents): narrow context engine session lock - PR branch already contained follow-up commit before automerge: fix session lock runner build types - PR branch already contained follow-up commit before automerge: Release embedded session write lock before model I/O - PR branch already contained follow-up commit before automerge: fix(clawsweeper): address review for automerge-openclaw-openclaw-8289… Validation: - ClawSweeper review passed for head 4c6dd7ed6e560aa8149aaa00c39f45f3f36253d4. - Required merge gates passed before the squash merge. Prepared head SHA: 4c6dd7ed6e560aa8149aaa00c39f45f3f36253d4 Review: openclaw/openclaw#82891 (comment) Co-authored-by: Alex Knight <15041791+amknight@users.noreply.github.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>

Summary: - The PR narrows embedded PI session transcript write-lock scope, adds stale/max-hold config plumbing, and updates affected transcript, doctor, gateway, SDK, Codex mirroring, docs, and regression-test surfaces. - Reproducibility: yes. Current main source still holds the embedded session write lock from early attempt set ... cksmith Testbox contention proof on unmodified main; I did not rerun the live repro in this read-only pass. Automerge notes: - PR branch already contained follow-up commit before automerge: fix(agents): narrow context engine session lock - PR branch already contained follow-up commit before automerge: fix session lock runner build types - PR branch already contained follow-up commit before automerge: Release embedded session write lock before model I/O - PR branch already contained follow-up commit before automerge: fix(clawsweeper): address review for automerge-openclaw-openclaw-8289… Validation: - ClawSweeper review passed for head 4c6dd7e. - Required merge gates passed before the squash merge. Prepared head SHA: 4c6dd7e Review: openclaw#82891 (comment) Co-authored-by: Alex Knight <15041791+amknight@users.noreply.github.com> Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>

openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime commands Command implementations agents Agent runtime and tooling extensions: codex size: XL maintainer Maintainer-authored PR labels May 17, 2026

clawsweeper Bot added P2 Normal backlog priority with limited blast radius. impact:message-loss Channel message delivery can be lost, duplicated, or misrouted. impact:session-state Session, memory, transcript, context, or agent state can drift or corrupt. labels May 17, 2026

clawsweeper Bot mentioned this pull request May 17, 2026

Make session write lock configurable + narrow lock scope (avoid timeout=All models failed) #13744

Closed

amknight marked this pull request as ready for review May 17, 2026 05:43

amknight force-pushed the ak/session-write-lock-contention branch from 3b87580 to 037323c Compare May 17, 2026 06:08

amknight marked this pull request as draft May 17, 2026 06:08

amknight marked this pull request as ready for review May 17, 2026 06:25

amknight marked this pull request as draft May 17, 2026 06:54

amknight marked this pull request as ready for review May 17, 2026 08:21

clawsweeper Bot added the clawsweeper:automerge Maintainer opted this PR into bounded ClawSweeper-reviewed automerge label May 17, 2026

amknight and others added 6 commits May 17, 2026 11:46

fix(agents): release embedded session lock before model io

9a9c557

fix(agents): narrow context engine session lock

96094fb

fix session lock runner build types

d1c7832

Release embedded session write lock before model I/O

4513cae

fix(clawsweeper): address review for automerge-openclaw-openclaw-8289…

fd2fe46

…1 (1)

fix(clawsweeper): address review for automerge-openclaw-openclaw-8289…

4c6dd7e

…1 (2)

clawsweeper Bot force-pushed the ak/session-write-lock-contention branch from a53710b to 4c6dd7e Compare May 17, 2026 11:46

clawsweeper Bot merged commit 8a060b2 into main May 17, 2026
111 checks passed

clawsweeper Bot deleted the ak/session-write-lock-contention branch May 17, 2026 11:54

github-actions Bot mentioned this pull request May 17, 2026

📡 Upstream Digest — 2026-05-17 13:00 UTC curtismercier/openclaw-mods#881

Open

cael-dandelion-cult mentioned this pull request May 18, 2026

fix(agents): rethrow EmbeddedAttemptSessionTakeoverError before model fallback #83436

Open

ronan-dandelion-cult mentioned this pull request May 19, 2026

cure-3: catalog writers to session jsonl during embedded prompt-release window karmaterminal/openclaw#710

Open

This was referenced May 20, 2026

fix(agents): tolerate in-process session writes during prompt release #84250

Merged

[Feature]: Add before_agent_run hook support to embedded runner #84405

Closed

bhattman-dev mentioned this pull request May 20, 2026

Discord embedded-run prep wedge before strict-agentic, recovery skips sessionId=unknown lanes #84477

Closed

clawsweeper Bot mentioned this pull request May 25, 2026

fix(agents): file-scoped prompt-window guard for same-session embedded races #86067

Closed

openperf mentioned this pull request May 26, 2026

fix(agents): release embedded-attempt session lock on every exit path #86427

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Release embedded session write lock before model I/O#82891

Release embedded session write lock before model I/O#82891
clawsweeper[bot] merged 6 commits into
mainfrom
ak/session-write-lock-contention

amknight commented May 17, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented May 17, 2026 •

edited

Loading

Uh oh!

amknight commented May 17, 2026

Uh oh!

clawsweeper Bot commented May 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

amknight commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Reproduction

Verification

Real behavior proof

Uh oh!

clawsweeper Bot commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amknight commented May 17, 2026

Uh oh!

clawsweeper Bot commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

amknight commented May 17, 2026 •

edited

Loading

clawsweeper Bot commented May 17, 2026 •

edited

Loading

clawsweeper Bot commented May 17, 2026 •

edited

Loading