fix(agent): persist plugin-transformed responses so resume replays what the user saw#44244
fix(agent): persist plugin-transformed responses so resume replays what the user saw#44244AIalliAI wants to merge 1 commit into
Conversation
…at the user saw The transform_llm_output hook fired after _persist_session, and its result was never written back into the assistant message. The user saw the transformed text for the current turn, but result["messages"], the JSON session log, and the SQLite session DB all kept the raw model output — which was then replayed as context on the next turn or after a session resume. Move persistence after the transform_llm_output hook and sync the transformed text into the turn's last assistant text message first. The sync stops at the turn boundary and never rewrites a tool-call or non-text message. Persistence stays before post_llm_call, which is observability-only (its return value is ignored) and whose plugins may read the session store expecting the completed turn to be present. Fixes NousResearch#44239 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Verification review — reviewed the diff and 206-line test file; no issues found. The fix correctly reorders Test coverage is thorough:
|
|
Requesting maintainer review — this is ready to land from my side. Standalone fork CI is pending first-run approval here; the rollup branch in #44061 carrying this session's batch is fully green on upstream CI (all test shards, typecheck, e2e). |
Summary
Fixes #44239
The
transform_llm_outputhook fires infinalize_turnafter_persist_session, and its result is never written back into the assistant message. The user sees the transformed text for the current turn, butresult["messages"], the JSON session log, and the SQLite session DB all keep the raw model output — which is then replayed as context on the next turn or after a session resume. The wholeresponse_transformeddelivery chain (gateway message edit, ACP final-response delivery) ensures the transformed text is what the user sees, so the raw text resurfacing on resume is clearly unintended.Changes —
agent/turn_finalizer.py_persist_sessionafter thetransform_llm_outputhook block.final_responseback into the turn's last assistant text message via a new_sync_final_response_to_last_assistanthelper. The sync walks backwards only within the current turn (stops at the user-message boundary, mirroring the reasoning extraction) and never rewrites a tool-call or non-text (multimodal) assistant message.messages[-1], exactly as before), so a transformed recovery/explainer response can never resurrect a message the drop removed.post_llm_call: that hook is observability-only perdocs/observability/README.md(its return value is ignored in code — there is no override path, contrary to what [Bug]: transform_llm_output and post_llm_call hooks both produce final_response/history mismatch — same root cause as #14894 #44239 and [Bug]: post_llm_call response overrides are applied after persistence, causing final_response/history mismatch #14894 assume), and its plugins may read the session store expecting the completed turn to be present. This also means this PR takes a deliberately different approach from fix(agent): run post_llm_call hook before session persistence #14913, which moves persistence belowpost_llm_call.Scope notes
While verifying #44239 I found parts of it don't match the code:
post_llm_call"override_response path" it describes does not exist —_invoke_hook("post_llm_call", ...)'s return value is discarded. So there is nothing to persist for that hook; this PR fixes the half of the issue that is real (transform_llm_output).Tests
New
tests/agent/test_44239_transform_persist_order.py(7 tests) drivesfinalize_turndirectly with a stub agent and a patchedinvoke_hook:result["messages"][-1]and the persisted snapshot;transform_llm_output→ persist →post_llm_call;tests/agent/+tests/run_agent/(5773 passed) plus the gateway/ACPresponse_transformedsuites pass; the 9 failures present are pre-existing environment/order-dependent flakes that also fail on cleanmain(OAuth credential-file tests) or pass in isolation on this branch (vision routing / provider parity).Related
post_llm_call; see scope note above.🤖 Generated with Claude Code