Skip to content

fix(agent): persist plugin-transformed responses so resume replays what the user saw#44244

Open
AIalliAI wants to merge 1 commit into
NousResearch:mainfrom
AIalliAI:fix/44239-persist-transformed-response
Open

fix(agent): persist plugin-transformed responses so resume replays what the user saw#44244
AIalliAI wants to merge 1 commit into
NousResearch:mainfrom
AIalliAI:fix/44239-persist-transformed-response

Conversation

@AIalliAI

Copy link
Copy Markdown
Contributor

Summary

Fixes #44239

The transform_llm_output hook fires in finalize_turn after _persist_session, and its result is never written back into the assistant message. The user sees the transformed text for the current turn, but result["messages"], the JSON session log, and the SQLite session DB all keep the raw model output — which is then replayed as context on the next turn or after a session resume. The whole response_transformed delivery chain (gateway message edit, ACP final-response delivery) ensures the transformed text is what the user sees, so the raw text resurfacing on resume is clearly unintended.

Changes — agent/turn_finalizer.py

Scope notes

While verifying #44239 I found parts of it don't match the code:

  • The post_llm_call "override_response path" it describes does not exist — _invoke_hook("post_llm_call", ...)'s return value is discarded. So there is nothing to persist for that hook; this PR fixes the half of the issue that is real (transform_llm_output).
  • The "reference implementation" link in the issue (M1p0/Project-Rin patch) returns 404.

Tests

New tests/agent/test_44239_transform_persist_order.py (7 tests) drives finalize_turn directly with a stub agent and a patched invoke_hook:

  • transformed text lands in both result["messages"][-1] and the persisted snapshot;
  • ordering: transform_llm_output → persist → post_llm_call;
  • untransformed turns persist the raw text unchanged;
  • the sync helper skips tool-call messages, multimodal content, and never crosses the turn boundary.

tests/agent/ + tests/run_agent/ (5773 passed) plus the gateway/ACP response_transformed suites pass; the 9 failures present are pre-existing environment/order-dependent flakes that also fail on clean main (OAuth credential-file tests) or pass in isolation on this branch (vision routing / provider parity).

Related

🤖 Generated with Claude Code

…at the user saw

The transform_llm_output hook fired after _persist_session, and its
result was never written back into the assistant message. The user saw
the transformed text for the current turn, but result["messages"], the
JSON session log, and the SQLite session DB all kept the raw model
output — which was then replayed as context on the next turn or after a
session resume.

Move persistence after the transform_llm_output hook and sync the
transformed text into the turn's last assistant text message first.
The sync stops at the turn boundary and never rewrites a tool-call or
non-text message. Persistence stays before post_llm_call, which is
observability-only (its return value is ignored) and whose plugins may
read the session store expecting the completed turn to be present.

Fixes NousResearch#44239

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent loop, run_agent.py, prompt builder comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have labels Jun 11, 2026
@liuhao1024

Copy link
Copy Markdown
Contributor

Verification review — reviewed the diff and 206-line test file; no issues found.

The fix correctly reorders _persist_session to run after the transform_llm_output hook, then syncs the transformed response back into the turn's last assistant message via _sync_final_response_to_last_assistant. The new helper walks backwards within the current turn (stops at the user message boundary) and only updates plain-text assistant messages — tool-call messages and multimodal content are correctly skipped.

Test coverage is thorough:

  • Transformed response persisted correctly
  • Hook → persist → post_llm_call ordering enforced
  • Untransformed responses pass through raw
  • Tool-call and turn-boundary skips verified
  • Non-text assistant content skipped

@AIalliAI

Copy link
Copy Markdown
Contributor Author

Requesting maintainer review — this is ready to land from my side. Standalone fork CI is pending first-run approval here; the rollup branch in #44061 carrying this session's batch is fully green on upstream CI (all test shards, typecheck, e2e).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent loop, run_agent.py, prompt builder comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: transform_llm_output and post_llm_call hooks both produce final_response/history mismatch — same root cause as #14894

3 participants