fix(gateway): auto-deliver video_generate output without a model-emitted MEDIA tag#45786
fix(gateway): auto-deliver video_generate output without a model-emitted MEDIA tag#45786Julientalbot wants to merge 1 commit into
Conversation
…ted MEDIA tag
video_generate returns a structured {"success": true, "video": "<url-or-abs-path>"}
result, but it was absent from the auto-append allowlist and the JSON field map,
so a generated video was delivered ONLY if the model restated a correctly
formatted MEDIA: tag in its prose reply. When the model omits or mis-formats the
tag (e.g. wraps it in Markdown), the video is silently never attached.
Generalize the existing deterministic image_generate auto-append into a per-tool
field map (image_generate -> host_image/image/agent_visible_image,
video_generate -> video) and add video_generate to the producer-tool allowlist,
so the path is read from the tool's structured result rather than from the
model's free text. All four existing guards are reused unchanged: producer-tool
allowlist (write_file/terminal etc. excluded), success-gate, _TOOL_MEDIA_RE
(rejects remote URLs and non-deliverable extensions), and current-turn isolation
+ history dedup. TTS keeps its existing MEDIA:-scan path (which also carries the
[[audio_as_voice]] directive).
Adds TestMediaExtraction coverage: video_generate structured path delivered,
remote URL ignored, failed generation ignored, history dedup, and a guard that
the "video" field is only harvested from the allowlisted producer tool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Duplicate of #43065 — same mechanism: add |
|
Closing as a duplicate of #43065 — thanks @alt-glitch for the catch. #43065 (opened 2026-06-09) lands the same mechanism first: Two deltas from this PR that may be worth folding into #43065, offered there, not as a reason to keep this one open:
Deferring to the earlier PR. 👍 |
What does this PR do?
Makes generated-video delivery model-agnostic, and generalizes the existing
deterministic media auto-append so it covers the whole producer-tool family
instead of just
image_generate.The bug.
video_generatereturns a structured result —{"success": true, "video": "<url-or-abs-path>"}(seeagent/video_gen_provider.pysuccess_response) — but it was absent fromboth
_AUTO_APPEND_MEDIA_TOOL_NAMESand_JSON_MEDIA_TOOL_PATH_FIELDSingateway/run.py. So a generated video is delivered only if the model restatesa correctly-formatted
MEDIA:tag in its prose reply. When the model omits thetag, or wraps/mis-formats it (e.g. Markdown emphasis), the file is silently never
attached and the user gets nothing.
image_generatealready avoids this byreading its path from the tool's JSON result; videos had no such path.
The fix. Generalize the proven
image_generateJSON branch into a smallper-tool field map and add
video_generateto the allowlist:The path is now read from the tool's structured result, so delivery no longer
depends on how (or whether) the model restates it in free text. This is the
model-agnostic counterpart to the prose-parsing path.
Related Issue
Companion to #45773 (
fix(gateway): deliver MEDIA: tags wrapped in Markdown emphasis) — different layer, disjoint files. #45773 hardens the prose-parsepath (
extract_mediaregex inbase.py) for hand-authored / pre-existing paths;this PR fixes the structured-harvest path (auto-append in
run.py) fortool-produced media. They are independent and can land in either order. No
separate issue filed — the gap is fully described above.
Type of Change
Changes Made
gateway/run.py— add"video_generate"to_AUTO_APPEND_MEDIA_TOOL_NAMES;turn
_JSON_MEDIA_TOOL_PATH_FIELDSinto a per-tool field map; generalize thetool_name == "image_generate"branch tojson_fields = MAP.get(tool_name).Docstring/comments updated. No interface, schema, or config change.
tests/gateway/test_media_extraction.py—video_generatecoverage:structured path delivered, remote URL ignored, failed generation ignored,
history dedup, and a guard that the
"video"field is only harvested from theallowlisted producer tool (a
execute_codelook-alike result is notdelivered).
How to Test
video_generateresult{"success": true, "video": "/tmp/clip.mp4"}with a text-only assistant reply → video not delivered.
_collect_auto_append_media_tags(...)returns["MEDIA:/tmp/clip.mp4"]→ delivered, independent of the model's prose.pytest tests/gateway/test_media_extraction.py -q→ all pass (incl. theexisting
image_generate/ TTS / stale-leak / compression-fallback cases).False-positive safety
All four existing guards are reused unchanged:
image_generate,video_generate, TTS areharvested; generic file producers (
write_file,terminal, …) are excluded,so an internal/temp file they write is never auto-delivered.
payload.get("success")must be truthy._TOOL_MEDIA_RE— requires a deliverable extension + a real local-pathanchor, so remote URLs (a provider may return a URL in
video) andnon-media artifacts are rejected (delivery then falls back to the gateway's
existing URL-download path — unchanged, out of scope here).
Checklist
Code
fix(gateway):)pytest tests/ -q— ran the media zone:tests/gateway/test_media_extraction.py,test_platform_base.py,test_run_tool_media_re.py(196 passed, 2 skipped); change is localized to the auto-append pathDocumentation & Housekeeping
gateway/run.py