fix(bedrock, anthropic): translate OpenAI file content on tool-result path#26710
Conversation
… path
OpenAI Chat Completions `{type: "file", file: {file_data: "data:application/pdf;..."}}`
content blocks inside tool messages were silently dropped when translated to
Bedrock Converse and direct Anthropic. Additionally, PDFs sent via `image_url`
data URIs were either dropped (Bedrock) or wrapped as `type: "image"` and
rejected by the API (Anthropic).
- _convert_to_bedrock_tool_call_result: add `type: "file"` branch; pass through
document blocks produced by BedrockImageProcessor for PDF `image_url` URIs.
Single choke point covers both sync and async converse paths.
- convert_to_anthropic_tool_result: add `type: "file"` branch delegating to
`anthropic_process_openai_file_message`; branch `image_url` on data-URI mime
type so non-image mimes route through the file helper to produce document
blocks.
- AnthropicMessagesToolResultParam.content union extended to accept
`AnthropicMessagesDocumentParam` alongside text and image.
- Add 6 tests (3 Bedrock + 3 Anthropic) covering file-PDF, image_url-PDF, and
image_url-PNG regression.
Fixes BerriAI#24641
Supersedes BerriAI#24646 with an OpenAI-native approach and test coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Duplicate the three Bedrock and three Anthropic tool-result tests into tests/test_litellm/ so they're picked up by `make test-unit` (and its coverage report). The originals in tests/llm_translation/ stay — they run under integration and remain the canonical translation-suite regression cases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…correctness fixes - openai/codex#20046 lock InterAgentCommunication commentary-phase contract - BerriAI/litellm#26710 file content on Bedrock+Anthropic tool-result path - google-gemini/gemini-cli#26136 disconnect extension-backed MCP clients via clientKey
Greptile SummaryThis PR fixes two silent-drop bugs in the OpenAI→Bedrock and OpenAI→Anthropic tool-result converters: Confidence Score: 5/5Safe to merge — fixes are isolated to tool-result translation helpers with good test coverage and no regressions in existing behavior. No P0/P1 issues found. The one P2 (container_upload type gap on the Anthropic path) is an unlikely edge case with a clear, non-silent failure mode. Prior review threads were fully addressed in the head commit. Tests are comprehensive and mock-only. No files require special attention.
|
| Filename | Overview |
|---|---|
| litellm/litellm_core_utils/prompt_templates/factory.py | Core fix: adds type: "file" branch and document-passthrough to both Bedrock and Anthropic tool-result converters; logic is sound with one minor type-safety gap for container_upload file_ids on the Anthropic path. |
| litellm/types/llms/anthropic.py | Extends AnthropicMessagesToolResultParam.content union to include AnthropicMessagesDocumentParam, correctly reflecting what the Anthropic API accepts in tool_result content. |
| tests/test_litellm/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_factory.py | Adds 5 Anthropic tool-result tests (PDF via file, PDF via image_url, PNG regression, unsupported-mime stays on image path, text/plain document routing); all use mocked/inline data, no real network calls. |
| tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py | Adds 5 Bedrock tool-result tests (PDF via file, PDF via image_url, file_id mock, both-None raises, PNG regression); the file_id test correctly patches BedrockImageProcessor to avoid network calls. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Tool Message content block] --> B{content type?}
B -->|text| C[Append text block]
B -->|image_url| D{Is document data URI?\napplication/pdf or text/plain}
B -->|file| E[Get file_data / file_id]
D -->|Yes| F[anthropic_process_openai_file_message\nsynth file object]
D -->|No| G[create_anthropic_image_param\nimage block]
E --> H{both None?}
H -->|Yes - Bedrock| I[raise BadRequestError]
H -->|Yes - Anthropic| J[raise Exception via helper]
H -->|No| K[anthropic_process_openai_file_message\nor BedrockImageProcessor.process_image_sync]
F --> L[Append document block to tool_result]
G --> M[Append image block to tool_result]
K --> N{block type?}
N -->|document| L
N -->|image| M
N -->|container_upload| O[⚠️ Appended but invalid\nin tool_result — API rejects]
Reviews (2): Last reviewed commit: "address Greptile review feedback on tool..." | Re-trigger Greptile
- Tighten _is_anthropic_document_data_uri to match the mimes Anthropic
actually accepts as base64 `document` source ({application/pdf,
text/plain}). The previous application/* + text/* prefix match would
route e.g. data:application/json URIs through the document path,
producing blocks the Anthropic API rejects. Unsupported mimes now
stay on the existing image code path (same failure mode as before the
fix — no regression, just stops introducing a new one).
- On the Bedrock tool-result `type: "file"` branch, accept either
file_data or file_id and raise BadRequestError on both-None, mirroring
the user-message _process_file_message pattern. Previously a file
block with only file_id was silently dropped.
- Consolidate the six new PDF tool-result tests under tests/test_litellm/
only (the PR template's required location and where the unit-test CI
workflow runs with coverage). The duplicate copies under
tests/llm_translation/ added drift risk with no additional coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Relevant issues
Fixes #24641
Supersedes #24646 (OpenAI-native approach with test coverage; see "Why a new PR" below)
Linear ticket
N/A — external contributor
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory — this PR adds 6 new tests undertests/llm_translation/(the long-standing home for Bedrock/Anthropic translation tests). Happy to relocate totests/test_litellm/if maintainers prefer.make test-unit— verified locally on translation-unit subsets (see CI links below once runs populate).filecontent) passthrough on the tool-result path for Bedrock Converse and direct Anthropic.@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewCI (LiteLLM team)
Link:
Link:
Links:
Screenshots / Proof of Fix
Before (current
litellm_internal_staging/v1.83.7-stable)When a caller sends a PDF as an OpenAI Chat Completions
filecontent block inside a tool message:{"role": "tool", "tool_call_id": "...", "content": [ {"type": "file", "file": {"file_data": "data:application/pdf;base64,...", "filename": "x.pdf"}}, ]}…the translated Bedrock
toolResult.contentis an empty list[], and the Anthropic tool_result content has zero document blocks. The model sees empty content and responds as if the tool returned nothing.After
The same input now translates to:
toolResult.content[0].documentwithformat: "pdf"and the base64 payload as source bytes.{"type": "document", "source": {"type": "base64", "media_type": "application/pdf", "data": ...}}block inside the tool_result content.Test output
All 6 new tests + the existing
test_bedrock_converse_translation_tool_messagepass:Type
🐛 Bug Fix
Changes
Root cause
The OpenAI → Bedrock and OpenAI → Anthropic tool-result converters in
litellm/litellm_core_utils/prompt_templates/factory.pyonly handledtextandimage_urlcontent types on the tool-message path. Two concrete bugs:Bedrock
_convert_to_bedrock_tool_call_result: No branch fortype: "file". Additionally, theimage_urlbranch calledBedrockImageProcessor.process_image_sync, which correctly returns a{"document": ...}block fordata:application/pdf;base64,...URIs — but the wrapper only appended when"image" in _block, so document blocks were silently dropped.Anthropic
convert_to_anthropic_tool_result: No branch fortype: "file". Theimage_urlbranch unconditionally calledcreate_anthropic_image_param, wrapping any data URI (includingapplication/pdf) astype: "image"— which Anthropic rejects for non-image mime types.User-message paths already handle
type: "file"correctly (viaanthropic_process_openai_file_messagefor Anthropic and_process_file_messagefor Bedrock); the gap was specifically on tool-result content.Fix
Bedrock (
_convert_to_bedrock_tool_call_result):type: "file"branch that delegates toBedrockImageProcessor.process_image_syncand wraps the returned block asdocumentorimageappropriately.documentpassthrough in the existingimage_urlbranch so processor-produced document blocks (PDFs sent as image_url data URIs) are no longer dropped.This single fix covers both the sync (
_bedrock_converse_messages_pt) and async (_bedrock_converse_messages_pt_async) paths because both funnel tool-result translation through this helper.Anthropic (
convert_to_anthropic_tool_result):type: "file"branch that delegates to the existinganthropic_process_openai_file_messagehelper.image_urlbranch, branched on the data-URI mime type: forapplication/*andtext/*URIs, routes through the same file helper to produce a properdocumentblock; otherwise keeps existing image handling. Added new helper_is_anthropic_document_data_uri.AnthropicMessagesToolResultParam.contentunion inlitellm/types/llms/anthropic.pyto acceptAnthropicMessagesDocumentParam(previously text + image only), reflecting what the Anthropic API actually accepts inside tool_result content.Why a new PR (superseding #24646)
#24646 has been stalled for ~a month (blocked, no reviews, no tests). It handles the same user-facing bug by adding a
type: "document"branch — but that's not a documented OpenAI Chat Completions content type. It forces callers to:documentshapes in the OpenAI payloaddocumentsub-schema differs from Anthropic's)This PR takes the OpenAI-native approach: callers send the documented OpenAI Chat Completions
filecontent shape for PDFs, and LiteLLM handles the provider-specific translation. Also includes test coverage (the #1 reason #24646 stalled).Happy to close this PR if maintainers prefer to revive #24646 instead.
Tests added
Six new tests under
tests/llm_translation/, all driven via the existing public translation entrypoints (_bedrock_converse_messages_ptandconvert_to_anthropic_tool_result):Bedrock:
test_bedrock_tool_message_openai_file_pdf_becomes_document— new behavior.test_bedrock_tool_message_image_url_pdf_data_uri_becomes_document— regression for the silent-drop bug.test_bedrock_tool_message_image_url_png_still_becomes_image— locks in existing image-block behavior.Anthropic:
test_anthropic_tool_result_openai_file_pdf_becomes_documenttest_anthropic_tool_result_image_url_pdf_data_uri_becomes_documenttest_anthropic_tool_result_image_url_png_still_becomes_imageEach new-behavior test was developed via strict red→green TDD — verified failing against the unpatched code first, then implemented minimal code to pass.