Skip to content

fix(bedrock, anthropic): translate OpenAI file content on tool-result path#26710

Merged
Sameerlite merged 3 commits intoBerriAI:litellm_oss_stagingfrom
minznerjosh:fix/bedrock-anthropic-tool-result-pdf-content
Apr 29, 2026
Merged

fix(bedrock, anthropic): translate OpenAI file content on tool-result path#26710
Sameerlite merged 3 commits intoBerriAI:litellm_oss_stagingfrom
minznerjosh:fix/bedrock-anthropic-tool-result-pdf-content

Conversation

@minznerjosh
Copy link
Copy Markdown
Contributor

@minznerjosh minznerjosh commented Apr 28, 2026

Relevant issues

Fixes #24641
Supersedes #24646 (OpenAI-native approach with test coverage; see "Why a new PR" below)

Linear ticket

N/A — external contributor

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory — this PR adds 6 new tests under tests/llm_translation/ (the long-standing home for Bedrock/Anthropic translation tests). Happy to relocate to tests/test_litellm/ if maintainers prefer.
  • My PR passes all unit tests on make test-unit — verified locally on translation-unit subsets (see CI links below once runs populate).
  • My PR's scope is as isolated as possible, it only solves 1 specific problem — PDF (OpenAI file content) passthrough on the tool-result path for Bedrock Converse and direct Anthropic.
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

  • Branch creation CI run
    Link:
  • CI run for the last commit
    Link:
  • Merge / cherry-pick CI run
    Links:

Screenshots / Proof of Fix

Before (current litellm_internal_staging / v1.83.7-stable)

When a caller sends a PDF as an OpenAI Chat Completions file content block inside a tool message:

{"role": "tool", "tool_call_id": "...", "content": [
    {"type": "file", "file": {"file_data": "data:application/pdf;base64,...", "filename": "x.pdf"}},
]}

…the translated Bedrock toolResult.content is an empty list [], and the Anthropic tool_result content has zero document blocks. The model sees empty content and responds as if the tool returned nothing.

After

The same input now translates to:

  • Bedrock: toolResult.content[0].document with format: "pdf" and the base64 payload as source bytes.
  • Anthropic: a {"type": "document", "source": {"type": "base64", "media_type": "application/pdf", "data": ...}} block inside the tool_result content.

Test output

All 6 new tests + the existing test_bedrock_converse_translation_tool_message pass:

tests/llm_translation/test_bedrock_completion.py::test_bedrock_converse_translation_tool_message PASSED
tests/llm_translation/test_bedrock_completion.py::test_bedrock_tool_message_openai_file_pdf_becomes_document PASSED
tests/llm_translation/test_bedrock_completion.py::test_bedrock_tool_message_image_url_pdf_data_uri_becomes_document PASSED
tests/llm_translation/test_bedrock_completion.py::test_bedrock_tool_message_image_url_png_still_becomes_image PASSED
tests/llm_translation/test_anthropic_completion.py::test_anthropic_tool_result_openai_file_pdf_becomes_document PASSED
tests/llm_translation/test_anthropic_completion.py::test_anthropic_tool_result_image_url_pdf_data_uri_becomes_document PASSED
tests/llm_translation/test_anthropic_completion.py::test_anthropic_tool_result_image_url_png_still_becomes_image PASSED

Type

🐛 Bug Fix

Changes

Root cause

The OpenAI → Bedrock and OpenAI → Anthropic tool-result converters in litellm/litellm_core_utils/prompt_templates/factory.py only handled text and image_url content types on the tool-message path. Two concrete bugs:

  1. Bedrock _convert_to_bedrock_tool_call_result: No branch for type: "file". Additionally, the image_url branch called BedrockImageProcessor.process_image_sync, which correctly returns a {"document": ...} block for data:application/pdf;base64,... URIs — but the wrapper only appended when "image" in _block, so document blocks were silently dropped.

  2. Anthropic convert_to_anthropic_tool_result: No branch for type: "file". The image_url branch unconditionally called create_anthropic_image_param, wrapping any data URI (including application/pdf) as type: "image" — which Anthropic rejects for non-image mime types.

User-message paths already handle type: "file" correctly (via anthropic_process_openai_file_message for Anthropic and _process_file_message for Bedrock); the gap was specifically on tool-result content.

Fix

Bedrock (_convert_to_bedrock_tool_call_result):

  • Added a type: "file" branch that delegates to BedrockImageProcessor.process_image_sync and wraps the returned block as document or image appropriately.
  • Added a document passthrough in the existing image_url branch so processor-produced document blocks (PDFs sent as image_url data URIs) are no longer dropped.

This single fix covers both the sync (_bedrock_converse_messages_pt) and async (_bedrock_converse_messages_pt_async) paths because both funnel tool-result translation through this helper.

Anthropic (convert_to_anthropic_tool_result):

  • Added a type: "file" branch that delegates to the existing anthropic_process_openai_file_message helper.
  • In the image_url branch, branched on the data-URI mime type: for application/* and text/* URIs, routes through the same file helper to produce a proper document block; otherwise keeps existing image handling. Added new helper _is_anthropic_document_data_uri.
  • Extended AnthropicMessagesToolResultParam.content union in litellm/types/llms/anthropic.py to accept AnthropicMessagesDocumentParam (previously text + image only), reflecting what the Anthropic API actually accepts inside tool_result content.

Why a new PR (superseding #24646)

#24646 has been stalled for ~a month (blocked, no reviews, no tests). It handles the same user-facing bug by adding a type: "document" branch — but that's not a documented OpenAI Chat Completions content type. It forces callers to:

  • Speak provider-native document shapes in the OpenAI payload
  • Branch on target provider (Bedrock's document sub-schema differs from Anthropic's)

This PR takes the OpenAI-native approach: callers send the documented OpenAI Chat Completions file content shape for PDFs, and LiteLLM handles the provider-specific translation. Also includes test coverage (the #1 reason #24646 stalled).

Happy to close this PR if maintainers prefer to revive #24646 instead.

Tests added

Six new tests under tests/llm_translation/, all driven via the existing public translation entrypoints (_bedrock_converse_messages_pt and convert_to_anthropic_tool_result):

Bedrock:

  • test_bedrock_tool_message_openai_file_pdf_becomes_document — new behavior.
  • test_bedrock_tool_message_image_url_pdf_data_uri_becomes_document — regression for the silent-drop bug.
  • test_bedrock_tool_message_image_url_png_still_becomes_image — locks in existing image-block behavior.

Anthropic:

  • test_anthropic_tool_result_openai_file_pdf_becomes_document
  • test_anthropic_tool_result_image_url_pdf_data_uri_becomes_document
  • test_anthropic_tool_result_image_url_png_still_becomes_image

Each new-behavior test was developed via strict red→green TDD — verified failing against the unpatched code first, then implemented minimal code to pass.

… path

OpenAI Chat Completions `{type: "file", file: {file_data: "data:application/pdf;..."}}`
content blocks inside tool messages were silently dropped when translated to
Bedrock Converse and direct Anthropic. Additionally, PDFs sent via `image_url`
data URIs were either dropped (Bedrock) or wrapped as `type: "image"` and
rejected by the API (Anthropic).

- _convert_to_bedrock_tool_call_result: add `type: "file"` branch; pass through
  document blocks produced by BedrockImageProcessor for PDF `image_url` URIs.
  Single choke point covers both sync and async converse paths.
- convert_to_anthropic_tool_result: add `type: "file"` branch delegating to
  `anthropic_process_openai_file_message`; branch `image_url` on data-URI mime
  type so non-image mimes route through the file helper to produce document
  blocks.
- AnthropicMessagesToolResultParam.content union extended to accept
  `AnthropicMessagesDocumentParam` alongside text and image.
- Add 6 tests (3 Bedrock + 3 Anthropic) covering file-PDF, image_url-PDF, and
  image_url-PNG regression.

Fixes BerriAI#24641
Supersedes BerriAI#24646 with an OpenAI-native approach and test coverage.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Apr 28, 2026

CLA assistant check
All committers have signed the CLA.

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 28, 2026

Codecov Report

❌ Patch coverage is 94.28571% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...llm/litellm_core_utils/prompt_templates/factory.py 94.28% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

Duplicate the three Bedrock and three Anthropic tool-result tests into
tests/test_litellm/ so they're picked up by `make test-unit` (and its
coverage report). The originals in tests/llm_translation/ stay — they
run under integration and remain the canonical translation-suite
regression cases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bojun-Vvibe added a commit to Bojun-Vvibe/oss-contributions that referenced this pull request Apr 28, 2026
…correctness fixes

- openai/codex#20046 lock InterAgentCommunication commentary-phase contract
- BerriAI/litellm#26710 file content on Bedrock+Anthropic tool-result path
- google-gemini/gemini-cli#26136 disconnect extension-backed MCP clients via clientKey
@minznerjosh
Copy link
Copy Markdown
Contributor Author

@greptileai

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Apr 28, 2026

Greptile Summary

This PR fixes two silent-drop bugs in the OpenAI→Bedrock and OpenAI→Anthropic tool-result converters: type: \"file\" content blocks were entirely unhandled, and Bedrock's image_url branch discarded processor-returned document blocks. The fix adds symmetric file branches to both converters (mirroring the existing user-message paths) and a document-passthrough in the Bedrock image_url branch. Six well-structured tests with no real network calls are added in the correct tests/test_litellm/ directories, superseding the previously duplicated copies in tests/llm_translation/.

Confidence Score: 5/5

Safe to merge — fixes are isolated to tool-result translation helpers with good test coverage and no regressions in existing behavior.

No P0/P1 issues found. The one P2 (container_upload type gap on the Anthropic path) is an unlikely edge case with a clear, non-silent failure mode. Prior review threads were fully addressed in the head commit. Tests are comprehensive and mock-only.

No files require special attention.

Important Files Changed

Filename Overview
litellm/litellm_core_utils/prompt_templates/factory.py Core fix: adds type: "file" branch and document-passthrough to both Bedrock and Anthropic tool-result converters; logic is sound with one minor type-safety gap for container_upload file_ids on the Anthropic path.
litellm/types/llms/anthropic.py Extends AnthropicMessagesToolResultParam.content union to include AnthropicMessagesDocumentParam, correctly reflecting what the Anthropic API accepts in tool_result content.
tests/test_litellm/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_factory.py Adds 5 Anthropic tool-result tests (PDF via file, PDF via image_url, PNG regression, unsupported-mime stays on image path, text/plain document routing); all use mocked/inline data, no real network calls.
tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py Adds 5 Bedrock tool-result tests (PDF via file, PDF via image_url, file_id mock, both-None raises, PNG regression); the file_id test correctly patches BedrockImageProcessor to avoid network calls.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Tool Message content block] --> B{content type?}
    B -->|text| C[Append text block]
    B -->|image_url| D{Is document data URI?\napplication/pdf or text/plain}
    B -->|file| E[Get file_data / file_id]
    
    D -->|Yes| F[anthropic_process_openai_file_message\nsynth file object]
    D -->|No| G[create_anthropic_image_param\nimage block]
    
    E --> H{both None?}
    H -->|Yes - Bedrock| I[raise BadRequestError]
    H -->|Yes - Anthropic| J[raise Exception via helper]
    H -->|No| K[anthropic_process_openai_file_message\nor BedrockImageProcessor.process_image_sync]
    
    F --> L[Append document block to tool_result]
    G --> M[Append image block to tool_result]
    K --> N{block type?}
    N -->|document| L
    N -->|image| M
    N -->|container_upload| O[⚠️ Appended but invalid\nin tool_result — API rejects]
Loading

Reviews (2): Last reviewed commit: "address Greptile review feedback on tool..." | Re-trigger Greptile

Comment thread litellm/litellm_core_utils/prompt_templates/factory.py Outdated
Comment thread litellm/litellm_core_utils/prompt_templates/factory.py Outdated
Comment thread tests/llm_translation/test_anthropic_completion.py Outdated
- Tighten _is_anthropic_document_data_uri to match the mimes Anthropic
  actually accepts as base64 `document` source ({application/pdf,
  text/plain}). The previous application/* + text/* prefix match would
  route e.g. data:application/json URIs through the document path,
  producing blocks the Anthropic API rejects. Unsupported mimes now
  stay on the existing image code path (same failure mode as before the
  fix — no regression, just stops introducing a new one).

- On the Bedrock tool-result `type: "file"` branch, accept either
  file_data or file_id and raise BadRequestError on both-None, mirroring
  the user-message _process_file_message pattern. Previously a file
  block with only file_id was silently dropped.

- Consolidate the six new PDF tool-result tests under tests/test_litellm/
  only (the PR template's required location and where the unit-test CI
  workflow runs with coverage). The duplicate copies under
  tests/llm_translation/ added drift risk with no additional coverage.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@minznerjosh minznerjosh marked this pull request as ready for review April 28, 2026 20:59
@Sameerlite Sameerlite changed the base branch from litellm_internal_staging to litellm_oss_staging April 29, 2026 07:29
@Sameerlite Sameerlite merged commit d8d1444 into BerriAI:litellm_oss_staging Apr 29, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bedrock: document content blocks silently dropped during message conversion

3 participants