What happened?
When testing with tool message content having "image" type (e.g. a list of "text" and "image" content), it seems this image content is not passed to Anthropic. I tested this by asking a question about the image being sent.
I looked into this code and looks like tool image content is not supported yet?
|
def convert_to_anthropic_tool_result( |
|
message: Union[ChatCompletionToolMessage, ChatCompletionFunctionMessage] |
|
) -> AnthropicMessagesToolResultParam: |
|
""" |
|
OpenAI message with a tool result looks like: |
|
{ |
|
"tool_call_id": "tool_1", |
|
"role": "tool", |
|
"name": "get_current_weather", |
|
"content": "function result goes here", |
|
}, |
|
|
|
OpenAI message with a function call result looks like: |
|
{ |
|
"role": "function", |
|
"name": "get_current_weather", |
|
"content": "function result goes here", |
|
} |
|
""" |
|
|
|
""" |
|
Anthropic tool_results look like: |
|
{ |
|
"role": "user", |
|
"content": [ |
|
{ |
|
"type": "tool_result", |
|
"tool_use_id": "toolu_01A09q90qw90lq917835lq9", |
|
"content": "ConnectionError: the weather service API is not available (HTTP 500)", |
|
# "is_error": true |
|
} |
|
] |
|
} |
|
""" |
|
content_str: str = "" |
|
if isinstance(message["content"], str): |
|
content_str = message["content"] |
|
elif isinstance(message["content"], List): |
|
content_list = message["content"] |
|
for content in content_list: |
|
if content["type"] == "text": |
|
content_str += content["text"] |
|
|
|
anthropic_tool_result: Optional[AnthropicMessagesToolResultParam] = None |
|
## PROMPT CACHING CHECK ## |
|
cache_control = message.get("cache_control", None) |
|
if message["role"] == "tool": |
|
tool_message: ChatCompletionToolMessage = message |
|
tool_call_id: str = tool_message["tool_call_id"] |
|
|
|
# We can't determine from openai message format whether it's a successful or |
|
# error call result so default to the successful result template |
|
anthropic_tool_result = AnthropicMessagesToolResultParam( |
|
type="tool_result", tool_use_id=tool_call_id, content=content_str |
|
) |
|
|
|
if message["role"] == "function": |
|
function_message: ChatCompletionFunctionMessage = message |
|
tool_call_id = function_message.get("tool_call_id") or str(uuid.uuid4()) |
|
anthropic_tool_result = AnthropicMessagesToolResultParam( |
|
type="tool_result", tool_use_id=tool_call_id, content=content_str |
|
) |
|
|
|
if anthropic_tool_result is None: |
|
raise Exception(f"Unable to parse anthropic tool result for message: {message}") |
|
if cache_control is not None: |
|
anthropic_tool_result["cache_control"] = cache_control # type: ignore |
|
return anthropic_tool_result |
Relevant log output
Litellm request call: https://gist.github.com/ryanhoangt/52607ffe78bb6d3594a888ef52252dfc
Below is part of the message list:
{
'content': [
{
'type': 'text',
'text': "I'll help you navigate to the GitHub profile of ryanhoangt using the browser."
}],
'role': 'assistant',
'tool_calls': [
{
'index': 1,
'function': {
'arguments': '{"code": "goto(\'https://github.com/ryanhoangt\')"}',
'name': 'browser'
},
'id': 'tooluse_UxfOQT6jRq-SvoQ9La_1sA',
'type': 'function'
}
]
},
{
'content': [
{
'type': 'text',
'text': '[Current URL: https://github.com/ryanhoangt]\n[Focused element bid: 119]\n\n[Action executed successfully.]\n============== BEGIN accessibility tree ==============\nRootWebArea \'ryanhoangt (Ryan H. Tran) · GitHub\', focused\n\t[119] generic\n\t\t[120] generic\n\t\t\t[121] generic\n\t\t\t\t[122] link \'Skip to content\', clickable\n\t\t\t\t[123] generic\n\t\t\t\t\t[124] generic\n\t\t\t\t[135] ...... \'\'\n\t\t\t\t\t\t\t\t[3297] generic\n\t\t\t\t\t\t\t\t\t[3298] button \'Manage cookies\', clickable\n\t\t\t\t\t\t\t[3299] listitem \'\'\n\t\t\t\t\t\t\t\t[3300] generic\n\t\t\t\t\t\t\t\t\t[3301] button \'Do not share my personal information\', clickable\n\t\t\t[3302] generic\n\t\t[3314] generic, live=\'polite\', atomic, relevant=\'additions text\'\n\t\t[3315] generic, live=\'assertive\', atomic, relevant=\'additions text\'\n============== END accessibility tree ==============\nThe screenshot of the current page is shown below.\n'
},
{
'type': 'image_url',
'image_url': {
'url': 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABQAAAALQCAIAAABAH0oBAAEAAElEQVR4nOzdd5wTxRcA8PdmdpNcAQ6O3qsCIr0oRVRExY4ggvSmKD/sIkhREcQGIioICAjSBETAAiIWFOm9Kr33A64n2Z15vz82F8I1Djia974fPJPN7uzsJnuXtzPzBitUqgqMMcYY...5dYiqv4QZU3LKyomysjZWrki/wcFiijdWTvTPORdYuUCYYrZaav6l/CDGFY9QREaRVkl+fb/MylkYpBkE8pP8A8jJdm22aV4hAAAAAElFTkSuQmCC'
}
}
],
'role': 'tool',
'cache_control': {'type': 'ephemeral'},
'tool_call_id': 'tooluse_UxfOQT6jRq-SvoQ9La_1sA',
'name': 'browser'
}Twitter / LinkedIn details
No response
What happened?
When testing with tool message content having "image" type (e.g. a list of "text" and "image" content), it seems this image content is not passed to Anthropic. I tested this by asking a question about the image being sent.
I looked into this code and looks like tool image content is not supported yet?
litellm/litellm/llms/prompt_templates/factory.py
Lines 1128 to 1195 in 07223bd
Relevant log output
Litellm request call: https://gist.github.com/ryanhoangt/52607ffe78bb6d3594a888ef52252dfc
Below is part of the message list:
{ 'content': [ { 'type': 'text', 'text': "I'll help you navigate to the GitHub profile of ryanhoangt using the browser." }], 'role': 'assistant', 'tool_calls': [ { 'index': 1, 'function': { 'arguments': '{"code": "goto(\'https://github.com/ryanhoangt\')"}', 'name': 'browser' }, 'id': 'tooluse_UxfOQT6jRq-SvoQ9La_1sA', 'type': 'function' } ] }, { 'content': [ { 'type': 'text', 'text': '[Current URL: https://github.com/ryanhoangt]\n[Focused element bid: 119]\n\n[Action executed successfully.]\n============== BEGIN accessibility tree ==============\nRootWebArea \'ryanhoangt (Ryan H. Tran) · GitHub\', focused\n\t[119] generic\n\t\t[120] generic\n\t\t\t[121] generic\n\t\t\t\t[122] link \'Skip to content\', clickable\n\t\t\t\t[123] generic\n\t\t\t\t\t[124] generic\n\t\t\t\t[135] ...... \'\'\n\t\t\t\t\t\t\t\t[3297] generic\n\t\t\t\t\t\t\t\t\t[3298] button \'Manage cookies\', clickable\n\t\t\t\t\t\t\t[3299] listitem \'\'\n\t\t\t\t\t\t\t\t[3300] generic\n\t\t\t\t\t\t\t\t\t[3301] button \'Do not share my personal information\', clickable\n\t\t\t[3302] generic\n\t\t[3314] generic, live=\'polite\', atomic, relevant=\'additions text\'\n\t\t[3315] generic, live=\'assertive\', atomic, relevant=\'additions text\'\n============== END accessibility tree ==============\nThe screenshot of the current page is shown below.\n' }, { 'type': 'image_url', 'image_url': { 'url': 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABQAAAALQCAIAAABAH0oBAAEAAElEQVR4nOzdd5wTxRcA8PdmdpNcAQ6O3qsCIr0oRVRExY4ggvSmKD/sIkhREcQGIioICAjSBETAAiIWFOm9Kr33A64n2Z15vz82F8I1Djia974fPJPN7uzsJnuXtzPzBitUqgqMMcYY...5dYiqv4QZU3LKyomysjZWrki/wcFiijdWTvTPORdYuUCYYrZaav6l/CDGFY9QREaRVkl+fb/MylkYpBkE8pP8A8jJdm22aV4hAAAAAElFTkSuQmCC' } } ], 'role': 'tool', 'cache_control': {'type': 'ephemeral'}, 'tool_call_id': 'tooluse_UxfOQT6jRq-SvoQ9La_1sA', 'name': 'browser' }Twitter / LinkedIn details
No response