[Bug]: Anthropic tool calls with images is broken

What happened?

When using the new tool call functionality of anthropic and you add an image to your message (according to the openai's message format) the message conversion to anthropics format seems to break. This was not the case when running the same code before the recent anthropic tool change.

Here is what litellm log shows what it sends to anthropic (the input was just a single message with one image added):

POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/messages \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json'  -H 'anthropic-beta: tools-2024-04-04' \
-d '{'model': 'claude-3-opus-20240229', 'messages': [
{
  'model': 'claude-3-opus-20240229',
  'messages': [
    {
      'role': 'user',
      'content': [
        {
          'type': 'text',
          'text': [
            {
              'type': 'text',
              'text': [
                {
                  'type': 'text',
                  'text': 'Fact two: Look at the image., Question: What animal is in this image in one word?'
                },
                {
                  'type': 'image_url',
                  'image_url': {
                    'url': 'data:image/jpeg;base64,IMAGE_HERE',
                    'detail': 'high'
                  }
                }
              ]
            },
            {
              'type': 'image_url',
              'image_url': {
                'url': 'data:image/jpeg;base64,/IMAGE_HERE',
                'detail': 'high'
              }
            }
          ]
        },
        {
          'type': 'image',
          'source': {
            'type': 'base64',
            'media_type': 'image/jpeg',
            'data': 'IMAGEHERE'
          }
        }
      ]
    }
  ],
  'temperature': 0.0010000011,
  'top_p': 1.0,
  'max_tokens': 110,
  'tools': [
    {
      'name': 'answer',
      'description': "Answer the assistant's question exactly as requested.",
      'input_schema': {
        'type': 'object',
        'properties': {
          'answer_text': {
            'type': 'string',
            'description': 'the content of the answer'
          }
        },
        'required': [
          'answer_text'
        ]
      }
    }
  ],
'system': 'Fact one: There are 3 books in the library.'
}

Relevant log output

No response

Twitter / LinkedIn details

No response