This discussion is a spin-off of the brief conversation which happened here: #20600 (comment)

Description of the issue

• During the processing of a single message, Open WebUI correctly uses the OpenAI API format, with interleaved role: "assistant" messages with tool calls and role: "tool" messages with the tool outputs
• However, once the user has sent another message, Open WebUI incorrectly merges all consecutive assistant and tool messages into a single assistant message

Here's an example of an initial message with multiple tool calls, as sent to litellm:

{
  "model": "Claude Sonnet 4.5",
  "tools": [
    ...
  ],
  "stream": true,
  "messages": [
    {
      "role": "system",
      "content": "[...]"
    },
    {
      "role": "user",
      "content": "How many customers do we have?"
    },
    {
      "role": "assistant",
      "content": "I'll help you find out how many customers we have. Let me first check what data is available.",
      "tool_calls": [
        {
          "id": "toolu_01PNNJfVpmqwa5p8RYGoA9UQ",
          "type": "function",
          "index": 0,
          "function": {
            "name": "list_views",
            "arguments": "{}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "content": "[...]",
      "tool_call_id": "toolu_01PNNJfVpmqwa5p8RYGoA9UQ"
    },
    {
      "role": "assistant",
      "content": "",
      "tool_calls": [
        {
          "id": "toolu_016StFZmbRmw9hCaotEV8Sug",
          "type": "function",
          "index": 0,
          "function": {
            "name": "get_view",
            "arguments": "{\"view_name\": \"chat__customers\"}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "content": "[...]",
      "tool_call_id": "toolu_016StFZmbRmw9hCaotEV8Sug"
    },
    {
      "role": "assistant",
      "content": "",
      "tool_calls": [
        {
          "id": "toolu_01AkAt7CtriPqU2YQWvic5qx",
          "type": "function",
          "index": 0,
          "function": {
            "name": "query",
            "arguments": "{\"measures\": [\"chat__customers.dim_customers_count\"]}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "content": "[...]",
      "tool_call_id": "toolu_01AkAt7CtriPqU2YQWvic5qx"
    }
  ],
  "max_tokens": 64000
}

And here's what it's turned into when the user sends a followup message:

{
  "model": "Claude Sonnet 4.5",
  "tools": [
    ...
  ],
  "stream": true,
  "messages": [
    {
      "role": "system",
      "content": ""
    },
    {
      "role": "user",
      "content": "How many customers do we have?"
    },
    {
      "role": "assistant",
      "content": "I'll help you find out how many customers we have. Let me first check what data is available.\n\""Available Views:\\n\\n- chat__customers\\n  Title: Chat Customers\\n  Customer-centric view for data analysis. Base entity is customers, so includes all customers even those who have never placed an order or subscription. ... (litellm_truncated skipped 29331 chars) ...   ]\\n}"\"\n\""Rows 1-1 of 1:\\n\\nchat__customers.dim_customers_count\\r\\n1234\\n"\"\nPetDirect has **1,234 customers** in total."
    },
    {
      "role": "user",
      "content": "How many new customers this year?"
    }
  ],
  "max_tokens": 64000
}

Some things to note:

• The interleaved assistant and tool messages have been condensed down to a single role: "assistant" message
• Only the tool output is included, the tool calls themselves are entirely missing
• The tool output is HTML escaped - this is being fixed here: fix: decode HTML entities in tool call results for multi-turn conversations #20755

Impact

The impact of this API misuse is debated.

Is it worth implementing it? Nobody has issues with the current behaviour, otherwise someone would have flagged it, and it requires significant work for - ... for no performance gain and not to fix a bug either.

Logically, it's not hard to see how injecting text into previous assistant messages would cause LLMs to produce similar text themselves in subsequent messages. After all, LLMs are fundamentally pattern matching engines, and the influence of "I did this before and it worked, I should keep doing it" will always be present.

My experience is that Claude 4.5 models start hallucinating injected tool outputs after 5-6 messages, while Gemini models seem more resistant.

The hallucination is just as easy to reproduce with the HTML escaping fix applied, so this IS NOT simply an issue with the escaping. In fact, while I don't have conclusive data to prove this, it seems that the LLM is slightly more easily swayed when the injected tool output is correctly unescaped.

I also tried alleviating the issue via the system prompt, explicitly explaining to the AI how the output injection works, what to expect, and how to behave. This was unsuccessful, and actually made the problem occur sooner.

Related Issues

I've previously observed a similar issue with preservation of thinking output. I'm currently unable to use Claude models with thinking enabled, because the thinking output is not correctly preserved for followup messages. The underlying cause is the same. I submitted a PR here which was rejected: #18478

Conclusion

This issue is pretty debilitating for use of Open WebUI in my org, as users have to be specifically trained in how to spot these hallucinations and how to deal with them.

I'm quite happy to get my hands dirty and help fix this issue myself on my employer's time, however I would want some guarantees that my PR wouldn't simply be rejected before I begin.

TL;DR: Open WebUI doesn't preserve the original OpenAI API structured messages across multiple sessions/user turns, leading to tool call hallucinations, and preventing use of thinking models