Skip to content

Misc. bug: Gemma4 tool calling leaves unexpected tokens in tool calls #21316

@NickM-27

Description

@NickM-27

Name and Version

llama-server 5803c8d

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

srv          load: spawning server instance with args:
srv          load:   /app/llama-server
srv          load:   --host
srv          load:   127.0.0.1
srv          load:   --jinja
srv          load:   --no-mmproj-auto
srv          load:   --port
srv          load:   39547
srv          load:   --slot-save-path
srv          load:   /config/slots
srv          load:   --temperature
srv          load:   1.0
srv          load:   --top-k
srv          load:   64
srv          load:   --top-p
srv          load:   0.95
srv          load:   --webui
srv          load:   --webui-config-file
srv          load:   /config/webui-config.json
srv          load:   --webui-mcp-proxy
srv          load:   --alias
srv          load:   unsloth/gemma-4-26B-A4B-GGUF:Q5_K_XL
srv          load:   --batch-size
srv          load:   2048
srv          load:   --ctx-size
srv          load:   16000
srv          load:   --checkpoint-every-n-tokens
srv          load:   2048
srv          load:   --cache-type-k
srv          load:   q8_0
srv          load:   --cache-type-v
srv          load:   q8_0
srv          load:   --flash-attn
srv          load:   on
srv          load:   --hf-repo
srv          load:   unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q5_K_XL
srv          load:   --n-gpu-layers
srv          load:   -1
srv          load:   --parallel
srv          load:   1
srv          load:   --ubatch-size
srv          load:   2048

Problem description & steps to reproduce

Using Gemma4 and tool calling I get the following response back. It is not 100% clear to me if this is a parser issue and these should be removed or if we just need to filter them out

{
  "area": "Main Bedroom",
  "domain": "[<|\"|>light<|\"|>]",
  "name": "Bedroom Lights"
}

First Bad Commit

No response

Relevant log output

Logs
{
  "area": "Main Bedroom",
  "domain": "[<|\"|>light<|\"|>]",
  "name": "Bedroom Lights"
}

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions