Skip to content

[Bugfix] Gemma 4 chat template crash with missing tool name and tool id#42188

Merged
chaunceyjiang merged 1 commit into
vllm-project:mainfrom
yzong-rh:yzong-rh/gemma4-chat-template-tool-name
May 11, 2026
Merged

[Bugfix] Gemma 4 chat template crash with missing tool name and tool id#42188
chaunceyjiang merged 1 commit into
vllm-project:mainfrom
yzong-rh:yzong-rh/gemma4-chat-template-tool-name

Conversation

@yzong-rh

@yzong-rh yzong-rh commented May 9, 2026

Copy link
Copy Markdown
Contributor

Purpose

Gemma 4's chat template does not handle missing names in tool results correctly. The intended behavior is to default to "unknown" but it's kept as a None value instead. As a result, later in format_tool_response_block, the None tool name triggers a hard crash.

This bug manifests on the legacy tool_responses path when tool name is missing. It manifests on the OpenAI Chat Completions path when the following message does not contain a name and does not contain a tool_call_id (or if the tool_call_id does not match the tool call).

This defect is present in both examples/tool_chat_template_gemma4.jinja and default chat template from HF.

Test Plan

repro.py

import argparse
from pathlib import Path

from transformers import AutoTokenizer


parser = argparse.ArgumentParser()
parser.add_argument(
    "--chat-template",
    type=Path,
    help="Path to a Jinja chat template override.",
)
args = parser.parse_args()

apply_chat_template_kwargs = {}
if args.chat_template is not None:
    apply_chat_template_kwargs["chat_template"] = (
        args.chat_template.expanduser().read_text(encoding="utf-8"))

tok = AutoTokenizer.from_pretrained(
    "google/gemma-4-31B-it",
)
messages = [
    {"role": "user", "content": "hi"},
    {
        "role": "assistant",
        "tool_calls": [
            {
                "id": "call_B",
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "arguments": {"city": "SF"},
                },
            }
        ],
    },
    {
        "role": "tool",
        "tool_call_id": "call_A",  # missing or incorrect tool_call_id
        "content": "sunny",
    },
]
print(
    tok.apply_chat_template(
        messages,
        tokenize=False,
        **apply_chat_template_kwargs,
    ))
pytest tests/renderers/test_gemma4_chat_template.py
pytest tests/tool_parsers/test_gemma4_tool_parser.py
pytest tests/tool_use/test_gemma4_responses_adjust_request.py

Test Result

repro.py --chat-template examples/tool_chat_template_gemma4.jinja

crashes with TypeError: can only concatenate str (not "NoneType") to str before this fix.

Unit tests pass

Made with Cursor

Signed-off-by: Yifan <yzong@redhat.com>
@mergify

mergify Bot commented May 9, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--42188.org.readthedocs.build/en/42188/

@mergify mergify Bot added documentation Improvements or additions to documentation bug Something isn't working labels May 9, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the examples/tool_chat_template_gemma4.jinja template to use the boolean=True flag with the default filter when resolving tool names. This change ensures that the 'unknown' fallback is used for empty strings as well as undefined values. I have no feedback to provide as there were no review comments to evaluate.

@yzong-rh

yzong-rh commented May 9, 2026

Copy link
Copy Markdown
Contributor Author

cc @bbrowning

Happy to add unit tests but this is really an edge case found while debugging #42189

@yzong-rh yzong-rh marked this pull request as ready for review May 9, 2026 23:49

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@chaunceyjiang chaunceyjiang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chaunceyjiang chaunceyjiang added the ready ONLY add when PR is ready to merge/full CI is needed label May 11, 2026
@chaunceyjiang chaunceyjiang enabled auto-merge (squash) May 11, 2026 02:18
@chaunceyjiang chaunceyjiang merged commit b168752 into vllm-project:main May 11, 2026
20 checks passed
yiliu30 pushed a commit to yiliu30/vllm-fork that referenced this pull request May 11, 2026
@bbrowning

Copy link
Copy Markdown
Collaborator

@yzong-rh Do these changes need to make their way to the upstream Gemma4 official chat templates in the HF repo? If so, we'll need to coordinate with @lucianommartins alongside other upstream template changes being reviewed.

@yzong-rh

Copy link
Copy Markdown
Contributor Author

@yzong-rh Do these changes need to make their way to the upstream Gemma4 official chat templates in the HF repo?

I think so... Is there a channel we discuss and coordinate this?

weifang231 pushed a commit to weifang231/eb-vllm that referenced this pull request May 13, 2026
mfylcek pushed a commit to mfylcek/vllm that referenced this pull request May 19, 2026
jhu960213 pushed a commit to jhu960213/vllm that referenced this pull request May 20, 2026
mvanhorn pushed a commit to mvanhorn/vllm that referenced this pull request Jun 4, 2026
…id (vllm-project#42188)

Signed-off-by: Yifan <yzong@redhat.com>
Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
knight0528 pushed a commit to knight0528/vllm that referenced this pull request Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants