this PR supercedes #43678

glm_4_7 (GLM-4.7 / GLM-5) also works well in our internal test.

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @chaunceyjiang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Documentation preview: https://vllm--45003.org.readthedocs.build/en/45003/

I tested this with GPT-OSS 120B using:

VLLM_ENFORCE_STRICT_TOOL_CALLING=true
--enable-auto-tool-choice
--tool-call-parser openai
--reasoning-parser openai_gptoss

I noticed the structural tag is not applied in the live GPT-OSS chat render path for tool_choice="required". The request for gpt goes through the _make_request_with_harmony() path which bypasses the generic preprocess_chat(... parser=self.parser ...) path where the structural_tag gets applied

And, if I force the generated Harmony structural tag, it leaks raw Harmony markers into assistant content (reproducible when using ambiguous prompts).
For prompt "Hi, how are you?" with tool_choice="required":

message.content = <|channel|>commentary to=functions.get_weather<|constrain|>json<|message|>{...}
tool_calls = []
finish_reason = stop

Also, the generated Harmony tag for tool_choice="required" contains: "at_least_one": false so even if wired into the live path, it does not appear to enforce required-tool semantics. Could you look into this? Personally, I think using json constraints for gpt oss is a lot simpler since it avoids the overall harmony leak but feel free to do your own deep-dive @chaunceyjiang @cjackal

@ankrovv Yeah, GPT-OSS isn't supported yet. As you mentioned, it currently goes through a separate code path.

We're in the process of unifying the Harmony and non-Harmony paths, and once that's done, we'll integrate it with structural_tags harmony tags as well.

Overall, using structural tags has been quite effective at improving tool-calling accuracy and reliability.

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @chaunceyjiang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Can we resolve the merge conflict so that we can get this one in? Thanks for the great work from everyone!

@yzong-rh Thanks for catching this so carefully.

Regarding the MiniMax issue, I believe the root cause is in xgrammar's built-in structural_tag implementation (https://github.com/mlc-ai/xgrammar/blob/main/python/xgrammar/builtin_structural_tag.py#L1455-L1456). I'll discuss it with their team later. In the meantime, I've reimplemented that part on our side, and the issue has been resolved.

As for the OpenAI/Harmony issue, the previous Harmony path bypassed structural_tag entirely. Since I hadn't tested that path yet, and because of your refactoring there, I haven't had a chance to verify the behavior with the new implementation. For now, I've temporarily disabled the OpenAI parser handling until I can properly test it.

I believe the root cause is in xgrammar's built-in structural_tag implementation (https://github.com/mlc-ai/xgrammar/blob/main/python/xgrammar/builtin_structural_tag.py#L1455-L1456). I'll discuss it with their team later.

Great, thank you. I went through the structural tags in XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS. minimax was indeed the only implementation with that behavior.

A remaining risk is kimi, whose is_reasoning_end allows "implicit reasoning end via tool call section".

When that triggers, the start of a tool section is already generated. xgrammar's tags would force the start of tool section to be generated again. But this only occurs if kimi "forgot" to close </think>. So not an xgrammar problem, and not a blocker.

For now, I've temporarily disabled the OpenAI parser handling until I can properly test it.

Sg. There might be some challenges due to the harmony channel format. Happy to look into it if you don't have the bandwidth.

Since this merged I'm seeing Qwen 3.6 models getting stuck in infinite whitespace generation loops regularly in relatively simple tool calling scenarios with our out of the box setup. It's easily triggered with things like BFCL multi_turn_base, resulting in extremely long generation times and timeouts while the model generates thousands of newline tokens per turn.

We've flip this on by default for a lot of models now in tool calling scenarios, and at least in some cases this has made things worse. We may need a more selective testing and case-by-case enabling of this.