Does this mean we don’t need to write a parser anymore, and it will be automatically generated from the chat template?

Does this mean we don’t need to write a parser anymore, and it will be automatically generated from the chat template?

Yup, that's the gist of it.

This feels almost magical. How does it work? Does it detect common patterns in the rendered template output? What happens if the chat template requires additional arguments?

This feels almost magical. How does it work? Does it detect common patterns in the rendered template output? What happens if the chat template requires additional arguments?

Yeah, it does differential analysis - it prepares different inputs to the template and then tests the outputs, for example, by using a the same function signature with a different name you can identify where the function name goes, by using the same function with one and two parameters you can identify how parameters are passed etc. etc.

The nice thing is, I managed to squish it to just 2k lines of code (1k for analysis and 1k for helpers), so it's not even that bloated.

As for custom inputs - I assume standard inputs here and that's what most template makers try to adhere to anyway. If not, you end up with a custom handler like for Ministral - but as a followup I want to separate handlers from parsers (since passing extra params is much eaasier than handling an entire template from scratch) or even add autodetection for common custom keywords (we're going to have to support "reasoning" in addition to "reasoning_content" at some point because vLLM is moving to that).

This approach does not seem to work well for models like Kimi-K2-Thinking, which may generate tool calls inside the thinking block, while the chat template itself automatically closes the thinking block correctly. In other words, the model’s behavior does not seem to be fully aligned with the assumptions made by the chat template. Is that understanding correct? I noticed that you have removed all parsers.

Additionally, I am planning to add a new custom parser for MiroThinker. Its official chat template does not accurately reflect the rendering logic actually used in their benchmarks. Is there a recommended starting point for implementing such a parser for the new parsing architecture?

I've heard of those mysterious tool calls inside thinking blocks for K2-Thinking, but I've yet to know if they are an actual thing or if they are just an artifact of low quantization. To be honest, outside of the native provider, I haven't seen K2-Thinking implemented anywhere in a working fashion. The Chutes version that I tested quite a few times bugs out on tool calling extremely often.

I'm really skeptical of modifying anything based on hearsay and things "floating around". I remember the discussion here about interleaved thinking and I myself was convinced that meant models could have multiple <think> blocks until @aldehir pointed out that it's all a big misunderstanding and "interleaved thinking" is just the model having multiple message['assistant']['reasoning_content'] blocks next to message['assistant']['tool_call'] blocks. If I really see a working solution with open-sourced code anywhere that really demonstrates support for those thinking blocks, then sure, we can consider a special parser for K2-Thinking.

As for the Mirocode, I guess you're talking about adapting the Python code-based stuff they showed (the one that uses separate tags for MCP servers and code calling)? You can see how custom parsers are defined in chat.cpp, not much has changed besides the fact that since we use the PEG parser there's no longer a dedicated parse() and init() function and the entire parser is defined in the init. I'll probably separate the parsers into dedicated files soon.

All right, I've reached the "all tests passed" phase for test-chat, so I'm moving this officially out of draft. Will still test in practice but want to get all structural / architectural etc. issues out of the way in the meantime.

Glad than that here is still some real intelligence is involved. :D Hope its derailing took not too much time from you.

Nah, it's fine, I might add a case for end-of-generation as a terminating marker everywhere, not sure though whether having a response without content (just reasoning_content) would make much sense here either, it's a model error.

Just as an layman observer, could this be related to the issue where anthropic -> openai conversion is dropping reasoning blocks? #20090

Just as an layman observer, could this be related to the issue where anthropic -> openai conversion is dropping reasoning blocks? #20090

Most certainly could.

I've heard of those mysterious tool calls inside thinking blocks for K2-Thinking, but I've yet to know if they are an actual thing or if they are just an artifact of low quantization. To be honest, outside of the native provider, I haven't seen K2-Thinking implemented anywhere in a working fashion. The Chutes version that I tested quite a few times bugs out on tool calling extremely often.

Kimi K2 thinking is fixed in vllm as of a few days ago. Can confirm since I'm the one that fixed it. I also use opencode. vllm-project/vllm#33646

@koush oh, nice :) yeah, I implemented the tool-call-in-thinking-block once I saw it for myself in this branch as well :)

@l0nedigit could you provide some more details? (like what was the prompt?)

eh, I deleted it. Roo code was giving me an error stating it was unable to communicate with the api. So I tried a curl command:

curl http://localhost:8090/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "qwen3.5-27b-q8",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Say hello in one sentence."}
    ],
    "max_tokens": 50,
    "temperature": 0.7
  }'

That resulted in the error I pasted. When Iremoved temp/max_tokens, there was a response. Just now for some reason there is a very very long reponse time due to a lot of thinking for saying hello in one sentence. Prior, there was no thinking, which the 27b I thought was dense and a non-thinker by default 🤷 FWIW Roo code still replies back with "Unexpected API Response: The language model did not provide any assistant messages. This may indicate an issue with the API or the model's output."

TL;DR - comment deleted cause user error.

I'll test it more with Roo just to be sure.

I saw an error like that as well specifically on a non-streaming call, and I worked around it by disabling thinking. Streaming responses have been fine. Also qwen3.5 27b.

Hey thanks! @gtrak and @pwilkin

@gtrak have a beer on me this weekend ok? Thanks for the pro tip. Enabling streaming so far has produced better results.

I've been using this branch and 27b exclusively for opencode subagents over the last week and thinking has been great. That model at q4_k_s gives me 40 tok/s on a 4090 and it's doing all my code generation better than I expected. The only issue I have is an occasional crash and restart of llama.cpp if i try to use it in parallel, but I think there's another PR floating around that problem.

@pwilkin Hi, after this got merged, I'm getting the following error:

[58831] srv  update_slots: all slots are idle
[58831] srv          stop: all tasks already finished, no need to cancel
[58831] que    start_loop: waiting for new tasks
[58831] srv    operator(): got exception: {"error":{"code":500,"message":"Failed to parse input at pos 0: math,physics","type":"server_error"}}
[58831] srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 500
[58831] srv  log_server_r: request:  {"messages":[{"role":"system","content":"I am currently working on tagging dataset. I should tag all the problems accordingly using only tags in this list:\n   math - for when problems need specific mathematic solution\n   physics - when problem is a physics one\n   literature - for literature related problems\n\nI should also pay attention to the way I format the tags. My response should be only tags from the first category separated using comma.\nI should also pay attention to the way tags are desribed and apply them only when they match the correct criteria.\nI should also pay attention to the way tags are described and make sure all the relevant tags are applied based on their descriptions.\nI should not provide additional reasoning next to tag. I should provide only the correct list of tags as strings"},{"role":"user","content":"The game of NIM\n\nDetermine the best strategy for each player in the following two-player game. There\nare three piles, each of which contains some number of coins. Players alternate turns,\neach turn consisting of removing any (non-zero) number of coins from a single pile.\nThe goal is to be the person to remove the last coin(s)."}],"model":"Qwen3-1.7B-Q8_0.gguf","response_format":{"type":"json_schema","json_schema":{"schema":{"$defs":{"TagsEnum":{"enum":["math","physics","literature","<class '__main__.TagsEnum.Config'>"],"title":"TagsEnum","type":"string"}},"additionalProperties":true,"properties":{"tags":{"items":{"$ref":"#/$defs/TagsEnum"},"title":"Tags","type":"array"}},"required":["tags"],"title":"Problem","type":"object"},"name":"Problem","strict":true}},"stream":false}
[58831] srv  log_server_r: response: {"error":{"code":500,"message":"Failed to parse input at pos 0: math,physics","type":"server_error"}}
srv  server_http_: received response headers
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 500
srv  log_server_r: request:  {"messages":[{"role":"system","content":"I am currently working on tagging dataset. I should tag all the problems accordingly using only tags in this list:\n   math - for when problems need specific mathematic solution\n   physics - when problem is a physics one\n   literature - for literature related problems\n\nI should also pay attention to the way I format the tags. My response should be only tags from the first category separated using comma.\nI should also pay attention to the way tags are desribed and apply them only when they match the correct criteria.\nI should also pay attention to the way tags are described and make sure all the relevant tags are applied based on their descriptions.\nI should not provide additional reasoning next to tag. I should provide only the correct list of tags as strings"},{"role":"user","content":"The game of NIM\n\nDetermine the best strategy for each player in the following two-player game. There\nare three piles, each of which contains some number of coins. Players alternate turns,\neach turn consisting of removing any (non-zero) number of coins from a single pile.\nThe goal is to be the person to remove the last coin(s)."}],"model":"Qwen3-1.7B-Q8_0.gguf","response_format":{"type":"json_schema","json_schema":{"schema":{"$defs":{"TagsEnum":{"enum":["math","physics","literature","<class '__main__.TagsEnum.Config'>"],"title":"TagsEnum","type":"string"}},"additionalProperties":true,"properties":{"tags":{"items":{"$ref":"#/$defs/TagsEnum"},"title":"Tags","type":"array"}},"required":["tags"],"title":"Problem","type":"object"},"name":"Problem","strict":true}},"stream":false}
srv  log_server_r: response: 
srv    operator(): client request thread ended
srv    operator(): http: streamed chunk: {"error":{"code":500,"message":"Failed to parse input at pos 0: math,physics","type":"server_error"}}
srv    operator(): http: stream ended

when running server with json schema. I bisected and the issue was introduced in 566059a.

You should be able to reproduce with information in #20178 (that issue is unrelated, but it has all the scripts I used when I run into this).

Hello, after merging this branch, I also encountered an error like Failed to parse input at pos. Please see below for the full details. I am using the Qwen3-Coder model. Do you need any further information?

srv    operator(): got exception: {"error":{"code":500,"message":"Failed to parse input at pos 0: <tool_call>\n<function=list_directory>\n<parameter=path>\nAssets\n</parameter>\n<parameter=recursive>\n</parameter>\n</function>\n</tool_call>","type":"server_error"}}

Investigation on #20193 (Failed to parse input at pos ...) brought me here, likely this PR is the culprit.

Since this PR, I'm getting "No parser definition detected, assuming pure content parser." spammed in my terminal (with Qwen3.5 27B), for each generated token (at least with the server).