common : merge qwen3-coder and nemotron nano 3 parsers by aldehir · Pull Request #19765 · ggml-org/llama.cpp

aldehir · 2026-02-20T18:29:09Z

Users are experiencing several issues with Qwen3-Coder-Next. Until #18675 is merged in, this PR serves as a stop-gap by replacing the existing Qwen3-Coder parsing with the Nemotron Nano 3 PEG parsing variant already present.

This PR also adds parallel tool calling and fixes JSON schema support.

fixes #19382
fixes #19430
fixes #19304
supersedes #19503 and #19753

pwilkin

Can you please add some tests on schema parameters, especially an array (such as a "todolist" tool)?

pwilkin · 2026-02-20T19:06:13Z

I'll try to test it today with OpenCode to see how it works.

bfroemel · 2026-02-20T20:26:37Z

Can we also add the fix for #19513 ?

@pwilkin, I believe it was just removing the trim_trailing_space() filters from reasoning (L30) and content (L34) here,?

https://github.com/aldehir/llama.cpp/blob/e811d2263fc10d1b1d9ed036708d6c9b6f798a7f/common/chat-peg-parser.cpp#L25-L36

aldehir · 2026-02-20T20:50:16Z

Can we also add the fix for #19513 ?

As I understand it, the model generates the following tokens:

llama.cpp > curl http://localhost:8080/tokenize -d '{"content": "call:\n\n<tool_call>", "parse_special": true, "with_pieces": true}'
{"tokens":[{"id":6659,"piece":"call"},{"id":1447,"piece":":\n\n"},{"id":151657,"piece":"<tool_call>"}]}

But when we parse it, we strip \n (the template always adds a \n before <tool_call>), and we feed it back as:

llama.cpp > curl http://localhost:8080/tokenize -d '{"content": "call:\n<tool_call>", "parse_special": true, "with_pieces": true}'
{"tokens":[{"id":6659,"piece":"call"},{"id":510,"piece":":\n"},{"id":151657,"piece":"<tool_call>"}]}

Which causes issues? This makes sense to me. It's a bit annoying because ultimately it depends on the model + template. In this case, we need to strip a single \n because it is added back in the chat template.

pwilkin · 2026-02-20T21:30:44Z

@aldehir Oh, this is a fun one :)

This causes issues because in the tokenizer, :\n\n is a token. A separate one from ":". And apparently that's a trick used to train the model toolcalling - if the previous token was the ":\n\n", the next token is more likely to be the tool call start.

When you pass this back as ":\n", this is no longer the same token. Therefore, the model at some point gets confused and stops doing tool calls if enough of those bad historical calls are done because it doesn't know what to do after a token it wasn't trained to call tools on.

aldehir · 2026-02-20T21:34:33Z

This causes issues because in the tokenizer, :\n\n is a token. A separate one from ":". And apparently that's a trick used to train the model toolcalling - if the previous token was the ":\n\n", the next token is more likely to be the tool call start.

Upon further inspection, I am struggling to see the issue.

The chat template will strip the message content and append a \n.

        {%- if message.content is defined and message.content is string and message.content | trim | length > 0 %}
            {{- '\n' + message.content | trim + '\n' }}
        {%- endif %}

https://huggingface.co/Qwen/Qwen3-Coder-Next/blob/main/chat_template.jinja

Then, it prepends a \n for tool calls

            {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }

Assuming the end of the content is :\n\n, even if we stripped trailing whitespace, the template will add back the \n\n. Due to the trimming in the template, it actually doesn't matter if we strip or not.

bfroemel · 2026-02-20T21:38:15Z

@aldehir Are you sure that the template adds the newline again?

I believe the tool call preamble (as it occurs with codex use/potentially related to the responses API) is rendered as a "normal" assistant message:

        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>\n' }}

Anyway, I can confirm that currently it's an issue with this PR, @pwilkin's fix (#18675 (comment)) lead to the apparently expected format and the issue was gone.

bfroemel · 2026-02-20T21:41:53Z

oh, remembering - I fell for the same "trap" :) #18675 (comment) (marked it as off-topic, contains the actually rendered prompt missing the required newlines)

aldehir · 2026-02-20T21:51:27Z

oh, remembering - I fell for the same "trap" :) #18675 (comment) (marked it as off-topic, contains the actually rendered prompt missing the required newlines)

I see. It's a bit weird that the preamble would be its own message. This means the client needs to perform another request to trigger a tool call, which doesn't quite align with the idea that retaining :\n\n yields uninterrupted sessions. Unless the Responses API is incorrectly formatting the input to the template.

aldehir · 2026-02-20T21:56:39Z

Sure enough, the Responses API is the culprit:
https://github.com/ggml-org/llama.cpp/blob/b908baf1825b1a89afef87b09e22c32af2ca6548/tools/server/server-common.cpp#L1217C1-L1241C50

Rather than generate a single message with content + tool_call, it generates two messages. I think this needs to be fixed instead, because feeding two separate messages will undoubtedly differ from how the model was trained.

bfroemel · 2026-02-20T22:04:52Z

Here just for reference a snippet from a responses API request as generated by codex (this example misses the 2 newlines at the end of tool preamble message), input array elements:

    {
      "type": "message",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "Let me search for the main agent loop or turn handling:"
        }
      ]
    },
    {
      "type": "function_call",
      "name": "exec_command",
      "arguments": "{\"cmd\":\"rg --files /home/b/work/codex-new/codex-rs/core/src | xargs -I{} rg -l \\\"turn\\\\|loop\\\" {} 2>/dev/null | head -20\",\"justification\":\"Find files related to turn/loop\"}",
      "call_id": "fc_o1wk8CXzBhlun0u8esiYzJjZuz58M5Iz"
    },
    {
      "type": "function_call_output",
      "call_id": "fc_o1wk8CXzBhlun0u8esiYzJjZuz58M5Iz",
      "output": "Chunk ID: 7054eb\nWall time: 0.4461 seconds\nProcess exited with code 0\nOriginal token count: 0\nOutput:\n"
    },
    {
      "type": "function_call",
      "name": "exec_command",
      "arguments": "{\"cmd\":\"find /home/b/work/codex-new/codex-rs/core -type f -name \\\"*.rs\\\" | head -50\",\"justification\":\"List core module files\"}",
      "call_id": "fc_VP2ElL5yP9Y2knTm3gTh79VGOmn8s39e"
    },

Not seeing a case where content + tool call could be currently possible with https://github.com/openingnow/llama.cpp/blob/292f6908cdc6abb5c38581e34fa141973e5aba82/tools/server/server-common.cpp#L1072

I believe in the responses API there is no way to have type "message" contain also "function_call" elements. https://developers.openai.com/api/reference/resources/responses#(resource)%20responses%20%3E%20(model)%20response_input_item%20%3E%20(schema)

aldehir · 2026-02-20T22:07:20Z

Yes, I believe this is called a "leaky abstraction" 😊.

We should add a post step in the conversion that merges consecutive assistant messages. That seems like the most correction solution.

pwilkin · 2026-02-20T22:22:19Z

Aight, merging this one and we'll do a followup on the Responses problem.

aldehir · 2026-02-20T22:23:38Z

@bfroemel expect to see a PR for the Responses API here shortly. If you don't mind, I'd like to enlist your services to help test it out.

bfroemel · 2026-02-20T22:23:54Z

Hmm, so the model generates content output + tool call(s) - everything in a single generation; so single response. codex (as a responses API client) processes that response and builds the conversation in the input array for the next request with tool call responses. There it ends up as several elements (output_text message, tool calls, tool responses).

IMO that's fine and wouldn't immediately require to do post processing (trying to get multiple responses input elements into single chat completions elements), as long as it ends up as the expected prompt for the model (I looked at the rendered prompt - it's imo indistinguishable)? but ofc fine with any other solution that might be more clean/maintainable - especially if @pwilkin's fix could be problematic for other models.

/edit: adding a rendered prompt snippet (with the required newlines)

<|im_start|>assistant
Let me look at the parse_turn_item function:

<|im_end|>
<|im_start|>assistant
<tool_call>
<function=exec_command>
<parameter=cmd>
grep -n "fn parse_turn_item" /home/b/work/codex-new/codex-rs/core/src/*.rs
</parameter>
<parameter=justification>
Find parse_turn_item function
</parameter>
</function>
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
Chunk ID: d74fd7
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 32
Output:
/home/b/work/codex-new/codex-rs/core/src/event_mapping.rs:96:pub fn parse_turn_item(item: &ResponseItem) -> Option<TurnItem> {

</tool_response><|im_end|>
<|im_start|>assistant
<tool_call>
<function=exec_command>
<parameter=cmd>
sed -n '96,200p' /home/b/work/codex-new/codex-rs/core/src/event_mapping.rs
</parameter>
<parameter=justification>
Read parse_turn_item function
</parameter>
</function>
</tool_call><|im_end|>

aldehir · 2026-02-20T22:38:08Z

This is what I expect:

<|im_start|>assistant
Let me look at the parse_turn_item function:

<tool_call>
<function=exec_command>
<parameter=cmd>

The additional <|im_end|>...<|im_start|> will likely cause the model to in-context learn and not generate a tool call in a single generation.

* Fix tool call for Qwen3.5 Loosely based on mainline changes from: * ggml-org/llama.cpp#19635 * ggml-org/llama.cpp#19765 Also need to change the grammar to allow the model to make multiple tool calls in a row. This was likely broken for Qwen3 Coder prior to this commit. * Fix the grammar for the subsequent parameters after the first one

* common : migrate qwen3-coder to PEG parsing variant * cont : add JSON parameter test

* Fix tool call for Qwen3.5 Loosely based on mainline changes from: * ggml-org/llama.cpp#19635 * ggml-org/llama.cpp#19765 Also need to change the grammar to allow the model to make multiple tool calls in a row. This was likely broken for Qwen3 Coder prior to this commit. * Fix the grammar for the subsequent parameters after the first one

* Better estimate for max. nuber of compute nodes * Just in case server: fix crash from adaptive p (ikawrakow#1304) Co-authored-by: firecoperana <firecoperana> Fix tool call for Qwen3.5 (ikawrakow#1300) * Fix tool call for Qwen3.5 Loosely based on mainline changes from: * ggml-org/llama.cpp#19635 * ggml-org/llama.cpp#19765 Also need to change the grammar to allow the model to make multiple tool calls in a row. This was likely broken for Qwen3 Coder prior to this commit. * Fix the grammar for the subsequent parameters after the first one Graph parallel for Qwen3-Next (ikawrakow#1292) * WIP * This works, but is slower than split mode layer Fix llm_arch_is_hybrid (ikawrakow#1305) Fix max nodes (again) (ikawrakow#1306) Fix typo in merge-up-gate-experts argument (ikawrakow#1311) llama-quantize: --dry-run option (ikawrakow#1309) Slightly better graph parallel for Qwen3-Next (ikawrakow#1307) * Make sure we pick the reduced tensor from the right GPU * Minor Minor delta-net tweak (ikawrakow#1308) * Make sure we pick the reduced tensor from the right GPU * Minor * Minor delta-net tweak adaptive p: collect probability before logit bias (ikawrakow#1314) server: propagate task index to response objects for batch requests (ikawrakow#1303) When multiple prompts are sent in a single /v1/completions request, each response needs to carry the correct index so the client can match results to their corresponding prompts. The index field was not being set on partial responses, final responses, or embedding responses, causing batch results to all report index 0. Set res->index = slot.task->index in send_partial_response, send_final_response, and send_embedding. Generated with [Devin](https://cli.devin.ai/docs) Co-authored-by: Joshua Jolley <jjolley@clearwateranalytics.com> Co-authored-by: Devin <noreply@cognition.ai> Llama-quantize: Partial requant feature (ikawrakow#1313) * Partial Requant feature for llama-quantize - Inspired by the recently portcopied --dry-run feature. - Allows to partially requantize a split quantized .gguf by requantizing only the missing splits in the destination directory. - Works both for GGUF which are split tensors by tensors, or by group of several tensors (though this one is not very much tested beyond 2 tensors by split). - Vibe coded. * Create output directory if it doesn't exist in llama-quantize * Create output directory if it doesn't exist in gguf-split * Add exit when directory fails to be created on Windows * Use std::filesystem * cleanup Display the size of the tensors overriden during the tensor loading (ikawrakow#1318) * Display the size of the tensors overriden during the tensor loading Ex: `Tensor blk.60.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.60.ffn_up_exps.weight buffer type overriden to CPU` become `Tensor blk.60.ffn_up_exps.weight (size = 668467200 bytes) buffer type overriden to CPU Tensor blk.60.ffn_gate_exps.weight (size = 668467200 bytes) buffer type overriden to CPU` And pass in debug the later displayed size of the unnamed buffer overrides. Ex : `llm_load_tensors: CPU buffer size = XXX.XX MiB` That double display is cluttering the screen without being very informative. * change bytes display to MiB. Co-authored-by: Kawrakow <iwankawrakow@gmail.com> --------- Co-authored-by: Kawrakow <iwankawrakow@gmail.com> Fused delta-net (ikawrakow#1315) * Revive fused delta-net * Add command line argument for fused delta net * Simplify/improve CUDA delta-net * Add -fdn to llama-bench * More CUDA fused delta net optimizations * CPU optimizations * Much faster fused delta-net on the CPU It seems it is faster than the chunked implementation! * Change meaning of fdn from bool flag to threshold value * Use eps = 1e-6 * Give some nodes a name Fix KT quantization yet again (ikawrakow#1321) * Fix KT quantization yet again * Add same 1e-16f check for all quants in iqk_uantize.cpp * Fixes for k-quants * Also this one server: enable checkpoint for recurrent models (ikawrakow#1310) * server: enable checkpoint for recurrent models create checkpoint after cancel fix ban string and rm context during rewind add checkpoint interval only save recurrent cache * save checkpoint during pp --------- Co-authored-by: firecoperana <firecoperana> Faster quantization for MoE models with many experts (ikawrakow#1322) Fused delta net 2 (ikawrakow#1320) * Revive fused delta-net * Add command line argument for fused delta net * Simplify/improve CUDA delta-net * Add -fdn to llama-bench * More CUDA fused delta net optimizations * CPU optimizations * Much faster fused delta-net on the CPU It seems it is faster than the chunked implementation! * Change meaning of fdn from bool flag to threshold value * Use eps = 1e-6 * Give some nodes a name * Don't re-apply L2 norm - it has already been done * This seems quite a bit better * More tweaks * Restore per context buffer size log Not everybody uses models split in 2000 parts, and those who do, actually want to see the biffer sizes. iAdding support for dense Qwen-3.5 models (ikawrakow#1326) add directio to llama-bench

* common : migrate qwen3-coder to PEG parsing variant * cont : add JSON parameter test

common : migrate qwen3-coder to PEG parsing variant

e397b94

aldehir requested review from ggerganov and pwilkin as code owners February 20, 2026 18:29

github-actions bot added the testing Everything test related label Feb 20, 2026

aldehir mentioned this pull request Feb 20, 2026

common : fix XML tool calls with duplicate parameter keys #19753

Open

pwilkin requested changes Feb 20, 2026

View reviewed changes

cont : add JSON parameter test

e811d22

pwilkin approved these changes Feb 20, 2026

View reviewed changes

pwilkin merged commit 94b0200 into ggml-org:master Feb 20, 2026
78 checks passed

aldehir mentioned this pull request Feb 21, 2026

server : merge contiguous Responses input items into a single assistant message #19773

Merged

github-actions bot mentioned this pull request Feb 21, 2026

Reddit News Daily 2026-02-21 gitlawr/reddit-daily-news#162

Open

loci-dev mentioned this pull request Feb 21, 2026

UPSTREAM PR #19773: server : merge contiguous Responses input items into a single assistant message auroralabs-loci/llama.cpp#1196

Open

0x62ash mentioned this pull request Feb 21, 2026

Build 8070 regression: prefill ~75 tok/s instead of ~250 tok/s kyuz0/amd-strix-halo-toolboxes#58

Open

aldehir mentioned this pull request Feb 21, 2026

Eval bug: Frequent Crash occurs randomly #19679

Open

mudler mentioned this pull request Feb 21, 2026

chore: ⬆️ Update ggml-org/llama.cpp to ba3b9c8844aca35ecb40d31886686326f22d2214 mudler/LocalAI#8613

Merged

sayap mentioned this pull request Feb 22, 2026

Fix tool call for Qwen3.5 ikawrakow/ik_llama.cpp#1300

Merged

4 tasks

thomas-0816 mentioned this pull request Feb 22, 2026

Eval bug: Parallel function calling with gpt-oss-20b not working #19814

Closed

liparetejas pushed a commit to liparetejas/llama.cpp that referenced this pull request Feb 23, 2026

common : merge qwen3-coder and nemotron nano 3 parsers (ggml-org#19765)

e352b8f

* common : migrate qwen3-coder to PEG parsing variant * cont : add JSON parameter test

bartowski1182 pushed a commit to bartowski1182/llama.cpp that referenced this pull request Mar 2, 2026

common : merge qwen3-coder and nemotron nano 3 parsers (ggml-org#19765)

0a8dafb

* common : migrate qwen3-coder to PEG parsing variant * cont : add JSON parameter test

ArberSephirotheca pushed a commit to ArberSephirotheca/llama.cpp that referenced this pull request Mar 3, 2026

common : merge qwen3-coder and nemotron nano 3 parsers (ggml-org#19765)

ffd6687

* common : migrate qwen3-coder to PEG parsing variant * cont : add JSON parameter test

Conversation

aldehir commented Feb 20, 2026

Uh oh!

pwilkin left a comment

Choose a reason for hiding this comment

Uh oh!

pwilkin commented Feb 20, 2026

Uh oh!

bfroemel commented Feb 20, 2026

Uh oh!

aldehir commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Feb 20, 2026

Uh oh!

aldehir commented Feb 20, 2026

Uh oh!

bfroemel commented Feb 20, 2026

Uh oh!

bfroemel commented Feb 20, 2026

Uh oh!

aldehir commented Feb 20, 2026

Uh oh!

aldehir commented Feb 20, 2026

Uh oh!

bfroemel commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldehir commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Feb 20, 2026

Uh oh!

Uh oh!

aldehir commented Feb 20, 2026

Uh oh!

bfroemel commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldehir commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aldehir commented Feb 20, 2026 •

edited

Loading

bfroemel commented Feb 20, 2026 •

edited

Loading

aldehir commented Feb 20, 2026 •

edited

Loading

bfroemel commented Feb 20, 2026 •

edited

Loading