[Bugfix][Tool Parser] Fix Kimi-K2 tool call arguments when using tool_choice#44934
Open
ssam18 wants to merge 3 commits into
Open
[Bugfix][Tool Parser] Fix Kimi-K2 tool call arguments when using tool_choice#44934ssam18 wants to merge 3 commits into
ssam18 wants to merge 3 commits into
Conversation
…_choice Kimi-K2 emits tool calls as special-token markup and relies on its parser to recover the JSON arguments. When tool_choice names a function or is set to "required", the server took the standard JSON path and copied the raw model output straight into the arguments, so the returned arguments came back full of <|tool_calls_section_begin|> markup instead of the actual JSON. Setting supports_required_and_named = False routes those cases through the parser, the same way the GLM4-MoE parsers already handle this kind of markup. Fixes vllm-project#44906 Signed-off-by: Samaresh Kumar Singh <ssam3003@gmail.com> Signed-off-by: Samaresh Kumar Singh <ssam3003@gmail.com>
4 tasks
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
…ed-tool-choice # Conflicts: # vllm/tool_parsers/kimi_k2_tool_parser.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes #44906.
Kimi-K2 emits tool calls as special-token markup (
<|tool_calls_section_begin|> ... <|tool_call_argument_begin|> {...} <|tool_call_end|> ...) and relies on its tool parser to pull the JSON arguments back out. This works fine fortool_choice: "auto", but when the request names a function (or setstool_choice: "required") the server takes the standard JSON path and copies the raw model output straight into the call arguments. So the arguments come back full of the markup instead of the actual JSON, which is exactly what the issue reports.The Kimi parser was inheriting the default
supports_required_and_named = True, which tells the server the model already produces clean JSON for named and required tool choice. That is not true for this format. Setting it toFalseroutes named and required choices throughextract_tool_callsinstead, the same way the GLM4-MoE parsers already handle their markup-based format. The change is a one line flag onKimiK2ToolParserplus a regression test.Test Plan
pytest tests/tool_parsers/test_kimi_k2_tool_parser.pyTest Result
The new
test_named_and_required_routed_through_parserpasses and the existing Kimi-K2 parser tests still pass. I also confirmed against the exact payload from the issue that once the named and required paths go through the parser,extract_tool_callsreturnsname=get_weatherandarguments={"city": "Boston"}rather than the raw markup.