Test before:

After:

/gemini summary

Summary of Changes

Hello! Gemini here, providing a summary of this pull request. This PR focuses on addressing two key issues within the DeepSeekV3Detector component. Firstly, it corrects the assignment of the tool_index for ToolCallItems during streaming, ensuring it reflects the sequential position of the tool call in the response rather than its index in the initial tool list provided in the request. Secondly, it enhances the detector's ability to correctly parse and handle multiple consecutive tool calls within a single streaming response, which was previously causing subsequent calls to be misinterpreted as normal text. The changes involve updating the streaming parsing logic to maintain state across tool calls, manage the internal buffer more intelligently, and correctly identify and index each tool call as it streams.

Highlights

• Fix incorrect tool_index: The tool_index assigned to streamed ToolCallItems now correctly represents the order of the tool call in the model's response, rather than the index of the tool definition in the initial request.
• Support multi-tool call streaming: The detector can now correctly parse and yield multiple tool calls that appear consecutively in the streaming output, even if subsequent calls lack the initial bot_token prefix.
• Improved buffer management: The internal buffer logic has been updated to preserve content that might contain subsequent tool calls after one is completed, preventing data loss and enabling multi-tool call detection.
• Maintain adapter state: Logic was added to maintain state arrays (prev_tool_call_arr, streamed_args_for_tool) compatible with the adapter.py requirements for handling streaming tool calls.

Changelog

• python/sglang/srt/function_call/deepseekv3_detector.py
  • Added self.current_tool_id to track the index of the current tool call being processed during streaming.
  • Modified the buffer clearing condition in parse_streaming_increment to keep the buffer if a tool call start token (<｜tool call begin｜>) is present, allowing detection of subsequent tool calls without the bot_token.
  • Initialized state tracking arrays (prev_tool_call_arr, streamed_args_for_tool) and current_tool_id upon detecting the first tool call.
  • Ensured state tracking arrays are dynamically sized to accommodate multiple tool calls.
  • Changed tool_index assignment in ToolCallItem creation to use self.current_tool_id.
  • Stored initial tool call name and updated arguments in self.prev_tool_call_arr for adapter compatibility.
  • Appended streamed argument increments to self.streamed_args_for_tool.
  • Updated buffer management upon completion of a tool call to remove only the completed call's content, preserving any subsequent text (potential next tool call).
  • Incremented self.current_tool_id after a tool call is fully parsed.

Activity

• CatherineSue posted screenshots demonstrating the behavior before and after the changes.
• zhyncs requested this summary using the /gemini summary command.

/gemini review