feat: circuit breaker for infinite tool-call loops#14059
Closed
gzsiang wants to merge 7 commits into
Closed
Conversation
Collaborator
|
Clean resubmission of #12632. Note: files_changed includes unrelated modules (ComfyUI, i18n scripts, browser_tool, file_tools, web_tools) — branch likely needs rebasing/cleanup. |
Add a circuit breaker mechanism in _invoke_tool() and the sequential execution path to detect and break infinite tool-call loops. When the same tool with identical arguments is called 3 consecutive times, the circuit breaker triggers and returns an error, forcing the LLM to try a different strategy. The breaker uses MD5 hashing of (tool_name, json_args) to create a stable signature, then checks if the last N calls all have the same signature. After triggering, the counter resets so future calls can succeed. - Added _consecutive_tool_calls and _circuit_breaker_threshold to AIAgent.__init__ - Added circuit breaker logic to _invoke_tool() (concurrent path) - Added circuit breaker logic to _execute_tool_calls_sequential() (sequential path) - All 22 existing tests pass
…shold - Change error message from Chinese to English for upstream compatibility - Increase default threshold from 3 to 5 to avoid false positives (legitimate retry scenarios exist, e.g. network jitter) - Fix indentation bug in sequential execution path - All 22 existing tests pass
Prevents the LLM from retrying different variations of a failing tool call. When a tool returns errors/empty results N times consecutively (regardless of argument changes), the counter triggers and forces a strategy change. - Added _tool_failure_count dict and _tool_failure_threshold (5) in __init__ - Added _check_tool_failure() method to detect failures and update counter - Added check calls in both concurrent and sequential execution paths - Each tool has independent counter; success resets counter to 0
…_breaker_threshold from 5 to 3 - Reduces _tool_failure_threshold from 5 to 3 - Improves circuit breaker error message with specific suggestions - Helps stop tool-loop failures earlier
When a tool call fails repeatedly, the compression model is asked for a suggestion. This gives the main model a fresh perspective to break out of tool loops. - Added _get_tool_suggestion() for retry threshold failures - Added _get_consecutive_suggestion() for identical-call circuit breaker - Suggestions are appended to the error message with a 💡 emoji - Gracefully falls back if compression model is unavailable
When a tool call fails repeatedly, suggestions are provided even without a compression model configured. - Added _heuristic_suggestion() — pattern matching on error messages - Added _heuristic_consecutive_suggestion() — 20+ tool-specific suggestions - Compression model is tried first, then falls back to heuristics - Works zero-config — no LLM needed for basic suggestions
Replaced complex pattern-matching heuristics with simple generic prompts. - Removed _heuristic_suggestion() (8 pattern rules) - Removed _heuristic_consecutive_suggestion() (20+ tool mappings) - Fallback is now just a direct text suggestion - Compression model still tries first for smarter suggestions - Net: -90 lines of dead pattern logic
6a3fd14 to
2ae59cb
Compare
Author
|
Thanks for the heads up! I have rebased the branch onto a clean upstream/main and cherry-picked only the circuit breaker commits. The diff now touches only run_agent.py - no more unrelated modules. |
Author
|
已关闭。原因:原方案已完全重构。 原 PR 使用基于参数 hash 的硬判断,默认开启,阈值 3。 新方案:
新 PR:#16749 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements a circuit breaker mechanism to detect and interrupt infinite tool-call loops in the agent.
Problem
The agent can enter infinite tool-call loops where it repeatedly calls the same (or similar) tools without making progress, consuming tokens and time indefinitely.
Solution
Changes
Testing
Note: This is a clean resubmission of #12632, which inadvertently included unrelated i18n commits.