fix(goals): pause loop when judge errors and response is terminal (#27585)#27752
Closed
briandevans wants to merge 1 commit into
Closed
fix(goals): pause loop when judge errors and response is terminal (#27585)#27752briandevans wants to merge 1 commit into
briandevans wants to merge 1 commit into
Conversation
508c9ac to
76fd236
Compare
76fd236 to
fb53bbd
Compare
…usResearch#27585) When the goal judge (an auxiliary model call) raises an API/transport error, judge_goal fails OPEN to `("continue", "judge error: ...", False)` on the assumption that a broken judge must not wedge progress. The turn budget is the only backstop. That assumption breaks when the assistant has already produced a terminal response on the same turn — e.g. `"Goal is complete. I'm stopping here because the requested goal is complete."`. The fail-open re-queues a continuation prompt; the agent re-emits a near-identical terminal message; the judge keeps erroring; the gateway channel (Discord/Telegram/etc.) gets spammed every few seconds until the turn budget runs out. This adds a safe-stop fallback in `GoalManager.evaluate_after_turn`: when the verdict is `continue` AND the reason carries the `_JUDGE_ERROR_REASON_PREFIX` AND the response matches one of a small set of terminal phrasing patterns, pause the loop with a descriptive reason instead of queueing another turn. Parse-failure auto-pause already guards the bad-judge-output case; this guards the unreachable-judge case. `judge_goal`'s error path now emits the prefix via the `_JUDGE_ERROR_REASON_PREFIX` constant so the contract is explicit. Tests cover: - judge API error + terminal response → paused (regression for NousResearch#27585) - judge API error + non-terminal response → still continues (fail-open preserved on in-flight goals) - judge OK returns continue + response uses terminal phrasing → still continues (successful judge retains authority) - judge done + terminal response → status=done, safe-stop does not fire - Unit coverage of the terminal-phrase detector across positive and negative cases. Fixes NousResearch#27585 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fb53bbd to
9db61bc
Compare
Contributor
Author
|
Housekeeping: closing to keep my open-PR set focused on actively-reviewed work. This has been open ~14d without maintainer review and the surrounding code has continued to move, so it's unlikely to land as-is. The underlying fix still stands — happy to reopen and rebase if it would be useful. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
When the goal judge raises an API/transport error (e.g.
BadRequestError,ConnectionError, 4xx),judge_goaldeliberately fails OPEN to("continue", "judge error: ...", False)so a broken judge doesn't wedge progress. That assumption breaks when the assistant has already emitted a terminal response on the same turn —evaluate_after_turnre-queues a continuation prompt, the agent re-emits a near-identical terminal message, the judge keeps erroring, and the gateway channel (Discord/Telegram/etc.) gets spammed with the same completion message every few seconds until the turn budget runs out (16 of 20 turns in the reporter's case). This PR adds a safe-stop fallback: when the verdict iscontinueAND the reason starts with_JUDGE_ERROR_REASON_PREFIX("judge error:") AND the assistant's response matches a small set of terminal-phrasing patterns ("goal is complete", "i'm/i am stopping", "nothing left to do"), pause the loop instead of queueing another turn. Fail-open is preserved for in-flight goals (non-terminal response → still continue), so a transient judge outage mid-task does not wedge legitimate progress.Related Issue
Fixes #27585
Type of Change
Changes Made
hermes_cli/goals.py—judge_goal's error path now emits the_JUDGE_ERROR_REASON_PREFIXshared constant so the producer/consumer contract is explicit.GoalManager.evaluate_after_turnadds the safe-stop fallback: terminal-phrase regex match + judge-error prefix → pause withpaused_reason="goal judge unreachable; assistant response appears terminal — pausing to avoid spam".tests/hermes_cli/test_goals.py— newTestJudgeErrorSafeStopclass (judge API error + terminal response → paused; judge API error + non-terminal → still continues; judge OK + terminal phrasing → still continues; judge done + terminal phrasing → status=done) andTestResponseLooksTerminalparametrized helper coverage.How to Test
uv run --with pytest --with pytest-xdist --with pytest-asyncio python3 -m pytest tests/hermes_cli/test_goals.py -v— 65 passed.test_judge_api_error_with_terminal_response_auto_pausesfails withassert True is Falseon theshould_continueassertion — exactly matching the issue's reproduction. With the fix applied, it passes alongside the existing 61 tests.Checklist
Code
fix(scope):,feat(scope):, etc.)Documentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/ASibling Audit
Related