fix(agent): log outer-loop exceptions at ERROR with traceback#32264
Merged
Conversation
The outer 'except Exception' guard in run_conversation() captures exceptions raised inside the agent loop (during streaming, tool dispatch, message construction, etc.) and prints a one-line summary to the screen. The traceback was only logged at DEBUG, so it never landed in errors.log (WARNING+) and was lost. For intermittent failures — the most important kind to debug — users saw 'Error during OpenAI-compatible API call #N: <message>' on screen with no way to recover the call site. Switching to logger.exception() emits the full traceback at ERROR so it goes to both agent.log and errors.log automatically. This is a pure logging change; control flow is unchanged.
Contributor
🔎 Lint report:
|
bridge25
pushed a commit
to bridge25/hermes-agent
that referenced
this pull request
May 27, 2026
…search#32264) The outer 'except Exception' guard in run_conversation() captures exceptions raised inside the agent loop (during streaming, tool dispatch, message construction, etc.) and prints a one-line summary to the screen. The traceback was only logged at DEBUG, so it never landed in errors.log (WARNING+) and was lost. For intermittent failures — the most important kind to debug — users saw 'Error during OpenAI-compatible API call #N: <message>' on screen with no way to recover the call site. Switching to logger.exception() emits the full traceback at ERROR so it goes to both agent.log and errors.log automatically. This is a pure logging change; control flow is unchanged.
teknium1
added a commit
that referenced
this pull request
May 27, 2026
) Background processes whose command contains `gh pr view --json statusCheckRollup` or `gh pr checks | jq` now get a runtime hint in the result pointing at the canonical green-ci-policy snippets. The homebrew shape has caused at least seven silent CI-watcher failures in the past two weeks (#31329, #31448, #31695, #31709, #31745, #32264, #33131) — each one a different jq/awk/grep variation of the same fundamental problem (stdout buffering, jq null-key edge cases, conclusion-vs-status confusion, TTY-only banner grepping). The skill that documents this anti-pattern is excellent, but a skill only fires if the agent loads it. The tool surface fires on every misuse. This is the embed-footguns-in-tool-surface pattern from PR #31289 applied to a recurring failure mode that's outgrown skill-only enforcement. Detector is deliberately narrow — flags two specific shapes: 1. Any command containing `statusCheckRollup` (the JSON-API path — conclusion vs status field semantics keep burning us). 2. `gh pr view` / `gh pr checks` combined with `jq` (gh pr checks doesn't emit JSON, so any `| jq` here is confused intent; the canonical column-2 poller uses awk-on-tabs, not jq). Does NOT flag the blessed column-2 awk-on-tabs poller (which uses `awk -F"\t" "\==\"pending\""`) or the exit-code-driven `gh pr checks $PR >/dev/null` snippet. Hint composes with the existing background-without-notify_on_complete hint — both can fire on the same call. Each is independently actionable. Tests: - 4 new cases in tests/tools/test_notify_on_complete.py - test_homebrew_ci_poller_via_statusCheckRollup_emits_hint (positive) - test_homebrew_ci_poller_via_gh_pr_checks_piped_to_jq_emits_hint (positive) - test_canonical_column2_awk_poller_does_not_emit_homebrew_hint (negative) - test_canonical_gh_pr_checks_exit_code_loop_does_not_emit_hint (negative) - test_non_ci_background_command_does_not_emit_homebrew_hint (negative) - 30/30 passing (was 26)
mathias3
pushed a commit
to mathias3/hermes-agent
that referenced
this pull request
May 28, 2026
…search#32264) The outer 'except Exception' guard in run_conversation() captures exceptions raised inside the agent loop (during streaming, tool dispatch, message construction, etc.) and prints a one-line summary to the screen. The traceback was only logged at DEBUG, so it never landed in errors.log (WARNING+) and was lost. For intermittent failures — the most important kind to debug — users saw 'Error during OpenAI-compatible API call #N: <message>' on screen with no way to recover the call site. Switching to logger.exception() emits the full traceback at ERROR so it goes to both agent.log and errors.log automatically. This is a pure logging change; control flow is unchanged.
mathias3
pushed a commit
to mathias3/hermes-agent
that referenced
this pull request
May 28, 2026
…sResearch#33142) Background processes whose command contains `gh pr view --json statusCheckRollup` or `gh pr checks | jq` now get a runtime hint in the result pointing at the canonical green-ci-policy snippets. The homebrew shape has caused at least seven silent CI-watcher failures in the past two weeks (NousResearch#31329, NousResearch#31448, NousResearch#31695, NousResearch#31709, NousResearch#31745, NousResearch#32264, NousResearch#33131) — each one a different jq/awk/grep variation of the same fundamental problem (stdout buffering, jq null-key edge cases, conclusion-vs-status confusion, TTY-only banner grepping). The skill that documents this anti-pattern is excellent, but a skill only fires if the agent loads it. The tool surface fires on every misuse. This is the embed-footguns-in-tool-surface pattern from PR NousResearch#31289 applied to a recurring failure mode that's outgrown skill-only enforcement. Detector is deliberately narrow — flags two specific shapes: 1. Any command containing `statusCheckRollup` (the JSON-API path — conclusion vs status field semantics keep burning us). 2. `gh pr view` / `gh pr checks` combined with `jq` (gh pr checks doesn't emit JSON, so any `| jq` here is confused intent; the canonical column-2 poller uses awk-on-tabs, not jq). Does NOT flag the blessed column-2 awk-on-tabs poller (which uses `awk -F"\t" "\==\"pending\""`) or the exit-code-driven `gh pr checks $PR >/dev/null` snippet. Hint composes with the existing background-without-notify_on_complete hint — both can fire on the same call. Each is independently actionable. Tests: - 4 new cases in tests/tools/test_notify_on_complete.py - test_homebrew_ci_poller_via_statusCheckRollup_emits_hint (positive) - test_homebrew_ci_poller_via_gh_pr_checks_piped_to_jq_emits_hint (positive) - test_canonical_column2_awk_poller_does_not_emit_homebrew_hint (negative) - test_canonical_gh_pr_checks_exit_code_loop_does_not_emit_hint (negative) - test_non_ci_background_command_does_not_emit_homebrew_hint (negative) - 30/30 passing (was 26)
Bryce-huang
pushed a commit
to wbkunlun/hermes-agent
that referenced
this pull request
May 29, 2026
…search#32264) The outer 'except Exception' guard in run_conversation() captures exceptions raised inside the agent loop (during streaming, tool dispatch, message construction, etc.) and prints a one-line summary to the screen. The traceback was only logged at DEBUG, so it never landed in errors.log (WARNING+) and was lost. For intermittent failures — the most important kind to debug — users saw 'Error during OpenAI-compatible API call #N: <message>' on screen with no way to recover the call site. Switching to logger.exception() emits the full traceback at ERROR so it goes to both agent.log and errors.log automatically. This is a pure logging change; control flow is unchanged. #AI commit#
Bryce-huang
pushed a commit
to wbkunlun/hermes-agent
that referenced
this pull request
May 29, 2026
…sResearch#33142) Background processes whose command contains `gh pr view --json statusCheckRollup` or `gh pr checks | jq` now get a runtime hint in the result pointing at the canonical green-ci-policy snippets. The homebrew shape has caused at least seven silent CI-watcher failures in the past two weeks (NousResearch#31329, NousResearch#31448, NousResearch#31695, NousResearch#31709, NousResearch#31745, NousResearch#32264, NousResearch#33131) — each one a different jq/awk/grep variation of the same fundamental problem (stdout buffering, jq null-key edge cases, conclusion-vs-status confusion, TTY-only banner grepping). The skill that documents this anti-pattern is excellent, but a skill only fires if the agent loads it. The tool surface fires on every misuse. This is the embed-footguns-in-tool-surface pattern from PR NousResearch#31289 applied to a recurring failure mode that's outgrown skill-only enforcement. Detector is deliberately narrow — flags two specific shapes: 1. Any command containing `statusCheckRollup` (the JSON-API path — conclusion vs status field semantics keep burning us). 2. `gh pr view` / `gh pr checks` combined with `jq` (gh pr checks doesn't emit JSON, so any `| jq` here is confused intent; the canonical column-2 poller uses awk-on-tabs, not jq). Does NOT flag the blessed column-2 awk-on-tabs poller (which uses `awk -F"\t" "\==\"pending\""`) or the exit-code-driven `gh pr checks $PR >/dev/null` snippet. Hint composes with the existing background-without-notify_on_complete hint — both can fire on the same call. Each is independently actionable. Tests: - 4 new cases in tests/tools/test_notify_on_complete.py - test_homebrew_ci_poller_via_statusCheckRollup_emits_hint (positive) - test_homebrew_ci_poller_via_gh_pr_checks_piped_to_jq_emits_hint (positive) - test_canonical_column2_awk_poller_does_not_emit_homebrew_hint (negative) - test_canonical_gh_pr_checks_exit_code_loop_does_not_emit_hint (negative) - test_non_ci_background_command_does_not_emit_homebrew_hint (negative) - 30/30 passing (was 26) #AI commit#
mosaiq-systems
pushed a commit
to mosaiq-systems/hermes-agent
that referenced
this pull request
May 29, 2026
…search#32264) The outer 'except Exception' guard in run_conversation() captures exceptions raised inside the agent loop (during streaming, tool dispatch, message construction, etc.) and prints a one-line summary to the screen. The traceback was only logged at DEBUG, so it never landed in errors.log (WARNING+) and was lost. For intermittent failures — the most important kind to debug — users saw 'Error during OpenAI-compatible API call #N: <message>' on screen with no way to recover the call site. Switching to logger.exception() emits the full traceback at ERROR so it goes to both agent.log and errors.log automatically. This is a pure logging change; control flow is unchanged.
mosaiq-systems
pushed a commit
to mosaiq-systems/hermes-agent
that referenced
this pull request
May 29, 2026
…sResearch#33142) Background processes whose command contains `gh pr view --json statusCheckRollup` or `gh pr checks | jq` now get a runtime hint in the result pointing at the canonical green-ci-policy snippets. The homebrew shape has caused at least seven silent CI-watcher failures in the past two weeks (NousResearch#31329, NousResearch#31448, NousResearch#31695, NousResearch#31709, NousResearch#31745, NousResearch#32264, NousResearch#33131) — each one a different jq/awk/grep variation of the same fundamental problem (stdout buffering, jq null-key edge cases, conclusion-vs-status confusion, TTY-only banner grepping). The skill that documents this anti-pattern is excellent, but a skill only fires if the agent loads it. The tool surface fires on every misuse. This is the embed-footguns-in-tool-surface pattern from PR NousResearch#31289 applied to a recurring failure mode that's outgrown skill-only enforcement. Detector is deliberately narrow — flags two specific shapes: 1. Any command containing `statusCheckRollup` (the JSON-API path — conclusion vs status field semantics keep burning us). 2. `gh pr view` / `gh pr checks` combined with `jq` (gh pr checks doesn't emit JSON, so any `| jq` here is confused intent; the canonical column-2 poller uses awk-on-tabs, not jq). Does NOT flag the blessed column-2 awk-on-tabs poller (which uses `awk -F"\t" "\==\"pending\""`) or the exit-code-driven `gh pr checks $PR >/dev/null` snippet. Hint composes with the existing background-without-notify_on_complete hint — both can fire on the same call. Each is independently actionable. Tests: - 4 new cases in tests/tools/test_notify_on_complete.py - test_homebrew_ci_poller_via_statusCheckRollup_emits_hint (positive) - test_homebrew_ci_poller_via_gh_pr_checks_piped_to_jq_emits_hint (positive) - test_canonical_column2_awk_poller_does_not_emit_homebrew_hint (negative) - test_canonical_gh_pr_checks_exit_code_loop_does_not_emit_hint (negative) - test_non_ci_background_command_does_not_emit_homebrew_hint (negative) - 30/30 passing (was 26)
gweeteve
pushed a commit
to gweeteve/hermes-agent
that referenced
this pull request
Jun 2, 2026
…search#32264) The outer 'except Exception' guard in run_conversation() captures exceptions raised inside the agent loop (during streaming, tool dispatch, message construction, etc.) and prints a one-line summary to the screen. The traceback was only logged at DEBUG, so it never landed in errors.log (WARNING+) and was lost. For intermittent failures — the most important kind to debug — users saw 'Error during OpenAI-compatible API call #N: <message>' on screen with no way to recover the call site. Switching to logger.exception() emits the full traceback at ERROR so it goes to both agent.log and errors.log automatically. This is a pure logging change; control flow is unchanged.
gweeteve
pushed a commit
to gweeteve/hermes-agent
that referenced
this pull request
Jun 2, 2026
…sResearch#33142) Background processes whose command contains `gh pr view --json statusCheckRollup` or `gh pr checks | jq` now get a runtime hint in the result pointing at the canonical green-ci-policy snippets. The homebrew shape has caused at least seven silent CI-watcher failures in the past two weeks (NousResearch#31329, NousResearch#31448, NousResearch#31695, NousResearch#31709, NousResearch#31745, NousResearch#32264, NousResearch#33131) — each one a different jq/awk/grep variation of the same fundamental problem (stdout buffering, jq null-key edge cases, conclusion-vs-status confusion, TTY-only banner grepping). The skill that documents this anti-pattern is excellent, but a skill only fires if the agent loads it. The tool surface fires on every misuse. This is the embed-footguns-in-tool-surface pattern from PR NousResearch#31289 applied to a recurring failure mode that's outgrown skill-only enforcement. Detector is deliberately narrow — flags two specific shapes: 1. Any command containing `statusCheckRollup` (the JSON-API path — conclusion vs status field semantics keep burning us). 2. `gh pr view` / `gh pr checks` combined with `jq` (gh pr checks doesn't emit JSON, so any `| jq` here is confused intent; the canonical column-2 poller uses awk-on-tabs, not jq). Does NOT flag the blessed column-2 awk-on-tabs poller (which uses `awk -F"\t" "\==\"pending\""`) or the exit-code-driven `gh pr checks $PR >/dev/null` snippet. Hint composes with the existing background-without-notify_on_complete hint — both can fire on the same call. Each is independently actionable. Tests: - 4 new cases in tests/tools/test_notify_on_complete.py - test_homebrew_ci_poller_via_statusCheckRollup_emits_hint (positive) - test_homebrew_ci_poller_via_gh_pr_checks_piped_to_jq_emits_hint (positive) - test_canonical_column2_awk_poller_does_not_emit_homebrew_hint (negative) - test_canonical_gh_pr_checks_exit_code_loop_does_not_emit_hint (negative) - test_non_ci_background_command_does_not_emit_homebrew_hint (negative) - 30/30 passing (was 26)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Outer-loop exceptions in the agent conversation loop now log the full traceback at ERROR level instead of DEBUG, so they land in
errors.logand become diagnosable.What was wrong
The
except Exceptionguard inrun_conversation()catches anything raised inside the agent loop (streaming, tool dispatch, message construction, post-turn hooks, etc.). It prints a one-line summary to the screen and logged the traceback at DEBUG — which doesn't reacherrors.log(WARNING+). Users hitting intermittent failures saw❌ Error during OpenAI-compatible API call #N: <message>with no way to recover the call site afterward.How this surfaced
While salvaging issue #492, a
terminaltool call raisedRuntimeError: Could not determine home directory.mid-loop (from aPath("~/...").expanduser()somewhere in the dispatch path, exact site unknown). The CLI showed the one-liner. The next tool call worked fine. Investigation today found:errors.logwas empty for that session — thelogger.debug(..., exc_info=True)swallowed the tracebackpathlib._make_child_relpathraise whenos.path.expanduser("~")can't resolve, but without a stack trace there are 50+ candidate call sitesChange
logger.debug("Outer loop error in API call #%d", api_call_count, exc_info=True)→logger.exception("Outer loop error in API call #%d", api_call_count)logger.exception()emits at ERROR with the traceback automatically. Control flow is unchanged; this is pure observability.Validation
python -c "import agent.conversation_loop")Follow-up
When this fix is in main and someone hits the
Could not determine home directoryrecurrence (or any other intermittent outer-loop failure),errors.logwill now have the full stack trace and we can fix the root cause directly.Infographic