Skip to content

fix(telegram): prevent duplicate message delivery on send timeout#5153

Merged
teknium1 merged 1 commit into
mainfrom
hermes/hermes-f2f17778
Apr 5, 2026
Merged

fix(telegram): prevent duplicate message delivery on send timeout#5153
teknium1 merged 1 commit into
mainfrom
hermes/hermes-f2f17778

Conversation

@teknium1

@teknium1 teknium1 commented Apr 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes duplicate message delivery when send_message() times out but the message was already delivered.

The Bug

TimedOut is a subclass of NetworkError in python-telegram-bot. Both retry layers treat it as a transient connection error:

  • Inner loop (send() in telegram.py): catches NetworkError → retries 3×
  • Outer loop (_send_with_retry() in base.py): matches "timeout"/"timed out" in error strings → retries 2×

Worst case: up to 9 delivery attempts for a single message.

But send_message is not idempotent — if the request reached Telegram and the HTTP response timed out, the message is already delivered. Retrying sends duplicates.

The Fix

Inner loop (telegram.py):

Outer loop (base.py):

  • Remove "timeout", "timed out", "readtimeout", "writetimeout" from _RETRYABLE_ERROR_PATTERNS
  • Add "connecttimeout" (safe to retry — connection was never established)
  • Keep "network" (other platforms still need it)
  • Add _is_timeout_error() + early return to prevent plain-text fallback path on timeouts (would also cause duplicate delivery)

Connection errors (ConnectionReset, ConnectError, etc.) are still retried — these fail before the request reaches the server.

Test results

  • test_send_retry.py: 28 tests including new timeout/connect-timeout classification tests
  • test_telegram_thread_fallback.py: 7 tests including new test_send_does_not_retry_timeout
  • Full gateway suite: 1885 passed (4 pre-existing failures in matrix voice + signal phone redaction)
  • E2E validation: inner loop single attempt on TimedOut, outer layer no retry, connect timeouts still retry correctly

Credit: @tmdgusya (PR #3899), @barun1997 (PR #3904) for identifying the bug and proposing fixes.

Closes #3899, closes #3904.

TimedOut is a subclass of NetworkError in python-telegram-bot. The
inner retry loop in send() and the outer _send_with_retry() in base.py
both treated it as a transient connection error and retried — but
send_message is not idempotent. When the request reaches Telegram but
the HTTP response times out, the message is already delivered. Retrying
sends duplicates. Worst case: up to 9 copies (inner 3x × outer 3x).

Inner loop (telegram.py):
- Import TimedOut separately, isinstance-check before generic
  NetworkError retry (same pattern as BadRequest carve-out from #3390)
- Re-raise immediately — no retry
- Mark as retryable=False in outer exception handler

Outer loop (base.py):
- Remove 'timeout', 'timed out', 'readtimeout', 'writetimeout' from
  _RETRYABLE_ERROR_PATTERNS (read/write timeouts are delivery-ambiguous)
- Add 'connecttimeout' (safe — connection never established)
- Keep 'network' (other platforms still need it)
- Add _is_timeout_error() + early return to prevent plain-text fallback
  on timeout errors (would also cause duplicate delivery)

Connection errors (ConnectionReset, ConnectError, etc.) are still
retried — these fail before the request reaches the server.

Credit: tmdgusya (PR #3899), barun1997 (PR #3904) for identifying the
bug and proposing fixes.

Closes #3899, closes #3904.
@teknium1 teknium1 merged commit 85cefc7 into main Apr 5, 2026
3 of 4 checks passed
naoironman-hue pushed a commit to naoironman-hue/hermes-agent that referenced this pull request Apr 5, 2026
…usResearch#5153)

TimedOut is a subclass of NetworkError in python-telegram-bot. The
inner retry loop in send() and the outer _send_with_retry() in base.py
both treated it as a transient connection error and retried — but
send_message is not idempotent. When the request reaches Telegram but
the HTTP response times out, the message is already delivered. Retrying
sends duplicates. Worst case: up to 9 copies (inner 3x × outer 3x).

Inner loop (telegram.py):
- Import TimedOut separately, isinstance-check before generic
  NetworkError retry (same pattern as BadRequest carve-out from NousResearch#3390)
- Re-raise immediately — no retry
- Mark as retryable=False in outer exception handler

Outer loop (base.py):
- Remove 'timeout', 'timed out', 'readtimeout', 'writetimeout' from
  _RETRYABLE_ERROR_PATTERNS (read/write timeouts are delivery-ambiguous)
- Add 'connecttimeout' (safe — connection never established)
- Keep 'network' (other platforms still need it)
- Add _is_timeout_error() + early return to prevent plain-text fallback
  on timeout errors (would also cause duplicate delivery)

Connection errors (ConnectionReset, ConnectError, etc.) are still
retried — these fail before the request reaches the server.

Credit: tmdgusya (PR NousResearch#3899), barun1997 (PR NousResearch#3904) for identifying the
bug and proposing fixes.

Closes NousResearch#3899, closes NousResearch#3904.
jooray added a commit to jooray/hermes-agent that referenced this pull request Apr 5, 2026
* upstream/main: (29 commits)
  style: use module-level re import instead of local import re as _re
  Preserve numeric credential labels in auth removal
  Honor provider reset windows in pooled credential failover
  docs: update docstring to mention Fireworks strict validation
  test: add strict API validation tests for Fireworks compatibility
  test: add test for _should_sanitize_tool_calls()
  refactor: use _should_sanitize_tool_calls in run_conversation()
  refactor: use _should_sanitize_tool_calls in _handle_max_iterations()
  refactor: use _should_sanitize_tool_calls in flush_memories()
  feat: add _should_sanitize_tool_calls() method
  test(redact): add regression tests for lowercase variable redaction (NousResearch#4367) (NousResearch#5185)
  docs(skill): claude-code v2.2 — add cheat sheet commands, env vars, rules, advanced features (NousResearch#5158)
  fix(telegram): prevent duplicate message delivery on send timeout (NousResearch#5153)
  fix: strip MEDIA: directives from streamed gateway messages (NousResearch#5152)
  docs(skill): comprehensive claude-code skill rewrite v2.0 (NousResearch#5155)
  fix(security): guard cron script against path traversal and redact output
  feat: add exit code context for common CLI tools in terminal results (NousResearch#5144)
  fix: move pre_llm_call plugin context to user message, preserve prompt cache (NousResearch#5146)
  fix: --yolo and other flags silently dropped when placed before 'chat' subcommand (NousResearch#5145)
  fix: include approval metadata in terminal tool results (NousResearch#5141)
  ...
lightx added a commit to lightx/hermes-agent that referenced this pull request Apr 5, 2026
…s-obsidian toolset

Merges 33 upstream commits while preserving local NixOS compatibility fixes:
- agent/auxiliary_client.py: deferred imports for get_hermes_home() and _AUTH_JSON_PATH
- agent/credential_pool.py: __getattr__ lazy-loading for hermes_cli.auth imports
- hermes_cli/config.py: lazy-load tool_backend_helpers to break circular deps

Adds hermes-obsidian toolset for Obsidian vault semantic search via ChromaDB.

Upstream highlights:
- fix: use main provider model for auxiliary tasks on non-aggregator providers (NousResearch#5091)
- feat: /model command — models.dev primary database + --provider flag (NousResearch#5181)
- feat(gateway): live-stream /update output + interactive prompt buttons (NousResearch#5180)
- fix(telegram): prevent duplicate message delivery on send timeout (NousResearch#5153)
- 54 new web design templates in popular-web-designs skill
- gitnexus-explorer skill for GitHub/GitLab reconnaissance
Tommyeds pushed a commit to Tommyeds/hermes-agent that referenced this pull request Apr 12, 2026
…usResearch#5153)

TimedOut is a subclass of NetworkError in python-telegram-bot. The
inner retry loop in send() and the outer _send_with_retry() in base.py
both treated it as a transient connection error and retried — but
send_message is not idempotent. When the request reaches Telegram but
the HTTP response times out, the message is already delivered. Retrying
sends duplicates. Worst case: up to 9 copies (inner 3x × outer 3x).

Inner loop (telegram.py):
- Import TimedOut separately, isinstance-check before generic
  NetworkError retry (same pattern as BadRequest carve-out from NousResearch#3390)
- Re-raise immediately — no retry
- Mark as retryable=False in outer exception handler

Outer loop (base.py):
- Remove 'timeout', 'timed out', 'readtimeout', 'writetimeout' from
  _RETRYABLE_ERROR_PATTERNS (read/write timeouts are delivery-ambiguous)
- Add 'connecttimeout' (safe — connection never established)
- Keep 'network' (other platforms still need it)
- Add _is_timeout_error() + early return to prevent plain-text fallback
  on timeout errors (would also cause duplicate delivery)

Connection errors (ConnectionReset, ConnectError, etc.) are still
retried — these fail before the request reaches the server.

Credit: tmdgusya (PR NousResearch#3899), barun1997 (PR NousResearch#3904) for identifying the
bug and proposing fixes.

Closes NousResearch#3899, closes NousResearch#3904.
angelburgosrosado pushed a commit to angelburgosrosado/hermes-agent that referenced this pull request Apr 27, 2026
…usResearch#5153)

TimedOut is a subclass of NetworkError in python-telegram-bot. The
inner retry loop in send() and the outer _send_with_retry() in base.py
both treated it as a transient connection error and retried — but
send_message is not idempotent. When the request reaches Telegram but
the HTTP response times out, the message is already delivered. Retrying
sends duplicates. Worst case: up to 9 copies (inner 3x × outer 3x).

Inner loop (telegram.py):
- Import TimedOut separately, isinstance-check before generic
  NetworkError retry (same pattern as BadRequest carve-out from NousResearch#3390)
- Re-raise immediately — no retry
- Mark as retryable=False in outer exception handler

Outer loop (base.py):
- Remove 'timeout', 'timed out', 'readtimeout', 'writetimeout' from
  _RETRYABLE_ERROR_PATTERNS (read/write timeouts are delivery-ambiguous)
- Add 'connecttimeout' (safe — connection never established)
- Keep 'network' (other platforms still need it)
- Add _is_timeout_error() + early return to prevent plain-text fallback
  on timeout errors (would also cause duplicate delivery)

Connection errors (ConnectionReset, ConnectError, etc.) are still
retried — these fail before the request reaches the server.

Credit: tmdgusya (PR NousResearch#3899), barun1997 (PR NousResearch#3904) for identifying the
bug and proposing fixes.

Closes NousResearch#3899, closes NousResearch#3904.
02356abc pushed a commit to 02356abc/hermes-agent that referenced this pull request May 14, 2026
…usResearch#5153)

TimedOut is a subclass of NetworkError in python-telegram-bot. The
inner retry loop in send() and the outer _send_with_retry() in base.py
both treated it as a transient connection error and retried — but
send_message is not idempotent. When the request reaches Telegram but
the HTTP response times out, the message is already delivered. Retrying
sends duplicates. Worst case: up to 9 copies (inner 3x × outer 3x).

Inner loop (telegram.py):
- Import TimedOut separately, isinstance-check before generic
  NetworkError retry (same pattern as BadRequest carve-out from NousResearch#3390)
- Re-raise immediately — no retry
- Mark as retryable=False in outer exception handler

Outer loop (base.py):
- Remove 'timeout', 'timed out', 'readtimeout', 'writetimeout' from
  _RETRYABLE_ERROR_PATTERNS (read/write timeouts are delivery-ambiguous)
- Add 'connecttimeout' (safe — connection never established)
- Keep 'network' (other platforms still need it)
- Add _is_timeout_error() + early return to prevent plain-text fallback
  on timeout errors (would also cause duplicate delivery)

Connection errors (ConnectionReset, ConnectError, etc.) are still
retried — these fail before the request reaches the server.

Credit: tmdgusya (PR NousResearch#3899), barun1997 (PR NousResearch#3904) for identifying the
bug and proposing fixes.

Closes NousResearch#3899, closes NousResearch#3904.
olympus-terminal pushed a commit to olympus-terminal/hermes-agent that referenced this pull request May 16, 2026
…usResearch#5153)

TimedOut is a subclass of NetworkError in python-telegram-bot. The
inner retry loop in send() and the outer _send_with_retry() in base.py
both treated it as a transient connection error and retried — but
send_message is not idempotent. When the request reaches Telegram but
the HTTP response times out, the message is already delivered. Retrying
sends duplicates. Worst case: up to 9 copies (inner 3x × outer 3x).

Inner loop (telegram.py):
- Import TimedOut separately, isinstance-check before generic
  NetworkError retry (same pattern as BadRequest carve-out from NousResearch#3390)
- Re-raise immediately — no retry
- Mark as retryable=False in outer exception handler

Outer loop (base.py):
- Remove 'timeout', 'timed out', 'readtimeout', 'writetimeout' from
  _RETRYABLE_ERROR_PATTERNS (read/write timeouts are delivery-ambiguous)
- Add 'connecttimeout' (safe — connection never established)
- Keep 'network' (other platforms still need it)
- Add _is_timeout_error() + early return to prevent plain-text fallback
  on timeout errors (would also cause duplicate delivery)

Connection errors (ConnectionReset, ConnectError, etc.) are still
retried — these fail before the request reaches the server.

Credit: tmdgusya (PR NousResearch#3899), barun1997 (PR NousResearch#3904) for identifying the
bug and proposing fixes.

Closes NousResearch#3899, closes NousResearch#3904.
gweeteve pushed a commit to gweeteve/hermes-agent that referenced this pull request Jun 2, 2026
…usResearch#5153)

TimedOut is a subclass of NetworkError in python-telegram-bot. The
inner retry loop in send() and the outer _send_with_retry() in base.py
both treated it as a transient connection error and retried — but
send_message is not idempotent. When the request reaches Telegram but
the HTTP response times out, the message is already delivered. Retrying
sends duplicates. Worst case: up to 9 copies (inner 3x × outer 3x).

Inner loop (telegram.py):
- Import TimedOut separately, isinstance-check before generic
  NetworkError retry (same pattern as BadRequest carve-out from NousResearch#3390)
- Re-raise immediately — no retry
- Mark as retryable=False in outer exception handler

Outer loop (base.py):
- Remove 'timeout', 'timed out', 'readtimeout', 'writetimeout' from
  _RETRYABLE_ERROR_PATTERNS (read/write timeouts are delivery-ambiguous)
- Add 'connecttimeout' (safe — connection never established)
- Keep 'network' (other platforms still need it)
- Add _is_timeout_error() + early return to prevent plain-text fallback
  on timeout errors (would also cause duplicate delivery)

Connection errors (ConnectionReset, ConnectError, etc.) are still
retried — these fail before the request reaches the server.

Credit: tmdgusya (PR NousResearch#3899), barun1997 (PR NousResearch#3904) for identifying the
bug and proposing fixes.

Closes NousResearch#3899, closes NousResearch#3904.
Egavasyug pushed a commit to Egavasyug/hermes-agent that referenced this pull request Jun 10, 2026
…usResearch#5153)

TimedOut is a subclass of NetworkError in python-telegram-bot. The
inner retry loop in send() and the outer _send_with_retry() in base.py
both treated it as a transient connection error and retried — but
send_message is not idempotent. When the request reaches Telegram but
the HTTP response times out, the message is already delivered. Retrying
sends duplicates. Worst case: up to 9 copies (inner 3x × outer 3x).

Inner loop (telegram.py):
- Import TimedOut separately, isinstance-check before generic
  NetworkError retry (same pattern as BadRequest carve-out from NousResearch#3390)
- Re-raise immediately — no retry
- Mark as retryable=False in outer exception handler

Outer loop (base.py):
- Remove 'timeout', 'timed out', 'readtimeout', 'writetimeout' from
  _RETRYABLE_ERROR_PATTERNS (read/write timeouts are delivery-ambiguous)
- Add 'connecttimeout' (safe — connection never established)
- Keep 'network' (other platforms still need it)
- Add _is_timeout_error() + early return to prevent plain-text fallback
  on timeout errors (would also cause duplicate delivery)

Connection errors (ConnectionReset, ConnectError, etc.) are still
retried — these fail before the request reaches the server.

Credit: tmdgusya (PR NousResearch#3899), barun1997 (PR NousResearch#3904) for identifying the
bug and proposing fixes.

Closes NousResearch#3899, closes NousResearch#3904.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant