Skip to content

fix(telegram): detect wedged send path after reconnect storms (#31165)#31441

Merged
teknium1 merged 1 commit into
mainfrom
hermes/hermes-84dadc9b
May 24, 2026
Merged

fix(telegram): detect wedged send path after reconnect storms (#31165)#31441
teknium1 merged 1 commit into
mainfrom
hermes/hermes-84dadc9b

Conversation

@teknium1

@teknium1 teknium1 commented May 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Cron deliver: telegram:CHAT_ID no longer silently drops messages after sustained Telegram reconnect storms. After Bad Gateway / TimedOut bursts, the PTB httpx pool could enter a state where bot.send_message() returned a valid Message (real message_id) but never transmitted; TelegramAdapter.send reported success and cron's live-adapter branch marked the run delivered.

Closes #31165.

How

Add _send_path_degraded to TelegramAdapter:

  • Set on _handle_polling_network_error (any reconnect attempt).
  • Cleared inside the existing _verify_polling_after_reconnect heartbeat probe, once getMe() confirms the Bot client is healthy.
  • While set, send() short-circuits with SendResult(success=False, retryable=True) so cron's live-adapter branch falls through to the standalone HTTP path (fresh session).

The heartbeat probe already runs 60s after each reconnect with a 10s timeout — we reuse it as the source of truth instead of duplicating a getMe() probe on every send.

Changes

  • gateway/platforms/telegram.py: 14 LOC. Field init + set on reconnect + clear on heartbeat + 4-line short-circuit in send(). getattr() read so test fixtures using object.__new__(TelegramAdapter) survive.
  • tests/gateway/test_telegram_send_path_health.py: 3 focused tests — healthy path delivers, degraded path short-circuits without calling send_message, storm-sets/heartbeat-clears the flag.

Validation

Result
tests/gateway/test_telegram_send_path_health.py (3 new) PASS
tests/gateway/test_telegram_*.py (11 files, 288 tests — all 10 object.__new__(TelegramAdapter) fixtures) PASS
tests/cron/ (14 files, 373 tests) PASS

Authored by @dskwe; reshaped onto current main with the minimum touch (14 LOC vs. original 39) — reuses the existing reconnect heartbeat instead of adding a per-send getMe() probe.

Infographic

telegram-wedge-detect

@github-actions

github-actions Bot commented May 24, 2026

Copy link
Copy Markdown
Contributor

🔎 Lint report: hermes/hermes-84dadc9b vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9080 on HEAD, 9077 on base (🆕 +3)

🆕 New issues (2):

Rule Count
unresolved-import 1
unresolved-attribute 1
First entries
tests/gateway/test_telegram_send_path_health.py:12: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/gateway/test_telegram_send_path_health.py:65: [unresolved-attribute] unresolved-attribute: Attribute `send_message` is not defined on `None` in union `Unknown | None`

✅ Fixed issues: none

Unchanged: 4834 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery platform/telegram Telegram bot adapter labels May 24, 2026
…#31165)

After sustained Bad Gateway / TimedOut reconnect cycles, the PTB httpx
client can enter a state where bot.send_message() returns a valid
Message (real message_id) but the message never reaches the recipient.
TelegramAdapter.send returns SendResult(success=True) and cron's
live-adapter branch marks the run delivered while the message is
silently dropped.

Add a _send_path_degraded flag. _handle_polling_network_error sets it
on reconnect storms; the existing _verify_polling_after_reconnect
heartbeat probe clears it once getMe() confirms the Bot client is
healthy. While the flag is set, send() short-circuits with
SendResult(success=False, retryable=True) so cron falls through to
the standalone delivery path (fresh HTTP session).

Closes #31165.

Co-authored-by: teknium1 <127238744+teknium1@users.noreply.github.com>
@teknium1 teknium1 force-pushed the hermes/hermes-84dadc9b branch from bb4fd7e to c0441cb Compare May 24, 2026 11:50
@alt-glitch

Copy link
Copy Markdown
Collaborator

Salvage of #31252 by @dskwe onto current main with test fixture compatibility fixes. Both fix #31165.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/telegram Telegram bot adapter type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cron Telegram live-adapter delivery can silently drop messages after reconnect storms

3 participants