fix(telegram): detect wedged send path after reconnect storms (#31165)#31252
Closed
dskwe wants to merge 2 commits into
Closed
fix(telegram): detect wedged send path after reconnect storms (#31165)#31252dskwe wants to merge 2 commits into
dskwe wants to merge 2 commits into
Conversation
…search#31165) After sustained Bad Gateway / TimedOut reconnect storms, the PTB httpx client can enter a state where bot.send_message() returns a valid Message object (real message_id) but the message never reaches the recipient. The gateway's polling path recovers (getUpdates works) but the send path (sendMessage) silently drops. This adds a _send_path_degraded flag to TelegramAdapter: - Set to True when _handle_polling_network_error fires - Cleared when _verify_polling_after_reconnect's getMe() probe passes - When degraded, send() runs a post-send getMe() probe; if the probe fails, returns SendResult(success=False) so callers (cron live-adapter branch) fall through to the standalone delivery path which uses a fresh HTTP session.
Contributor
|
Merged via #31441 with your authorship preserved (commit 476c897 on main). I reshaped the fix down to 14 LOC: reused the existing reconnect heartbeat probe in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Cron jobs configured with
deliver: telegram:<chat_id>can silently drop messages after the Telegram gateway experiences a reconnect storm (sustained Bad Gateway / TimedOut errors). The gateway logs report successful delivery (delivered to telegram:... via live adapter) but the message never arrives in Telegram. No error is surfaced to the user.Root Cause
After prolonged network error cycles, the python-telegram-bot library's internal httpx connection pool can enter a wedged state where:
bot.send_message()returns a validMessageobject (realmessage_id, no exception)getUpdates) recovers independently via reconnection logicsendMessage) silently fails with no error signalThe cron scheduler's live-adapter delivery path trusts the adapter's
SendResult.successreturn value, so a wedged send looks identical to a successful delivery.Fix
Add send-path health tracking to
TelegramAdapter:_send_path_degradedflag — setTruewhen_handle_polling_network_errorfires (reconnect storm detected)send()runs a lightweightgetMe()after the actual send. If the probe times out (5s) or fails, returnSendResult(success=False)instead of the false-positive result_verify_polling_after_reconnectclears the flag after a successfulgetMe()probe confirms recoverysuccess=Falselets the cron scheduler's existing fallback mechanism deliver via the standalone path (fresh HTTP session)Files changed:
gateway/platforms/telegram.py(+39 lines)Changes
gateway/platforms/telegram.py: Added_send_path_degraded,_last_reconnect_error_atfields; reconnect storm detection in error handler; post-send health probe insend(); flag clearing in reconnect verificationHow to Test
./venv/bin/python3 -m pytest tests/gateway/test_telegram_send_path_health.py -v— 5 new tests./venv/bin/python3 -m pytest tests/gateway/ -v— all existing gateway tests pass