fix(gateway): bound Telegram proxy httpx pools to stop fd leak#31885
Closed
konsisumer wants to merge 1 commit into
Closed
fix(gateway): bound Telegram proxy httpx pools to stop fd leak#31885konsisumer wants to merge 1 commit into
konsisumer wants to merge 1 commit into
Conversation
The Telegram adapter's general-request pool accumulates half-closed sockets when running behind a tunneling proxy: httpcore does not reliably release the underlying socket on ConnectError, so over long flaky-proxy runs CLOSED connections pile up until the process fd limit is hit and all sends fail. Cap the proxied pools and set a finite keepalive_expiry so idle/dead connections are evicted during pool maintenance instead of pinned for the process lifetime. All three knobs are env-tunable. Refs NousResearch#31599
Collaborator
|
Competing fix with #31687 — both address #31599 (Telegram proxy httpx fd leak). #31687 routes proxy-path pools through the shared |
This was referenced May 25, 2026
Contributor
Author
|
Closing — deferring to #31687 by @konsisumer which addresses the same. Reopen if that PR stalls. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The Telegram adapter accumulates half-closed sockets in its httpx general-request pool when running behind a tunneling proxy. As the reporter documents in #31599, httpcore does not reliably release the underlying socket on
ConnectError, so over long flaky-proxy runsCLOSEDconnections pile up (lsof showed 280/287 fds terminating at the local proxy port) until the process fd limit is hit and everybot.send_message()/set_my_commands()fails withhttpx.ConnectError: All connection attempts failed.The existing
_drain_polling_connectionsonly resets the polling pool (_request[0]) and deliberately leaves the general pool (_request[1]) untouched, so that leak vector was unbounded.This applies the reporter's prioritized suggestion (#1): when a proxy is configured, construct both proxied
HTTPXRequestpools with boundedhttpx.Limitsand a finitekeepalive_expiry. The cap stops unbounded growth, and the keepalive expiry lets httpx evict idle/dead connections during pool maintenance instead of pinning them for the process lifetime. All three knobs are env-tunable so high-throughput proxied deployments can widen them.This is a bounded mitigation of the leak, not a full cure of the upstream httpcore socket-release behavior — hence
Refsrather thanFixes. The reporter's deeper options (periodic general-pool drain, send-path heartbeat for observability) are left for maintainer direction.Related Issue
Refs #31599
Type of Change
Changes Made
gateway/platforms/telegram.py: add module-level_proxy_http_limits()helper that builds boundedhttpx.Limits(defaults:max_connections=20,max_keepalive_connections=10,keepalive_expiry=30.0s), each overridable viaHERMES_TELEGRAM_PROXY_MAX_CONNECTIONS/HERMES_TELEGRAM_PROXY_MAX_KEEPALIVE/HERMES_TELEGRAM_PROXY_KEEPALIVE_EXPIRY. Pass these limits throughhttpx_kwargswhen constructing the proxied request and get-updates pools. Non-proxy and fallback-IP paths are unchanged.tests/gateway/test_telegram_proxy_limits.py: unit tests for the helper — defaults, env overrides, and garbage-env fallback.How to Test
pytest tests/gateway/test_telegram_proxy_limits.py -q— verifies the limits defaults and env overrides.pytest tests/tools/test_send_message_telegram_proxy.py tests/gateway/test_proxy_mode.py -q— confirms the proxy send/construction paths still pass (27 passed locally).TELEGRAM_PROXYset, start the gateway and confirmlsof -p <pid> | wc -lstabilizes under flaky-proxy reconnect cycles instead of climbing past the fd limit; the proxied pools are now capped atmax_connections(20 by default) rather than 512.Checklist
Code
fix(gateway):)pytestsuites and they passDocumentation & Housekeeping
cli-config.yaml.exampleif I added/changed config keys — N/A (env-only knobs)CONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — N/A