fix: enable TCP keepalives to detect dead provider connections by teknium1 · Pull Request #10933 · NousResearch/hermes-agent

teknium1 · 2026-04-16T09:31:43Z

Summary

Fixes #10324 — When a custom provider drops a connection mid-stream, the TCP socket enters CLOSE-WAIT and epoll_wait blocks indefinitely. The agent hangs until manually killed.

Why Existing Defenses Don't Catch This

The codebase already has:

httpx read timeout (120s) — only fires when the socket layer reports no data. In CLOSE-WAIT, the kernel may not signal an error.
Stale stream detector (180s) — calls _force_close_tcp_sockets(), but sock.shutdown(SHUT_RDWR) on CLOSE-WAIT may not unblock a concurrent blocked read() on some kernels.
_cleanup_dead_connections() — only runs between requests, not during streaming.

All three are time-based and work once triggered, but they depend on the socket layer detecting the dead connection. Without keepalives, a silent CLOSE-WAIT socket generates no signal.

Fix

Inject SO_KEEPALIVE + keepalive parameters into the httpx transport:

Parameter	Value	Effect
`SO_KEEPALIVE`	1	Enable keepalive probes
`TCP_KEEPIDLE`	30s	Start probing after 30s idle
`TCP_KEEPINTVL`	10s	Probe every 10s
`TCP_KEEPCNT`	3	Give up after 3 failed probes

Dead peer detected within ~60s instead of hanging forever. Platform-aware: TCP_KEEPIDLE on Linux, TCP_KEEPALIVE on macOS. Falls back silently on unsupported platforms.

Verification

Confirmed on a real socket that keepalive parameters are applied:

Socket keepalive: 1, idle=30s, interval=10s, count=3

23 lines added. 75 provider parity tests pass.

When a custom provider drops a connection mid-stream, the TCP socket can enter CLOSE-WAIT and the httpx read timeout may never fire — epoll_wait blocks indefinitely because no data or error signal arrives. The agent hangs until manually killed. The existing defenses (httpx read timeout, stale stream detector, _force_close_tcp_sockets) are all time-based and work correctly once triggered, but they rely on the socket layer reporting the dead connection. Without TCP keepalives, the kernel has no reason to probe a silent connection. Fix: inject SO_KEEPALIVE + TCP_KEEPIDLE/KEEPINTVL/KEEPCNT into the httpx transport via socket_options. The kernel probes idle connections after 30s, retries every 10s, gives up after 3 failures — dead peer detected within ~60s instead of hanging forever. Platform-aware: uses TCP_KEEPIDLE on Linux, TCP_KEEPALIVE on macOS. Falls back silently if socket options aren't available (Windows, etc.). Closes #10324

…kwargs Copy client_kwargs at entry so the function can't leak injected transport state back to callers. This is preventive hardening after NousResearch#10933 (reverted in e07dbde) demonstrated how easy it is to poison self._client_kwargs: the PR added an httpx.Client to the dict, which was then re-read on the next request after the transport was torn down, yielding APIConnectionError with cause "Cannot send a request, as the client has been closed" on every retry. The underlying dead-connection issue (NousResearch#10324) is still open, so the next transport/keepalive attempt will sit in the same codepath — the copy locks the contract and a regression test pins it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Two new tests in tests/run_agent/ that pin the user-visible invariant behind AlexKucera's Discord report (2026-04-16): no matter how a future keepalive / transport fix for #10324 plumbs sockets in, sequential chats on the same AIAgent instance must all succeed. test_create_openai_client_reuse.py (no network, runs in CI): - test_second_create_does_not_wrap_closed_transport_from_first back-to-back _create_openai_client calls must not hand the same http_client (after an SDK close) to the second construction - test_replace_primary_openai_client_survives_repeated_rebuilds three sequential rebuilds via the real _replace_primary_openai_client entrypoint must each install a live client test_sequential_chats_live.py (opt-in, HERMES_LIVE_TESTS=1): - test_three_sequential_chats_across_client_rebuild real OpenRouter round trips, with an explicit _replace_primary_openai_client call between turns 2 and 3. Error-sentinel detector treats 'API call failed after 3 retries' replies as failures instead of letting them pass the naive truthy check (which is how a first draft of this test missed the bug it was meant to catch). Validation: clean main (post-revert, defensive copy present) -> all 4 tests PASS broken #10933 state (keepalive injection, no defensive copy) -> all 4 tests FAIL with precise messages pointing at #10933 Companion to taeuk178's test_create_openai_client_kwargs_isolation.py, which pins the syntactic 'don't mutate input dict' half of the same contract. Together they catch both the specific mechanism of #10933 and any other reimplementation that breaks the sequential-call invariant.

#11266) Two new tests in tests/run_agent/ that pin the user-visible invariant behind AlexKucera's Discord report (2026-04-16): no matter how a future keepalive / transport fix for #10324 plumbs sockets in, sequential chats on the same AIAgent instance must all succeed. test_create_openai_client_reuse.py (no network, runs in CI): - test_second_create_does_not_wrap_closed_transport_from_first back-to-back _create_openai_client calls must not hand the same http_client (after an SDK close) to the second construction - test_replace_primary_openai_client_survives_repeated_rebuilds three sequential rebuilds via the real _replace_primary_openai_client entrypoint must each install a live client test_sequential_chats_live.py (opt-in, HERMES_LIVE_TESTS=1): - test_three_sequential_chats_across_client_rebuild real OpenRouter round trips, with an explicit _replace_primary_openai_client call between turns 2 and 3. Error-sentinel detector treats 'API call failed after 3 retries' replies as failures instead of letting them pass the naive truthy check (which is how a first draft of this test missed the bug it was meant to catch). Validation: clean main (post-revert, defensive copy present) -> all 4 tests PASS broken #10933 state (keepalive injection, no defensive copy) -> all 4 tests FAIL with precise messages pointing at #10933 Companion to taeuk178's test_create_openai_client_kwargs_isolation.py, which pins the syntactic 'don't mutate input dict' half of the same contract. Together they catch both the specific mechanism of #10933 and any other reimplementation that breaks the sequential-call invariant.

Re-land of #10933, now guarded by the tests in #11266. ## The original problem (#10324) When a provider drops a TCP connection mid-stream, the socket can enter CLOSE-WAIT and ''epoll_wait'' may never fire — no data or error signal arrives, so the httpx read timeout never triggers and the agent hangs indefinitely. The other defenses (''_force_close_tcp_sockets'', stale stream detector) all ride on the socket layer reporting the dead connection, which it never does without probes. ## The fix Inject ''SO_KEEPALIVE'' + ''TCP_KEEPIDLE''/''KEEPINTVL''/''KEEPCNT'' into the httpx transport. Kernel probes after 30s idle, retries every 10s, gives up after 3 → dead peer detected within ~60s instead of hanging forever. Platform-aware: ''TCP_KEEPIDLE'' on Linux, ''TCP_KEEPALIVE'' on macOS. Silent no-op on Windows or anywhere the socket options aren't available. ## Why this one doesn't regress like #10933 did The original land (#10933) mutated ''client_kwargs'' in place when it injected the ''httpx.Client''. Since callers pass ''self._client_kwargs'' by reference, the injected client leaked into the instance state. After the first request, the OpenAI SDK closed its ''http_client'' — including the injected one. The next ''_create_openai_client'' call re-read the now-closed ''httpx.Client'' from ''self._client_kwargs'' and every subsequent chat raised ''APIConnectionError'' with cause ''RuntimeError: Cannot send a request, as the client has been closed'' (AlexKucera's Discord report, 2026-04-16). The defensive ''client_kwargs = dict(client_kwargs)'' copy already on main (taeuk178's #10978) means this injection only lands in the per-call local copy. Each ''_create_openai_client'' invocation gets its OWN fresh ''httpx.Client'' whose lifetime is tied to the paired ''OpenAI'' client. When that ''OpenAI'' client is closed (rebuild, teardown, credential rotation), its ''httpx.Client'' closes with it and the next call constructs a fresh one — no stale closed transport can be reused. ## Validation Full 4-test matrix all green (unit + live with real OpenRouter round trips, HERMES_LIVE_TESTS=1): tests/run_agent/test_create_openai_client_kwargs_isolation.py PASS tests/run_agent/test_create_openai_client_reuse.py PASS (2) tests/run_agent/test_sequential_chats_live.py PASS Socket options verified on the live httpx transport: _socket_options: [(1, 9, 1), (6, 4, 30), (6, 5, 10), (6, 6, 3)] = (SO_KEEPALIVE=1, TCP_KEEPIDLE=30s, TCP_KEEPINTVL=10s, TCP_KEEPCNT=3) Sequential-chat reproduction of the #10933 failure was explicitly run against this patch — the defensive copy on main prevents the closed transport from leaking back into ''self._client_kwargs'', so every rebuild constructs a fresh transport. Closes #10324

Re-land of #10933, now guarded by the tests in #11266. When a provider drops a TCP connection mid-stream, the socket can enter CLOSE-WAIT and ''epoll_wait'' may never fire — no data or error signal arrives, so the httpx read timeout never triggers and the agent hangs indefinitely. The other defenses (''_force_close_tcp_sockets'', stale stream detector) all ride on the socket layer reporting the dead connection, which it never does without probes. Inject ''SO_KEEPALIVE'' + ''TCP_KEEPIDLE''/''KEEPINTVL''/''KEEPCNT'' into the httpx transport. Kernel probes after 30s idle, retries every 10s, gives up after 3 → dead peer detected within ~60s instead of hanging forever. Platform-aware: ''TCP_KEEPIDLE'' on Linux, ''TCP_KEEPALIVE'' on macOS. Silent no-op on Windows or anywhere the socket options aren't available. The original land (#10933) mutated ''client_kwargs'' in place when it injected the ''httpx.Client''. Since callers pass ''self._client_kwargs'' by reference, the injected client leaked into the instance state. After the first request, the OpenAI SDK closed its ''http_client'' — including the injected one. The next ''_create_openai_client'' call re-read the now-closed ''httpx.Client'' from ''self._client_kwargs'' and every subsequent chat raised ''APIConnectionError'' with cause ''RuntimeError: Cannot send a request, as the client has been closed'' (AlexKucera's Discord report, 2026-04-16). The defensive ''client_kwargs = dict(client_kwargs)'' copy already on main (taeuk178's #10978) means this injection only lands in the per-call local copy. Each ''_create_openai_client'' invocation gets its OWN fresh ''httpx.Client'' whose lifetime is tied to the paired ''OpenAI'' client. When that ''OpenAI'' client is closed (rebuild, teardown, credential rotation), its ''httpx.Client'' closes with it and the next call constructs a fresh one — no stale closed transport can be reused. Full 4-test matrix all green (unit + live with real OpenRouter round trips, HERMES_LIVE_TESTS=1): tests/run_agent/test_create_openai_client_kwargs_isolation.py PASS tests/run_agent/test_create_openai_client_reuse.py PASS (2) tests/run_agent/test_sequential_chats_live.py PASS Socket options verified on the live httpx transport: _socket_options: [(1, 9, 1), (6, 4, 30), (6, 5, 10), (6, 6, 3)] = (SO_KEEPALIVE=1, TCP_KEEPIDLE=30s, TCP_KEEPINTVL=10s, TCP_KEEPCNT=3) Sequential-chat reproduction of the #10933 failure was explicitly run against this patch — the defensive copy on main prevents the closed transport from leaking back into ''self._client_kwargs'', so every rebuild constructs a fresh transport. Closes #10324

… (#11277) Re-land of #10933, now guarded by the tests in #11266. When a provider drops a TCP connection mid-stream, the socket can enter CLOSE-WAIT and ''epoll_wait'' may never fire — no data or error signal arrives, so the httpx read timeout never triggers and the agent hangs indefinitely. The other defenses (''_force_close_tcp_sockets'', stale stream detector) all ride on the socket layer reporting the dead connection, which it never does without probes. Inject ''SO_KEEPALIVE'' + ''TCP_KEEPIDLE''/''KEEPINTVL''/''KEEPCNT'' into the httpx transport. Kernel probes after 30s idle, retries every 10s, gives up after 3 → dead peer detected within ~60s instead of hanging forever. Platform-aware: ''TCP_KEEPIDLE'' on Linux, ''TCP_KEEPALIVE'' on macOS. Silent no-op on Windows or anywhere the socket options aren't available. The original land (#10933) mutated ''client_kwargs'' in place when it injected the ''httpx.Client''. Since callers pass ''self._client_kwargs'' by reference, the injected client leaked into the instance state. After the first request, the OpenAI SDK closed its ''http_client'' — including the injected one. The next ''_create_openai_client'' call re-read the now-closed ''httpx.Client'' from ''self._client_kwargs'' and every subsequent chat raised ''APIConnectionError'' with cause ''RuntimeError: Cannot send a request, as the client has been closed'' (AlexKucera's Discord report, 2026-04-16). The defensive ''client_kwargs = dict(client_kwargs)'' copy already on main (taeuk178's #10978) means this injection only lands in the per-call local copy. Each ''_create_openai_client'' invocation gets its OWN fresh ''httpx.Client'' whose lifetime is tied to the paired ''OpenAI'' client. When that ''OpenAI'' client is closed (rebuild, teardown, credential rotation), its ''httpx.Client'' closes with it and the next call constructs a fresh one — no stale closed transport can be reused. Full 4-test matrix all green (unit + live with real OpenRouter round trips, HERMES_LIVE_TESTS=1): tests/run_agent/test_create_openai_client_kwargs_isolation.py PASS tests/run_agent/test_create_openai_client_reuse.py PASS (2) tests/run_agent/test_sequential_chats_live.py PASS Socket options verified on the live httpx transport: _socket_options: [(1, 9, 1), (6, 4, 30), (6, 5, 10), (6, 6, 3)] = (SO_KEEPALIVE=1, TCP_KEEPIDLE=30s, TCP_KEEPINTVL=10s, TCP_KEEPCNT=3) Sequential-chat reproduction of the #10933 failure was explicitly run against this patch — the defensive copy on main prevents the closed transport from leaking back into ''self._client_kwargs'', so every rebuild constructs a fresh transport. Closes #10324

…esearch#10324) (NousResearch#10933) When a custom provider drops a connection mid-stream, the TCP socket can enter CLOSE-WAIT and the httpx read timeout may never fire — epoll_wait blocks indefinitely because no data or error signal arrives. The agent hangs until manually killed. The existing defenses (httpx read timeout, stale stream detector, _force_close_tcp_sockets) are all time-based and work correctly once triggered, but they rely on the socket layer reporting the dead connection. Without TCP keepalives, the kernel has no reason to probe a silent connection. Fix: inject SO_KEEPALIVE + TCP_KEEPIDLE/KEEPINTVL/KEEPCNT into the httpx transport via socket_options. The kernel probes idle connections after 30s, retries every 10s, gives up after 3 failures — dead peer detected within ~60s instead of hanging forever. Platform-aware: uses TCP_KEEPIDLE on Linux, TCP_KEEPALIVE on macOS. Falls back silently if socket options aren't available (Windows, etc.). Closes NousResearch#10324

…search#10933) (NousResearch#11266) Two new tests in tests/run_agent/ that pin the user-visible invariant behind AlexKucera's Discord report (2026-04-16): no matter how a future keepalive / transport fix for NousResearch#10324 plumbs sockets in, sequential chats on the same AIAgent instance must all succeed. test_create_openai_client_reuse.py (no network, runs in CI): - test_second_create_does_not_wrap_closed_transport_from_first back-to-back _create_openai_client calls must not hand the same http_client (after an SDK close) to the second construction - test_replace_primary_openai_client_survives_repeated_rebuilds three sequential rebuilds via the real _replace_primary_openai_client entrypoint must each install a live client test_sequential_chats_live.py (opt-in, HERMES_LIVE_TESTS=1): - test_three_sequential_chats_across_client_rebuild real OpenRouter round trips, with an explicit _replace_primary_openai_client call between turns 2 and 3. Error-sentinel detector treats 'API call failed after 3 retries' replies as failures instead of letting them pass the naive truthy check (which is how a first draft of this test missed the bug it was meant to catch). Validation: clean main (post-revert, defensive copy present) -> all 4 tests PASS broken NousResearch#10933 state (keepalive injection, no defensive copy) -> all 4 tests FAIL with precise messages pointing at NousResearch#10933 Companion to taeuk178's test_create_openai_client_kwargs_isolation.py, which pins the syntactic 'don't mutate input dict' half of the same contract. Together they catch both the specific mechanism of NousResearch#10933 and any other reimplementation that breaks the sequential-call invariant.

…esearch#10324) (NousResearch#11277) Re-land of NousResearch#10933, now guarded by the tests in NousResearch#11266. When a provider drops a TCP connection mid-stream, the socket can enter CLOSE-WAIT and ''epoll_wait'' may never fire — no data or error signal arrives, so the httpx read timeout never triggers and the agent hangs indefinitely. The other defenses (''_force_close_tcp_sockets'', stale stream detector) all ride on the socket layer reporting the dead connection, which it never does without probes. Inject ''SO_KEEPALIVE'' + ''TCP_KEEPIDLE''/''KEEPINTVL''/''KEEPCNT'' into the httpx transport. Kernel probes after 30s idle, retries every 10s, gives up after 3 → dead peer detected within ~60s instead of hanging forever. Platform-aware: ''TCP_KEEPIDLE'' on Linux, ''TCP_KEEPALIVE'' on macOS. Silent no-op on Windows or anywhere the socket options aren't available. The original land (NousResearch#10933) mutated ''client_kwargs'' in place when it injected the ''httpx.Client''. Since callers pass ''self._client_kwargs'' by reference, the injected client leaked into the instance state. After the first request, the OpenAI SDK closed its ''http_client'' — including the injected one. The next ''_create_openai_client'' call re-read the now-closed ''httpx.Client'' from ''self._client_kwargs'' and every subsequent chat raised ''APIConnectionError'' with cause ''RuntimeError: Cannot send a request, as the client has been closed'' (AlexKucera's Discord report, 2026-04-16). The defensive ''client_kwargs = dict(client_kwargs)'' copy already on main (taeuk178's NousResearch#10978) means this injection only lands in the per-call local copy. Each ''_create_openai_client'' invocation gets its OWN fresh ''httpx.Client'' whose lifetime is tied to the paired ''OpenAI'' client. When that ''OpenAI'' client is closed (rebuild, teardown, credential rotation), its ''httpx.Client'' closes with it and the next call constructs a fresh one — no stale closed transport can be reused. Full 4-test matrix all green (unit + live with real OpenRouter round trips, HERMES_LIVE_TESTS=1): tests/run_agent/test_create_openai_client_kwargs_isolation.py PASS tests/run_agent/test_create_openai_client_reuse.py PASS (2) tests/run_agent/test_sequential_chats_live.py PASS Socket options verified on the live httpx transport: _socket_options: [(1, 9, 1), (6, 4, 30), (6, 5, 10), (6, 6, 3)] = (SO_KEEPALIVE=1, TCP_KEEPIDLE=30s, TCP_KEEPINTVL=10s, TCP_KEEPCNT=3) Sequential-chat reproduction of the NousResearch#10933 failure was explicitly run against this patch — the defensive copy on main prevents the closed transport from leaking back into ''self._client_kwargs'', so every rebuild constructs a fresh transport. Closes NousResearch#10324