Bug Report: APIConnectionError: Connection error. on every API call — httpx.Client shared between primary and request clients due to dict mutation in _create_openai_client()

Summary

Every API call fails with APIConnectionError: Connection error. after the first successful call (or from the very first call in some configurations). The underlying cause is RuntimeError: Cannot send a request, as the client has been closed.

Environment

Hermes Agent v0.9.0 (2026.4.13)
Project: /Users/nekokiller/.hermes/hermes-agent
Python: 3.11.15
OpenAI SDK: 2.31.0
Update available: 64 commits behind — run 'hermes update'
version = "0.9.0"
yc-bench = ["yc-bench @ git+https://github.com/collinear-ai/yc-bench.git@bfb0c88062450f46341bd9a5298903fc2e952a5c ; python_version >= '3.12'"]
333cb82 (HEAD -> main) fix: improve interrupt responsiveness during concurrent tool execution and follow-up turns (#10935)
v2026.4.13-280-g333cb825

Root Cause

_create_openai_client() mutates the client_kwargs dict passed to it in-place by inserting an http_client key:

# run_agent.py
def _create_openai_client(self, client_kwargs: dict, *, reason: str, shared: bool) -> Any:
    ...
    if "http_client" not in client_kwargs:
        client_kwargs["http_client"] = httpx.Client(...)  # ← mutates caller's dict!
    client = OpenAI(**client_kwargs)
    return client

The agent initialization stores self._client_kwargs = client_kwargs and then passes the same dict object directly to _create_openai_client():

# __init__ (line ~1036, 1059)
self._client_kwargs = client_kwargs
self.client = self._create_openai_client(client_kwargs, reason="agent_init", shared=True)
# After this call, self._client_kwargs["http_client"] == primary's httpx.Client (H_primary)

From that point on, every call to _create_request_openai_client() does:

request_kwargs = dict(self._client_kwargs)  # shallow copy — includes http_client=H_primary!
return self._create_openai_client(request_kwargs, ...)
# _create_openai_client sees "http_client" already present → skips creating a new one
# → request client shares H_primary with the primary client

When the streaming request completes and the request client is closed (reason="stream_request_complete"), it closes the shared H_primary. On the next API call, _ensure_primary_openai_client() detects that the primary is closed and calls _replace_primary_openai_client() — which creates a new primary but again mutates self._client_kwargs, perpetuating the cycle.

Result: Every API call after the first fails immediately with Cannot send a request, as the client has been closed.

Affected Versions

Confirmed on the current main branch (as of 2026). The bug is in the _create_openai_client() method of run_agent.py.

Reproduction

The bug is reliably triggered when:

1. The agent is run inside the gateway (via _run_in_executor_with_context) — i.e. the normal Telegram/Discord usage
2. The agent uses the default streaming API path (stream=True)
3. The primary client has been initialized at least once (standard flow)

It does not trigger in one-shot standalone tests (python run_agent.py directly) because those processes exit after one conversation, before the client-closed state propagates.

Fix

Add client_kwargs = dict(client_kwargs) at the top of _create_openai_client() so the method never mutates the caller's dict:

def _create_openai_client(self, client_kwargs: dict, *, reason: str, shared: bool) -> Any:
    from agent.auxiliary_client import _validate_base_url, _validate_proxy_env_urls
    _validate_proxy_env_urls()
    _validate_base_url(client_kwargs.get("base_url"))
    # Work on a local copy so we never mutate the caller's dict.
    # Previously this method added "http_client" in-place, which caused
    # subsequent dict(self._client_kwargs) copies to share the same
    # httpx.Client instance with the primary — closing any request client
    # would silently close the primary too.
    client_kwargs = dict(client_kwargs)
    ...

This one-line fix ensures that every client (primary and per-request) gets its own independent httpx.Client instance, so closing a request client never affects the primary.

Additional Notes

• _replace_primary_openai_client() had the same mutation bug (passing self._client_kwargs directly). This should also be fixed to pass dict(self._client_kwargs), though after the fix to _create_openai_client() itself, the call-site copy becomes redundant.
• The bug is masked in unit tests because mock clients don't use real httpx.Client instances.
• The gateway's agent caching (_agent_cache) means agents are long-lived across multiple messages, making this bug consistently reproducible in production.