Skip to content

[Bug]: vision_analyze on a slow enough local model will timeout due to default 30s timout #2107

@smt923

Description

@smt923

Bug Description

vision_analyze fails with a timeout error when using a local model for the auxiliary vision client. The default 30s timeout in async_call_llm is too short for local models where image encoding alone can take 30s+ to complete (on very constrained hardware or large/slow models on consumer hardware etc)

Steps to Reproduce

  1. Use locally hosted model with vision
  2. Ask hermes to describe an image
  3. Wait 30~ seconds for timeout to occur

Expected Behavior

Image is analyzed successfully and the result is passed to the main model

Actual Behavior

Vision pre-analysis times out before the server returns the first token:

httpx.ReadTimeout
openai.APITimeoutError: Request timed out.
ERROR tools.vision_tools: Error analyzing image: Request timed out.

Affected Component

Tools (terminal, file ops, web, code execution, etc.)

Messaging Platform (if gateway-related)

No response

Operating System

Arch Linux (6.19.7-1)

Python Version

3.11.15

Hermes Version

v0.4.0 (2026.3.18)

Relevant Logs / Traceback

Expand: ERROR tools.vision_tools: Error analyzing image: Request timed out
2026-03-19 20:08:23,828 ERROR tools.vision_tools: Error analyzing image: Request timed out.
Traceback (most recent call last):
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
    raise exc
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 32, in read
    with map_exceptions(exc_map):
  File "/home/smt/.local/share/uv/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1604, in request
    response = await self._client.send(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpx/_client.py", line 1629, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpx/_client.py", line 1730, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
    with map_httpcore_exceptions():
  File "/home/smt/.local/share/uv/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/smt/.hermes/hermes-agent/tools/vision_tools.py", line 310, in vision_analyze_tool
    response = await async_call_llm(**call_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/agent/auxiliary_client.py", line 1512, in async_call_llm
    return await client.chat.completions.create(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/openai/resources/chat/completions/completions.py", line 2714, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1884, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/smt/.hermes/hermes-agent/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1622, in request
    raise APITimeoutError(request=request) from err
openai.APITimeoutError: Request timed out.

Root Cause Analysis (optional)

inside of vision_analyze_tool() the following call_kwargs is defined:

call_kwargs = {
            "task": "vision",
            "messages": messages,
            "temperature": 0.1,
            "max_tokens": 2000,
        }

without a timeout, the timeout ends up defaulting somewhere up the chain, many of them like async_call_llm() in agent/auxiliary_client.py have a default of 30

as a quick workaround I was able to add "timeout": 180, to call_kwargs which stopped the timeout

Proposed Fix (optional)

some kind of configuration or envvar would be useful as just bumping the default could obviously lead to some unwanted behavior for people who know their models should be returning fast, but for people playing around with local models and constrained hardware, some way to raise it would be useful

potentially also a more generous default if the provider is detected as custom/local would give a smoother experience for people using a wider range of configurations

Are you willing to submit a PR for this?

  • I'd like to fix this myself and submit a PR

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions