Skip to content

feat: concurrent tool execution with ThreadPoolExecutor#1152

Merged
teknium1 merged 1 commit into
mainfrom
hermes/hermes-f47f71c0
Mar 13, 2026
Merged

feat: concurrent tool execution with ThreadPoolExecutor#1152
teknium1 merged 1 commit into
mainfrom
hermes/hermes-f47f71c0

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

When the model returns multiple tool calls in a single response, they are now executed concurrently using a thread pool instead of sequentially. This significantly reduces wall-clock time when multiple independent tools are batched (e.g. parallel web_search, read_file, terminal calls).

Architecture

  • _execute_tool_calls() now dispatches between sequential and concurrent paths
  • Sequential path (renamed to _execute_tool_calls_sequential): unchanged original behavior, used for:
    • Single tool calls (no overhead needed)
    • Batches containing clarify (requires interactive user input)
  • Concurrent path (_execute_tool_calls_concurrent): uses ThreadPoolExecutor with up to 8 workers
  • _invoke_tool() extracted as shared tool invocation helper (handles both agent-level tools and registry-dispatched tools)

Safety Considerations

  • Pre-flight interrupt check: skips all tools if interrupted before starting
  • Per-tool exception handling: one failure doesn't crash the batch
  • Result truncation (100k char limit) applied per tool
  • Budget pressure injection after all tools complete
  • Checkpoints taken before file-mutating tools (before submitting to pool)
  • CLI spinner shows batch progress, then per-tool completion messages
  • _NEVER_PARALLEL_TOOLS frozenset controls which tools force sequential execution
  • _MAX_TOOL_WORKERS = 8 caps thread pool size

Display/UX

  • quiet_mode (CLI): Single spinner "running N tools concurrently" while executing, then individual cute messages per tool
  • verbose mode: Prints all tool calls upfront, then completion messages with timing

Tests

10 new tests in TestConcurrentToolExecution:

  • Dispatch routing (single → sequential, clarify → sequential, multi → concurrent)
  • All tools executed and results in correct order
  • Order preserved despite different tool execution times
  • Error handling (one tool fails, others succeed)
  • Interrupt before start skips all tools
  • Result truncation in concurrent path
  • _invoke_tool routing for regular and agent-level tools

Fixed 1 existing test (test_interrupt.py) that used MagicMock and needed real dispatch methods bound.

3374 tests passing, 0 new failures.

When the model returns multiple tool calls in a single response, they are
now executed concurrently using a thread pool instead of sequentially.
This significantly reduces wall-clock time when multiple independent tools
are batched (e.g. parallel web_search, read_file, terminal calls).

Architecture:
- _execute_tool_calls() dispatches to sequential or concurrent path
- Single tool calls and batches containing 'clarify' use sequential path
- Multiple non-interactive tools use ThreadPoolExecutor (max 8 workers)
- Results are collected and appended to messages in original order
- _invoke_tool() extracted as shared tool invocation helper

Safety:
- Pre-flight interrupt check skips all tools if interrupted
- Per-tool exception handling: one failure doesn't crash the batch
- Result truncation (100k char limit) applied per tool
- Budget pressure injection after all tools complete
- Checkpoints taken before file-mutating tools
- CLI spinner shows batch progress, then per-tool completion messages

Tests: 10 new tests covering dispatch logic, ordering, error handling,
interrupt behavior, truncation, and _invoke_tool routing.
@teknium1 teknium1 merged commit 0157253 into main Mar 13, 2026
1 check failed
angelburgosrosado pushed a commit to angelburgosrosado/hermes-agent that referenced this pull request Apr 27, 2026
…f47f71c0

feat: concurrent tool execution with ThreadPoolExecutor
02356abc pushed a commit to 02356abc/hermes-agent that referenced this pull request May 14, 2026
…f47f71c0

feat: concurrent tool execution with ThreadPoolExecutor
olympus-terminal pushed a commit to olympus-terminal/hermes-agent that referenced this pull request May 16, 2026
…f47f71c0

feat: concurrent tool execution with ThreadPoolExecutor
Egavasyug pushed a commit to Egavasyug/hermes-agent that referenced this pull request Jun 10, 2026
…f47f71c0

feat: concurrent tool execution with ThreadPoolExecutor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant