🔎 Lint report: hermes/hermes-2b79b6da vs origin/main

ruff

Total: 1 on HEAD, 0 on base (🆕 +1)

🆕 New issues (1):

scripts/tool_search_livetest.py:358: [PLW1514] `pathlib.Path(...).write_text` without explicit `encoding` argument

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9427 on HEAD, 9422 on base (🆕 +5)

🆕 New issues (5):

scripts/tool_search_livetest.py:410: [invalid-assignment] invalid-assignment: Object of type `bound method ToolRegistry.dispatch(name: str, args: dict[Unknown, Unknown], **kwargs) -> str` is not assignable to attribute `dispatch` of type `def dispatch(self, name: str, args: dict[Unknown, Unknown], **kwargs) -> str`
tests/tools/test_tool_search.py:15: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
model_tools.py:840: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to `<module 'tools.tool_search'>`
scripts/tool_search_livetest.py:375: [invalid-assignment] invalid-assignment: Object of type `def logging_dispatch(name, args, **kw) -> Unknown` is not assignable to attribute `dispatch` of type `def dispatch(self, name: str, args: dict[Unknown, Unknown], **kwargs) -> str`
scripts/tool_search_livetest.py:387: [invalid-argument-type] invalid-argument-type: Argument to `AIAgent.__init__` is incorrect: Expected `list[str]`, found `None`

✅ Fixed issues: none

Unchanged: 4890 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

Sharing a test report — we tried this PR as the inference layer for a small local model in a degraded-mode fallback lane. Posting in case the data is useful; not a review, just a write-up of what we did, the environment, the changes we needed on our end, and the results.

What we wanted to learn

Whether progressive tool disclosure as implemented here is a viable fit for our specific use case: a small (4B) instruction-tuned model running on consumer hardware behind an OpenAI-compatible llama.cpp endpoint, used as a degraded-mode fallback for an agent that normally runs on a large cloud model with ~23 tools enabled.

This is a narrow fit-for-purpose test from one specific angle, not an evaluation of the PR against its design goals.

Environment

Changes we made on our side before the test was stable

Three modifications were necessary in our environment. Sharing them so the conditions of the test are clear.

1. llama.cpp --predict 4096 cap. Without it, our 4B model occasionally dropped out of structured tool emission mid-loop and began producing natural-language planning monologue with no stop condition. One observed run reached 9,335 generated tokens in a single assistant turn. Client-side HTTP timeout from the Hermes side did not propagate a cancel to llama.cpp, so the slot stayed busy and subsequent requests failed with Connection error until the slot drained.

Adding --predict 4096 to llama-server caps a runaway at ~4 min wall and frees the slot cleanly with finish_reason=length. Legitimate longest answers in our test set were ~700 tokens, so the cap is non-disruptive. This is a llama.cpp-side change, not a PR concern — flagging because it was a prerequisite for getting clean test data.

2. Reduced coreset. We exposed roughly 10 tools to the model (memory, terminal, session_search, read_file, write_file, search_files, patch, todo, clarify, plus this PR's bridge tool) instead of our full ~23. This is a separate ongoing line of work on what a small-model-friendly coreset looks like; we tested both arms below using this reduced set as the "treatment" baseline and our full ~23-tool set as the "control" baseline.

3. Bridge-tool description tightening. We made small adjustments to the description text on the discovery tool to make the 4B model invoke it more reliably. Diff available if useful.

Test set

Six sequential prompts in a single session (chained via X-Hermes-Session-Id), covering: pure chat, deferred-memory write, recall, two-tools-mixed (terminal + memory), two-tools-chained (search + memory), and a summary turn that exposes hallucination if earlier tools didn't actually execute.

Step 5 referenced a phrase (HELLO_AB4_BASELINE) that had been seeded into the sandbox state.db in an earlier session, so a true-positive answer exists and can be distinguished from a hallucinated one.

Results

Both arms used --predict 4096 and were run after warm KV. Verification of "did a tool actually execute" was by direct inspection of state.db (tool_calls rows) rather than HTTP status, because in one arm we saw the model emitting tool-call-shaped JSON that arrived as plain content and was reported as 200 OK.

Arm A — full coreset, this PR's mechanism disabled (our baseline)

state.db shows 0 rows with non-NULL tool_calls for this arm.

Arm B — reduced coreset, this PR's mechanism enabled

state.db shows real tool_calls rows and corresponding role=tool results.

Reference — same six prompts against our normal large-model agent (Claude Opus-class)

5/6 real tool executions. Step 5 returned 0 hits because the seeded phrase lives in the sandbox state.db, not the production one — model correctly reported no result rather than fabricating one.

Token observations (informational)

Arm B wins on cold turns and on chat-heavy mixes. Arm B's per-turn cost grows when discovery results pile up in conversation history across many tool-using turns in close succession. Whether the net is positive depends on call mix; for our intended workload (chat-heavy with occasional tool use) Arm B is meaningfully cheaper.

Fit-for-our-purpose result

For the specific lane we're investigating (small local model behind an OpenAI-compatible llama.cpp endpoint as a degraded-mode fallback), this PR's mechanism produced a working agent in our test where our baseline configuration did not. We're going to keep iterating on this configuration internally and would not have a usable small-model lane without it.

We're not claiming this generalises beyond our setup, and we make no claim about whether this is the use case the PR was designed for — just sharing the data in case it's useful as one additional point.

Happy to share the test driver script, the bridge-tool description diff, or full session JSON dumps if any of that would help.

This comment was written by a Hermes Agent instance (Claude Opus class model running on the Hermes Agent stack) on behalf of its operator, who ran the experiment and reviewed the report before posting.

Live test results

Ran a real-model end-to-end test against Claude Haiku 4.5 via OpenRouter. Five
scenarios, each run twice (tool_search ON and OFF), with 20 fake MCP tools
registered. Harness and transcripts in scripts/tool_search_livetest.py,
scripts/analyze_livetest.py, and scripts/out/.

10/10 runs passed. Every expected underlying tool was invoked. Zero
hallucinated tool names. Zero attempts to route a core tool through tool_call.
Display unwrap working in the CLI activity feed.

Side-by-side

Sample trace (Scenario A, ON)

bridges:    tool_search('create github issue')
         →  tool_describe(github_create_issue)
         →  tool_call → github_create_issue
underlying: github_create_issue

Sample trace (Scenario D, ON) — the safety guarantee in action

underlying: read_file → slack_send_message
bridges:    tool_search('post message Slack channel')
         →  tool_describe(slack_send_message)
         →  tool_call → slack_send_message

Note that read_file was called directly, not through tool_call. The
model correctly identified it as a core tool already in the visible tools
array and skipped the bridge for it. This is the safety invariant the report
flagged and that the implementation enforces by construction.

Observed costs

• ON adds +3 to +4 model round trips per task with deferred tools
• Single-tool tasks: ~16-20s vs ~10-11s direct (~2× wall time)
• Multi-tool chains: ~20s vs ~14s (~1.4× wall time)
• Pure-knowledge prompts: 0 extra round trips (no spurious tool_search)
• Token savings on the static side are real and measurable; the cost is paid
  in latency and additional round trips on cold-cache tool invocations

Confidence to ship

Behavior matches design. The bridge tools are usable by a real model without
prompt-engineering tricks. The auto threshold (default 10% of context)
means small toolsets pay no overhead — the +3 round trip tax only applies
when the deferrable surface is large enough to justify it. Recommended:
ship with enabled: auto as the default (already the case in this PR).

Future work, separate from this PR:

• A2/A3 prompts to test smaller models (Qwen, GPT-5.2 nano) — Haiku 4.5
  is a strong model and may not surface retrieval-quality failures that
  weaker models would hit
• Larger toolset (50+ deferred tools) to stress retrieval ranking
• Cost measurements with real cached/uncached pricing data

Merged via #34493 (rebased onto current main, your commit authorship preserved in git log).

The salvage carried the feature forward and closed a toolset-scoping hole found in review: the bridge read its catalog from the global registry, so a restricted-toolset session (subagent / kanban worker / curated gateway session) could tool_search the whole process registry and tool_call any plugin/MCP tool it was never granted. Now scoped to the session's own toolsets, with a defense-in-depth gate in both the bridge dispatch and the executor unwrap.

Also: dropped the 11 checked-in scripts/out/*.json transcripts (kept the harness, gitignored the output dir), routed the harness's key loading through load_hermes_dotenv, added _redact_secrets() over transcript/console output, and encoding="utf-8" on all file I/O.

On main as a87f0a8 (+ 369075d, 7427b9d, 1709776, 18c9e89).