Skip to content

feat(webhook): per-subscription --toolset override (closes #32899)#32902

Open
rmaffeo wants to merge 1 commit into
NousResearch:mainfrom
rmaffeo:feat/webhook-per-route-toolset
Open

feat(webhook): per-subscription --toolset override (closes #32899)#32902
rmaffeo wants to merge 1 commit into
NousResearch:mainfrom
rmaffeo:feat/webhook-per-route-toolset

Conversation

@rmaffeo

@rmaffeo rmaffeo commented May 27, 2026

Copy link
Copy Markdown

Closes #32899.

Summary

Adds a per-subscription --toolset flag on hermes webhook subscribe so trusted, authenticated webhook routes can opt out of the safe-default toolset locked down in #30745, without widening the platform globally via platform_toolsets.webhook in config.yaml.

Motivation

#30745 (May 24 2026) correctly restricted the default hermes-webhook toolset to a safe subset (web_search, web_extract, vision_analyze, clarify) to keep prompt-injection from third-party webhook payloads — GitHub PR titles/comments, Stripe metadata, ntfy publishers — out of terminal, write_file, send_message, and other local-execution tools.

That's the right default. But Hermes's own bundled skills (ha-hermes-webhook-bridge, ha-cross-sensor-snapshot-gate, webhook-subscriptions) explicitly document the webhook adapter as the integration point for trusted, authenticated event sources too — Home Assistant via rest_command + per-route X-Gitlab-Token, Hubitat virtual switches, internal CI/cron. Those flows depend on terminal/write_file/send_message (fetch a snapshot, save to disk, run vision_analyze, hand it off to Telegram via send_message ... MEDIA:<path>). After #30745 every such route went silently broken: the gateway dispatches an agent that has no way to complete the task and either fails clearly or hallucinates "ok".

The only existing workaround is platform_toolsets.webhook: [hermes-cli] in config.yaml, which widens every webhook route — including the third-party-payload ones the lockdown was designed to protect.

Design

Per-subscription --toolset override:

hermes webhook subscribe ha-front-door \
  --toolset hermes-cli \
  --deliver log \
  --prompt "..."
  • New --toolset arg, repeatable (action="append"), accepts any toolset name in TOOLSETS (validated at subscribe time, rejected with a friendly error if unknown).
  • Persisted as toolsets: [...] in the subscription JSON. Subscriptions without it (the overwhelming majority — anything that takes payloads from outside the operator's network) keep the safe default.
  • WebhookAdapter populates MessageEvent.enabled_toolsets_override (new optional field on the dataclass) from route_config["toolsets"]. Empty list and missing field both normalise to None so the platform default still applies.
  • GatewayRunner._run_agent accepts enabled_toolsets_override. When set, composite names like hermes-cli are expanded to their CONFIGURABLE_TOOLSETS members the same way _get_platform_tools does, so callers can pass a single composite and get the right granular runtime set (not the raw composite name, which the agent runtime doesn't register as an individual toolset).
  • _handle_message forwards event.enabled_toolsets_override into _run_agent. No other adapter sets the field, so behaviour everywhere else is unchanged.
  • Unknown toolset names at request time are logged and ignored (route falls back to platform default), so a typo in a JSON-edited subscription degrades to the safe default rather than failing open.
  • hermes webhook list and the subscribe-output banner display the configured toolsets when present.

Proxy mode (_run_agent_via_proxy) is intentionally out of scope — the override is only meaningful for the local-agent path. The proxy variant still uses the platform default.

Security posture

The default for new subscriptions is unchanged. The widening is opt-in, per-route, requires the operator to explicitly pass --toolset, and is documented in the updated webhook-subscriptions skill with explicit guidance on when it's safe to use (authenticated LAN routes only) and when it's not (any webhook that receives payloads from outside operator control). The skill's "When NOT to use" section is deliberately blunt: "Anything the agent reads from those payloads becomes prompt-injectable text; with terminal or send_message exposed, that is a remote-code-execution surface."

The HMAC auth, idempotency, rate limiting, signature validation, and other safety rails from previous webhook hardening (#30745, #30200, dbf73e9, 15aa688) are untouched.

Tests

New cases (all passing):

  • tests/hermes_cli/test_webhook_cli.py:

    • test_toolset_override_persisted — single --toolset is stored.
    • test_toolset_override_repeatable — multiple --toolset flags accumulate.
    • test_toolset_unknown_rejected — unknown name aborts subscribe and the route isn't persisted.
    • test_no_toolset_omits_key — default case writes no toolsets field (so the gateway falls through to the platform default resolver).
  • tests/gateway/test_webhook_integration.py:

    • test_toolset_field_propagated_to_event — route with toolsets: ["hermes-cli"] yields event.enabled_toolsets_override == ["hermes-cli"].
    • test_no_toolset_field_leaves_override_none — route without the field yields None.
    • test_empty_toolset_list_treated_as_unsettoolsets: [] normalises to None.

Existing webhook + tools_config suites all still pass (126 tests in the immediate-vicinity suites, 160 in the broader sweep, no regressions).

Backwards compatibility

  • Subscriptions written before this change have no toolsets field → resolve to None → platform default → safe hermes-webhook toolset. No behaviour change.
  • MessageEvent.enabled_toolsets_override is an optional field with a default of None; no adapter except the webhook one sets it.
  • _run_agent adds the new kwarg with default=None; the second internal call site (followup_result, around line 17843) preserves None implicitly, which is correct for a different-event continuation.

Notes for reviewers

  • The auto-generated docs page at website/docs/user-guide/skills/bundled/devops/devops-webhook-subscriptions.md is not included in this PR — website/scripts/generate-skill-docs.py rewrote line endings on every skill page (LF vs the existing CRLF on disk), which would have made the diff unreviewable. A maintainer with the right local line-ending setup should rerun the generator post-merge to refresh the bundled page from the updated SKILL.md. The source skill is updated and the generator will pick it up.
  • The --toolset resolver imports toolsets.TOOLSETS and hermes_cli.tools_config.{CONFIGURABLE_TOOLSETS, _toolset_allowed_for_platform} lazily inside the override branch so the cold path stays zero-cost when the override isn't set.

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/gateway Gateway runner, session dispatch, delivery comp/cli CLI entry point, hermes_cli/, setup wizard platform/webhook Webhook / API server labels May 27, 2026
@rmaffeo rmaffeo force-pushed the feat/webhook-per-route-toolset branch 2 times, most recently from c7030c8 to e91b3ba Compare May 27, 2026 01:10
Webhook subscriptions default to the safe hermes-webhook toolset
(web_search, web_extract, vision_analyze, clarify) introduced in NousResearch#30745
to prevent prompt-injection from third-party payloads (GitHub, Stripe,
etc.) reaching local execution tools. That default is correct, but it
broke trusted, authenticated LAN routes (Home Assistant rest_command +
X-Gitlab-Token, internal CI runners) whose prompts genuinely need
terminal/file/send_message access — there was no way to widen toolsets
on those routes short of widening the entire platform globally.

Add --toolset (repeatable) on 'hermes webhook subscribe', persist as
toolsets: [...] on the route, propagate through MessageEvent
.enabled_toolsets_override to gateway._run_agent, which resolves
composite names (e.g. 'hermes-cli') against CONFIGURABLE_TOOLSETS the
same way _get_platform_tools does. Default behavior unchanged when
--toolset is omitted; unknown toolset names are rejected at subscribe
time and logged-and-ignored at request time.

Closes NousResearch#32899.
@rmaffeo rmaffeo force-pushed the feat/webhook-per-route-toolset branch from e91b3ba to 3f9f660 Compare May 27, 2026 01:10
@rmaffeo

rmaffeo commented May 27, 2026

Copy link
Copy Markdown
Author

Pushed two follow-up fixes after a self-review:

  1. Followup-call propagation. _run_agent's recursive call site at line ~17890 (queued follow-up dispatch) now forwards enabled_toolsets_override through. Latent today (no webhook flow currently queues follow-ups) but a future-foot-gun if one ever does; cheaper to fix now than to debug later. Comment in the diff explains the intent.

  2. Extracted _resolve_toolset_override helper + unit tests. The composite-expansion logic was previously inline inside _run_agent, making it hard to test without mocking out half the gateway. Pulled it out to a module-level helper at gateway.run._resolve_toolset_override(names, platform_key) with the full rationale (including why it deliberately skips the _DEFAULT_OFF_TOOLSETS subtraction that _get_platform_tools applies to implicit expansions) in the docstring.

New test file tests/gateway/test_run_toolset_override.py (11 cases) covers:

  • empty list short-circuit
  • individual configurable name kept as-is
  • composite hermes-cli expands to the load-bearing configurable members (web, vision, terminal, file, messaging)
  • unknown name logged-and-skipped, alone returns empty (safe-degrade contract)
  • non-string YAML coercion (defensive str() against [123] etc.)
  • per-platform allow-list filters disallowed toolsets (both direct and via composite expansion)
  • composite expansion does NOT apply _DEFAULT_OFF_TOOLSETS subtraction (pinned so a future refactor doesn't silently change behaviour)
  • works for other platform keys (telegram smoke test)
  • repeated unknown names log exactly once per occurrence

Full webhook + tools-config + run-helper suite (216 tests) still green. PR diff is now 9 files, +442/-1.

Still deferred to a follow-up PR if maintainers want it:

  • promoting _toolset_allowed_for_platform from a private helper in hermes_cli/tools_config.py to a public API, then having both _get_platform_tools and _resolve_toolset_override consume the same canonical resolver (rather than the gateway helper reaching across to a private name)
  • a dedicated hermes-trusted-webhook umbrella toolset so the docs don't have to suggest --toolset hermes-cli (semantically conflates "the toolset CLI users get" with "the full local toolset")
  • regenerating the auto-generated website/docs/user-guide/skills/bundled/devops/devops-webhook-subscriptions.md from the updated SKILL.md (excluded from this PR because the generator rewrites line endings on every page, which would have buried the diff)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have platform/webhook Webhook / API server type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-subscription toolset override for hermes webhook subscribe

2 participants