Skip to content

feat(egress): doctor + audit + Anthropic native — follow-up to #30179#35149

Open
Bartok9 wants to merge 1 commit into
NousResearch:feat/iron-proxyfrom
Bartok9:feat/iron-proxy-doctor-audit-anthropic
Open

feat(egress): doctor + audit + Anthropic native — follow-up to #30179#35149
Bartok9 wants to merge 1 commit into
NousResearch:feat/iron-proxyfrom
Bartok9:feat/iron-proxy-doctor-audit-anthropic

Conversation

@Bartok9

@Bartok9 Bartok9 commented May 30, 2026

Copy link
Copy Markdown
Contributor

Opened by Bartok at Daniel's request, in response to Teknium's invitation for a security review of the iron-proxy egress work — Teknium asked Catalin (@catalinmpit), who had deployed Hermes on a hardened Hetzner VPS (Tailscale/UFW/Cloudflare/fail2ban), to look over PR #30179. This is a new, complementary PR, not a review of #30179.

"How much of the capabilities do you think you have to tradeoff for a truly secure hermes..." — Teknium

This PR answers that by adding operability and closing one of the largest LLM-specific scope cuts in #30179without trading away security or ease of use. Every feature is opt-in or read-only, stdlib-only, and composes with #30179 rather than rewriting it.

Why this complements #30179

PR #30179 deliberately scoped out three things to keep the core landing tight:

  1. No health/diagnostic surface — an operator had to eyeball status + tail logs to know if egress was actually wired.
  2. No audit-log tooling — the daemon emits a structured audit stream but there was no way to view/search/aggregate it.
  3. Anthropic native (x-api-key) was left as warn-only "uncovered" — the single biggest LLM-specific gap, since Anthropic doesn't use Authorization: Bearer.

This PR fills exactly those three gaps. It builds on feat/iron-proxy as a follow-up.

What lands

1. hermes egress doctor — end-to-end health check

A single read-only command that runs 11 checks (binary, CA expiry, config parse, mappings/env, daemon liveness, listening socket, per-host reachability, token-swap correctness, uncovered providers, docker DNS, SSRF/IMDS guard) and prints brew doctor-style fix-it hints. Flags: --json (stable {checks[], summary{}}, exit 1 on any fail), --check NAME (repeatable), --no-network (CI/hermetic). Credentials in failure messages are redacted to ≤4 chars.

2. hermes egress audit — structured audit viewer

tail [-n N] [-f], grep PATTERN --since, stats --since, export --format json|csv. Anomaly detection in stats surfaces first-time upstream hosts (catches a quiet DNS-rebind to a newly-allowlisted host) and hosts with >5% 403 rate. -f uses a 250ms polling loop (no new dependency) and survives log rotation. --since takes 30m/2h/7d/1w, today, or ISO-8601.

3. Anthropic native (x-api-key) — opt-in via --with-anthropic

Emits a parallel secrets rule with match_headers: ["x-api-key"], mints a TokenMapping scoped to api.anthropic.com, and removes ANTHROPIC_API_KEY from the uncovered/blocked sets when opted in. Off by default — the key may be used via OpenRouter (Bearer) rather than the native endpoint, and we won't break that flow silently. TokenMapping gains an auth_header field with a back-compatible "Authorization" default; mappings.json round-trips it and old files load cleanly.

New surfaces

Surface Type
run_doctor(), DoctorReport, DoctorCheck, DOCTOR_CHECK_NAMES iron_proxy.py public API
iter_audit_log, aggregate_audit_stats, detect_audit_anomalies, parse_since, audit_log_path iron_proxy.py pure functions
discover_xapikey_mappings, auth_header on TokenMapping iron_proxy.py
hermes egress doctor, hermes egress audit {tail,grep,stats,export}, hermes egress setup --with-anthropic CLI
Health-checks + Audit-log + Anthropic-native doc sections website/docs/.../iron-proxy.md, cli-commands.md

Failure modes considered

  • No new dependencies — stdlib (datetime, socket, ssl, urllib, concurrent.futures, csv, re) plus existing pyyaml/rich/openssl. CA expiry is read via openssl x509 -enddate, not the cryptography package.
  • No auto-mutationdoctor and audit never start/stop/rewrite anything. No auto-rotate, no auto-restart.
  • Windowsrun_doctor returns a single clear "not supported on Windows" fail, mirroring _platform_asset_name.
  • Secret leakage_redact_secret caps failure-message tokens at last-4-chars; there's a test asserting the proxy token never appears in a 403 token-swap failure detail.
  • Did not touch the inbound hermes proxy OAuth aggregator, did not bump _IRON_PROXY_VERSION, did not rewrite any feat(egress): iron-proxy credential-injection firewall for sandboxes #30179 file beyond additive edits.

Validation

Item Before After
Unit tests (iron-proxy doctor) n/a 30 / 30
Unit tests (iron-proxy audit) n/a 11 / 11
Unit tests (iron-proxy anthropic) n/a 9 / 9
CLI dispatch tests added (doctor + --with-anthropic) 20 24
Full iron-proxy suite (parity with PR #30179) 101 / 101 155 / 155
hermes egress doctor --no-network exit code n/a 0 (hermetic fixture)

Coverage gaps

(real ones, not aspirational)

  • doctor's reachability and token-swap checks are exercised only with a monkeypatched _https_head_via_proxy — there's no live-network integration test (by design; the existing E2E test is gated behind a marker and uses a real binary). The actual HTTP-proxy wiring of _https_head_via_proxy (ProxyHandler + unverified SSL context) is not asserted end-to-end here.
  • audit tail -f follow-loop is not unit-tested (it's an infinite poll loop); only the non-follow path is covered.
  • docker-dns check has no test that actually runs Docker — it's skipped/mocked. Behavior on a real Linux host with --add-host is unverified by CI.
  • The audit log format is assumed from the feat(egress): iron-proxy credential-injection firewall for sandboxes #30179 docs; if the real iron-proxy v0.39 audit schema differs, the parser falls back to {_raw} but the stats/anomaly aggregations key on upstream_host/status, which would then be empty.

Ambiguity flags

(decisions made without confirming)

  • auth_header on TokenMapping rather than a separate parallel list — chosen so a single mapping set can mix Bearer + x-api-key and build_proxy_config stays a straight loop. This changes the dataclass signature (additive, defaulted) and the mappings.json schema (additive key, back-compat read).
  • with_anthropic persisted in config.yaml under proxy.with_anthropic so doctor and start agree on whether Anthropic is covered. New config key.
  • Doctor exit-code policy: warn+skip are non-fatal (exit 0), only fail → exit 1 — mirroring brew doctor.
  • token-swap semantics: treats 401 (and 200/404) as "swap fired, reached upstream" and 403 as "proxy refused / rule broken." This is the documented contract but depends on the proxy returning 403 specifically on allowlist/rule failure.

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard provider/anthropic Anthropic native Messages API labels May 30, 2026
@nujovich

nujovich commented Jun 7, 2026

Copy link
Copy Markdown

Really appreciate the auth_header field on TokenMapping with the back-compat default — that's the right call.

One thing worth flagging for plugin authors building observability on top of Hermes: the x-api-key path (and the Bearer path too, actually) makes the auth header a poor anchor for provider identity. The hook fires inside the sandbox, pre-swap — so what it sees is the proxy token, not the real key or a provider name.

The audit log schema already has the right field for this: upstream_host (api.anthropic.com, openrouter.ai, etc.) reflects where the request actually landed, post-allowlist. That's the clean source of truth for resolving provider identity in any plugin that needs it.

Might be worth a note in the plugin authoring guide once this lands — something like "if you need to know which provider served a request, read upstream_host from the audit log rather than inspecting the auth header."

@Bartok9

Bartok9 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Polish — improved description bullets

hermes egress doctor — 11-check end-to-end health suite (--json, --check, --no-network) with brew-doctor-style fix hints and credential redaction in failure output
hermes egress audit — structured log tailing + anomaly detection (first-time hosts, >5% 403 spikes) with pure functions for full testability
• Adds native Anthropic x-api-key support via hermes egress setup --with-anthropic (opt-in, off by default)
• +50 tests (30 doctor, 11 audit, 9 Anthropic) + 4 CLI dispatch tests; 101 existing iron-proxy tests untouched
• Stdlib + existing pyyaml/rich/openssl only — no new dependencies

…search#30179

Three composable features on top of the iron-proxy egress firewall:

1. hermes egress doctor — 11-check end-to-end health check with
   --json/--check/--no-network, brew-doctor-style fix-it hints, and
   credential redaction in failure messages.

2. hermes egress audit — structured audit log viewer (tail -f / grep
   --since / stats / export json|csv) with first-time-host and >5%-403
   anomaly detection. Pure functions iter_audit_log / aggregate_audit_stats
   / detect_audit_anomalies for testability.

3. Anthropic native (x-api-key) support via 'hermes egress setup
   --with-anthropic'. Adds a parallel secrets rule matching x-api-key,
   mints a TokenMapping for api.anthropic.com, and removes Anthropic from
   the uncovered/blocked sets when opted in. Off by default. TokenMapping
   gains an auth_header field (back-compatible default 'Authorization').

Tests: +50 (30 doctor, 11 audit, 9 anthropic) + 4 CLI dispatch tests.
Existing iron-proxy suite unchanged (101 -> still pass).
No new dependencies (stdlib + existing pyyaml/rich/openssl only).
@Bartok9 Bartok9 force-pushed the feat/iron-proxy-doctor-audit-anthropic branch from 1f7d24a to ccc1168 Compare June 11, 2026 03:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have provider/anthropic Anthropic native Messages API type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants