Skip to content

fix(agent): redact secrets in request_dump_*.json files#18750

Open
Tranquil-Flow wants to merge 2 commits into
NousResearch:mainfrom
Tranquil-Flow:fix/18707-redact-request-dump
Open

fix(agent): redact secrets in request_dump_*.json files#18750
Tranquil-Flow wants to merge 2 commits into
NousResearch:mainfrom
Tranquil-Flow:fix/18707-redact-request-dump

Conversation

@Tranquil-Flow

@Tranquil-Flow Tranquil-Flow commented May 2, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

_dump_api_request_debug() writes ~/.hermes/sessions/request_dump_*.json containing the live request body, headers, and provider error response. Two leak channels existed:

  1. Authorization header used the partial _mask_api_key_for_logs(), which keeps key[:8] and key[-4:] visible — enough to recognise an sk-proj-* prefix in a shared bug report. Tightened to Bearer ***.
  2. Body, error.body, error.response_text were written raw. Pasted credentials in user messages, keys quoted back in 401 bodies, and any other in-payload secret survived to disk unredacted.

The agent already ships agent/redact.py with regex coverage for every known provider key prefix and ENV=value / JSON-field assignment patterns — it just wasn't wired into this path.

This PR walks dump_payload recursively and applies redact_sensitive_text(value, force=True) to leaf string values before serialisation. Walking the structure (rather than the serialised text) is important: _ENV_ASSIGN_RE's greedy \S+ value group can otherwise consume past a JSON string's closing quote and corrupt the file. Operating on individual values keeps every regex bounded to a single string. force=True because this is a security boundary, not a logging preference: the dump must be safe regardless of the user's global security.redact_secrets setting.

This makes request_dump_*.json safe to attach to a GitHub issue or include in a tar -czf hermes.tar.gz ~/.hermes support bundle.

Related Issue

Fixes #18707

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • run_agent.py::_dump_api_request_debug — drop the _mask_api_key_for_logs(api_key) Authorization construction (and the now-unused api_key extraction); hard-code Bearer ***; recursively scrub leaf string values via redact_sensitive_text(force=True) before json.dumps; reuse the serialised string for both file write and the optional stdout echo
  • tests/run_agent/test_run_agent_codex_responses.py — add 5 redaction tests:
    • test_dump_redacts_sk_proj_key_in_authorization_header
    • test_dump_redacts_keys_embedded_in_request_body
    • test_dump_redacts_keys_in_error_response_text
    • test_dump_preserves_non_secret_metadata
    • test_dump_remains_valid_json_after_redaction (regression for the OPENAI_API_KEY=sk-proj-... in body case that, with text-level redaction, would have produced unparseable JSON)

How to Test

  1. python -m pytest tests/run_agent/test_run_agent_codex_responses.py -k dump -q — 7/7 pass (5 new + 2 existing URL tests)
  2. python -m pytest tests/run_agent/test_run_agent_codex_responses.py -q — full file 64/64 pass
  3. Manual: set OPENAI_API_KEY to a valid-format-but-revoked key, send any chat message; inspect the resulting request_dump_*.json — Authorization shows Bearer ***, message bodies with embedded sk-... strings show masked tokens, and json.loads(open(dump_file).read()) succeeds

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 15 (Darwin 24.6.0)

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

python -m pytest tests/run_agent/test_run_agent_codex_responses.py -k dump -q
# 7/7 pass (5 new + 2 existing URL tests)

python -m pytest tests/run_agent/test_run_agent_codex_responses.py -q
# 64/64 pass

@alt-glitch alt-glitch added type/security Security vulnerability or hardening P1 High — major feature broken, no workaround comp/agent Core agent loop, run_agent.py, prompt builder labels May 2, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related to #8518 and #18707 — this PR implements the full fix for secret redaction in request_dump_*.json files. Supersedes #8533.

@alt-glitch

Copy link
Copy Markdown
Collaborator

Related to #8518 and #18707.

@Tranquil-Flow Tranquil-Flow force-pushed the fix/18707-redact-request-dump branch 2 times, most recently from 8d59465 to 0b2c834 Compare May 25, 2026 14:03
@Tranquil-Flow Tranquil-Flow force-pushed the fix/18707-redact-request-dump branch from 0b2c834 to 3bc64fc Compare May 25, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent loop, run_agent.py, prompt builder P1 High — major feature broken, no workaround type/security Security vulnerability or hardening

Projects

None yet

2 participants