fix(nous-oauth): preserve obtained_at in pool + actionable message on RT reuse by teknium1 · Pull Request #15111 · NousResearch/hermes-agent

teknium1 · 2026-04-24T12:06:34Z

What this PR does

Two narrow Nous OAuth fixes, both motivated by @camelludo's #15099 report. After tracing the refresh-token persistence path end-to-end I couldn't find the 'client silently drops rotated RT' bug the report hypothesised — the pool + singleton write-back machinery is actually correct. What IS broken is narrower, and the report gives us enough evidence to fix both issues completely.

Fix 1: `_seed_from_singletons()` preserves mint/refresh timestamps

When the pool is seeded from the providers.nous singleton, the upsert payload was missing obtained_at, agent_key_obtained_at, expires_in, agent_key_expires_in, agent_key_id, and agent_key_reused. Fresh credentials showed up with obtained_at: None, which breaks freshness-sensitive consumers (self-heal hooks, pool pruning by age) — they treat just-minted credentials as older than they are. @camelludo explicitly called this out as the 'secondary issue' in the report.

Fix 2: Actionable message on `invalid_grant: refresh token reuse detected`

The Nous Portal server implements OAuth 2.1 RT rotation and returns invalid_grant: Refresh token reuse detected when a rotated RT gets used a second time. In real-world reports this almost always means an external process (monitoring script, custom self-heal hook, another install sharing ~/.hermes/auth.json) called POST /api/oauth/token with Hermes's RT without persisting the rotated value back. The generic reuse message gave users no clue about the external-process cause, so they reported it as a Hermes persistence bug.

The PR detects that specific error signature and rewrites it to:

Nous Portal detected refresh-token reuse and revoked this session.
This usually means an external process (monitoring script, custom self-heal hook, or another Hermes install sharing ~/.hermes/auth.json) called POST /api/oauth/token with Hermes's refresh token without persisting the rotated token back.
Nous refresh tokens are single-use — only Hermes may call the refresh endpoint. For health checks, use hermes auth status instead.
Re-authenticate with: hermes auth add nous

Generic invalid_grant errors (e.g. Refresh session has been revoked) keep their original server descriptions untouched.

Note on the reporter's primary diagnosis

@camelludo's report hypothesised that Hermes wasn't persisting the rotated RT. I traced this carefully and the persistence is working correctly:

refresh_nous_oauth_pure() persists the new RT into state (line 2118)
resolve_nous_runtime_credentials() holds _auth_store_lock() across the HTTP call and _persist_state()s immediately after rotation (lines 2359, 2426)
pool._refresh_entry() calls _sync_device_code_entry_to_auth_store() so the pool and singleton stay in sync

Their own comment 2 describes the actual trigger: a systemd-timer-triggered aliveness script POSTing to /api/oauth/token with each profile's RT, explicitly discarding the response. That's RT reuse — their script consumed RT1, got RT2, threw RT2 away, and the gateway's next refresh with RT1 triggered the OAuth server's reuse detection. Fix 2 makes that specific failure mode unmistakable in the error message so future reports can self-diagnose.

Validation

tests/agent/test_credential_pool.py — 45/45 pass (1 new)
tests/hermes_cli/test_auth_nous_provider.py — 12/12 pass (2 new)
E2E-verified: seeded singleton with obtained_at → pool entry keeps it; fake 400-with-reuse response → actionable rewritten message; fake 400-without-reuse → original description unchanged

… RT reuse Two narrow fixes motivated by #15099. 1. _seed_from_singletons() was dropping obtained_at, agent_key_obtained_at, expires_in, and friends when seeding device_code pool entries from the providers.nous singleton. Fresh credentials showed up with obtained_at=None, which broke downstream freshness-sensitive consumers (self-heal hooks, pool pruning by age) — they treated just-minted credentials as older than they actually were and evicted them. 2. When the Nous Portal OAuth 2.1 server returns invalid_grant with 'Refresh token reuse detected' in the error_description, rewrite the message to explain the likely cause (an external process consumed the rotated RT without persisting it back) and the mitigation. The generic reuse message led users to report this as a Hermes persistence bug when the actual trigger was typically a third-party monitoring script calling /api/oauth/token directly. Non-reuse errors keep their original server description untouched. Closes #15099. Regression tests: - tests/agent/test_credential_pool.py::test_nous_seed_from_singletons_preserves_obtained_at_timestamps - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_token_reuse_detection_surfaces_actionable_message - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_non_reuse_error_keeps_original_description

@konsisumer

Concurrent Hermes processes (e.g. cron jobs) refreshing a Nous OAuth token via resolve_nous_runtime_credentials() write the rotated tokens to auth.json. The calling process's pool entry becomes stale, and the next refresh against the already-rotated token triggers a 'refresh token reuse' revocation on the Nous Portal. _sync_nous_entry_from_auth_store() reads auth.json under the same lock used by resolve_nous_runtime_credentials, and adopts the newer token pair before refreshing the pool entry. This complements #15111 (which preserved the obtained_at timestamps through seeding). Partial salvage of #10160 by @konsisumer — only the agent/credential_pool.py changes + the 3 Nous-specific regression tests. The PR also touched 10 unrelated files (Dockerfile, tips.py, various tool tests) which were dropped as scope creep. Regression tests: - test_sync_nous_entry_from_auth_store_adopts_newer_tokens - test_sync_nous_entry_noop_when_tokens_match - test_nous_exhausted_entry_recovers_via_auth_store_sync

… RT reuse (NousResearch#15111) Two narrow fixes motivated by NousResearch#15099. 1. _seed_from_singletons() was dropping obtained_at, agent_key_obtained_at, expires_in, and friends when seeding device_code pool entries from the providers.nous singleton. Fresh credentials showed up with obtained_at=None, which broke downstream freshness-sensitive consumers (self-heal hooks, pool pruning by age) — they treated just-minted credentials as older than they actually were and evicted them. 2. When the Nous Portal OAuth 2.1 server returns invalid_grant with 'Refresh token reuse detected' in the error_description, rewrite the message to explain the likely cause (an external process consumed the rotated RT without persisting it back) and the mitigation. The generic reuse message led users to report this as a Hermes persistence bug when the actual trigger was typically a third-party monitoring script calling /api/oauth/token directly. Non-reuse errors keep their original server description untouched. Closes NousResearch#15099. Regression tests: - tests/agent/test_credential_pool.py::test_nous_seed_from_singletons_preserves_obtained_at_timestamps - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_token_reuse_detection_surfaces_actionable_message - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_non_reuse_error_keeps_original_description

@konsisumer

Concurrent Hermes processes (e.g. cron jobs) refreshing a Nous OAuth token via resolve_nous_runtime_credentials() write the rotated tokens to auth.json. The calling process's pool entry becomes stale, and the next refresh against the already-rotated token triggers a 'refresh token reuse' revocation on the Nous Portal. _sync_nous_entry_from_auth_store() reads auth.json under the same lock used by resolve_nous_runtime_credentials, and adopts the newer token pair before refreshing the pool entry. This complements NousResearch#15111 (which preserved the obtained_at timestamps through seeding). Partial salvage of NousResearch#10160 by @konsisumer — only the agent/credential_pool.py changes + the 3 Nous-specific regression tests. The PR also touched 10 unrelated files (Dockerfile, tips.py, various tool tests) which were dropped as scope creep. Regression tests: - test_sync_nous_entry_from_auth_store_adopts_newer_tokens - test_sync_nous_entry_noop_when_tokens_match - test_nous_exhausted_entry_recovers_via_auth_store_sync

… RT reuse (NousResearch#15111) Two narrow fixes motivated by NousResearch#15099. 1. _seed_from_singletons() was dropping obtained_at, agent_key_obtained_at, expires_in, and friends when seeding device_code pool entries from the providers.nous singleton. Fresh credentials showed up with obtained_at=None, which broke downstream freshness-sensitive consumers (self-heal hooks, pool pruning by age) — they treated just-minted credentials as older than they actually were and evicted them. 2. When the Nous Portal OAuth 2.1 server returns invalid_grant with 'Refresh token reuse detected' in the error_description, rewrite the message to explain the likely cause (an external process consumed the rotated RT without persisting it back) and the mitigation. The generic reuse message led users to report this as a Hermes persistence bug when the actual trigger was typically a third-party monitoring script calling /api/oauth/token directly. Non-reuse errors keep their original server description untouched. Closes NousResearch#15099. Regression tests: - tests/agent/test_credential_pool.py::test_nous_seed_from_singletons_preserves_obtained_at_timestamps - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_token_reuse_detection_surfaces_actionable_message - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_non_reuse_error_keeps_original_description

@konsisumer

Concurrent Hermes processes (e.g. cron jobs) refreshing a Nous OAuth token via resolve_nous_runtime_credentials() write the rotated tokens to auth.json. The calling process's pool entry becomes stale, and the next refresh against the already-rotated token triggers a 'refresh token reuse' revocation on the Nous Portal. _sync_nous_entry_from_auth_store() reads auth.json under the same lock used by resolve_nous_runtime_credentials, and adopts the newer token pair before refreshing the pool entry. This complements NousResearch#15111 (which preserved the obtained_at timestamps through seeding). Partial salvage of NousResearch#10160 by @konsisumer — only the agent/credential_pool.py changes + the 3 Nous-specific regression tests. The PR also touched 10 unrelated files (Dockerfile, tips.py, various tool tests) which were dropped as scope creep. Regression tests: - test_sync_nous_entry_from_auth_store_adopts_newer_tokens - test_sync_nous_entry_noop_when_tokens_match - test_nous_exhausted_entry_recovers_via_auth_store_sync

… RT reuse (NousResearch#15111) Two narrow fixes motivated by NousResearch#15099. 1. _seed_from_singletons() was dropping obtained_at, agent_key_obtained_at, expires_in, and friends when seeding device_code pool entries from the providers.nous singleton. Fresh credentials showed up with obtained_at=None, which broke downstream freshness-sensitive consumers (self-heal hooks, pool pruning by age) — they treated just-minted credentials as older than they actually were and evicted them. 2. When the Nous Portal OAuth 2.1 server returns invalid_grant with 'Refresh token reuse detected' in the error_description, rewrite the message to explain the likely cause (an external process consumed the rotated RT without persisting it back) and the mitigation. The generic reuse message led users to report this as a Hermes persistence bug when the actual trigger was typically a third-party monitoring script calling /api/oauth/token directly. Non-reuse errors keep their original server description untouched. Closes NousResearch#15099. Regression tests: - tests/agent/test_credential_pool.py::test_nous_seed_from_singletons_preserves_obtained_at_timestamps - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_token_reuse_detection_surfaces_actionable_message - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_non_reuse_error_keeps_original_description

@konsisumer

Concurrent Hermes processes (e.g. cron jobs) refreshing a Nous OAuth token via resolve_nous_runtime_credentials() write the rotated tokens to auth.json. The calling process's pool entry becomes stale, and the next refresh against the already-rotated token triggers a 'refresh token reuse' revocation on the Nous Portal. _sync_nous_entry_from_auth_store() reads auth.json under the same lock used by resolve_nous_runtime_credentials, and adopts the newer token pair before refreshing the pool entry. This complements NousResearch#15111 (which preserved the obtained_at timestamps through seeding). Partial salvage of NousResearch#10160 by @konsisumer — only the agent/credential_pool.py changes + the 3 Nous-specific regression tests. The PR also touched 10 unrelated files (Dockerfile, tips.py, various tool tests) which were dropped as scope creep. Regression tests: - test_sync_nous_entry_from_auth_store_adopts_newer_tokens - test_sync_nous_entry_noop_when_tokens_match - test_nous_exhausted_entry_recovers_via_auth_store_sync

… RT reuse (NousResearch#15111) Two narrow fixes motivated by NousResearch#15099. 1. _seed_from_singletons() was dropping obtained_at, agent_key_obtained_at, expires_in, and friends when seeding device_code pool entries from the providers.nous singleton. Fresh credentials showed up with obtained_at=None, which broke downstream freshness-sensitive consumers (self-heal hooks, pool pruning by age) — they treated just-minted credentials as older than they actually were and evicted them. 2. When the Nous Portal OAuth 2.1 server returns invalid_grant with 'Refresh token reuse detected' in the error_description, rewrite the message to explain the likely cause (an external process consumed the rotated RT without persisting it back) and the mitigation. The generic reuse message led users to report this as a Hermes persistence bug when the actual trigger was typically a third-party monitoring script calling /api/oauth/token directly. Non-reuse errors keep their original server description untouched. Closes NousResearch#15099. Regression tests: - tests/agent/test_credential_pool.py::test_nous_seed_from_singletons_preserves_obtained_at_timestamps - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_token_reuse_detection_surfaces_actionable_message - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_non_reuse_error_keeps_original_description

@konsisumer

Concurrent Hermes processes (e.g. cron jobs) refreshing a Nous OAuth token via resolve_nous_runtime_credentials() write the rotated tokens to auth.json. The calling process's pool entry becomes stale, and the next refresh against the already-rotated token triggers a 'refresh token reuse' revocation on the Nous Portal. _sync_nous_entry_from_auth_store() reads auth.json under the same lock used by resolve_nous_runtime_credentials, and adopts the newer token pair before refreshing the pool entry. This complements NousResearch#15111 (which preserved the obtained_at timestamps through seeding). Partial salvage of NousResearch#10160 by @konsisumer — only the agent/credential_pool.py changes + the 3 Nous-specific regression tests. The PR also touched 10 unrelated files (Dockerfile, tips.py, various tool tests) which were dropped as scope creep. Regression tests: - test_sync_nous_entry_from_auth_store_adopts_newer_tokens - test_sync_nous_entry_noop_when_tokens_match - test_nous_exhausted_entry_recovers_via_auth_store_sync

… RT reuse (NousResearch#15111) Two narrow fixes motivated by NousResearch#15099. 1. _seed_from_singletons() was dropping obtained_at, agent_key_obtained_at, expires_in, and friends when seeding device_code pool entries from the providers.nous singleton. Fresh credentials showed up with obtained_at=None, which broke downstream freshness-sensitive consumers (self-heal hooks, pool pruning by age) — they treated just-minted credentials as older than they actually were and evicted them. 2. When the Nous Portal OAuth 2.1 server returns invalid_grant with 'Refresh token reuse detected' in the error_description, rewrite the message to explain the likely cause (an external process consumed the rotated RT without persisting it back) and the mitigation. The generic reuse message led users to report this as a Hermes persistence bug when the actual trigger was typically a third-party monitoring script calling /api/oauth/token directly. Non-reuse errors keep their original server description untouched. Closes NousResearch#15099. Regression tests: - tests/agent/test_credential_pool.py::test_nous_seed_from_singletons_preserves_obtained_at_timestamps - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_token_reuse_detection_surfaces_actionable_message - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_non_reuse_error_keeps_original_description

@konsisumer

Concurrent Hermes processes (e.g. cron jobs) refreshing a Nous OAuth token via resolve_nous_runtime_credentials() write the rotated tokens to auth.json. The calling process's pool entry becomes stale, and the next refresh against the already-rotated token triggers a 'refresh token reuse' revocation on the Nous Portal. _sync_nous_entry_from_auth_store() reads auth.json under the same lock used by resolve_nous_runtime_credentials, and adopts the newer token pair before refreshing the pool entry. This complements NousResearch#15111 (which preserved the obtained_at timestamps through seeding). Partial salvage of NousResearch#10160 by @konsisumer — only the agent/credential_pool.py changes + the 3 Nous-specific regression tests. The PR also touched 10 unrelated files (Dockerfile, tips.py, various tool tests) which were dropped as scope creep. Regression tests: - test_sync_nous_entry_from_auth_store_adopts_newer_tokens - test_sync_nous_entry_noop_when_tokens_match - test_nous_exhausted_entry_recovers_via_auth_store_sync

… RT reuse (NousResearch#15111) Two narrow fixes motivated by NousResearch#15099. 1. _seed_from_singletons() was dropping obtained_at, agent_key_obtained_at, expires_in, and friends when seeding device_code pool entries from the providers.nous singleton. Fresh credentials showed up with obtained_at=None, which broke downstream freshness-sensitive consumers (self-heal hooks, pool pruning by age) — they treated just-minted credentials as older than they actually were and evicted them. 2. When the Nous Portal OAuth 2.1 server returns invalid_grant with 'Refresh token reuse detected' in the error_description, rewrite the message to explain the likely cause (an external process consumed the rotated RT without persisting it back) and the mitigation. The generic reuse message led users to report this as a Hermes persistence bug when the actual trigger was typically a third-party monitoring script calling /api/oauth/token directly. Non-reuse errors keep their original server description untouched. Closes NousResearch#15099. Regression tests: - tests/agent/test_credential_pool.py::test_nous_seed_from_singletons_preserves_obtained_at_timestamps - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_token_reuse_detection_surfaces_actionable_message - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_non_reuse_error_keeps_original_description

@konsisumer

Concurrent Hermes processes (e.g. cron jobs) refreshing a Nous OAuth token via resolve_nous_runtime_credentials() write the rotated tokens to auth.json. The calling process's pool entry becomes stale, and the next refresh against the already-rotated token triggers a 'refresh token reuse' revocation on the Nous Portal. _sync_nous_entry_from_auth_store() reads auth.json under the same lock used by resolve_nous_runtime_credentials, and adopts the newer token pair before refreshing the pool entry. This complements NousResearch#15111 (which preserved the obtained_at timestamps through seeding). Partial salvage of NousResearch#10160 by @konsisumer — only the agent/credential_pool.py changes + the 3 Nous-specific regression tests. The PR also touched 10 unrelated files (Dockerfile, tips.py, various tool tests) which were dropped as scope creep. Regression tests: - test_sync_nous_entry_from_auth_store_adopts_newer_tokens - test_sync_nous_entry_noop_when_tokens_match - test_nous_exhausted_entry_recovers_via_auth_store_sync

… RT reuse (NousResearch#15111) Two narrow fixes motivated by NousResearch#15099. 1. _seed_from_singletons() was dropping obtained_at, agent_key_obtained_at, expires_in, and friends when seeding device_code pool entries from the providers.nous singleton. Fresh credentials showed up with obtained_at=None, which broke downstream freshness-sensitive consumers (self-heal hooks, pool pruning by age) — they treated just-minted credentials as older than they actually were and evicted them. 2. When the Nous Portal OAuth 2.1 server returns invalid_grant with 'Refresh token reuse detected' in the error_description, rewrite the message to explain the likely cause (an external process consumed the rotated RT without persisting it back) and the mitigation. The generic reuse message led users to report this as a Hermes persistence bug when the actual trigger was typically a third-party monitoring script calling /api/oauth/token directly. Non-reuse errors keep their original server description untouched. Closes NousResearch#15099. Regression tests: - tests/agent/test_credential_pool.py::test_nous_seed_from_singletons_preserves_obtained_at_timestamps - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_token_reuse_detection_surfaces_actionable_message - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_non_reuse_error_keeps_original_description

@konsisumer

Concurrent Hermes processes (e.g. cron jobs) refreshing a Nous OAuth token via resolve_nous_runtime_credentials() write the rotated tokens to auth.json. The calling process's pool entry becomes stale, and the next refresh against the already-rotated token triggers a 'refresh token reuse' revocation on the Nous Portal. _sync_nous_entry_from_auth_store() reads auth.json under the same lock used by resolve_nous_runtime_credentials, and adopts the newer token pair before refreshing the pool entry. This complements NousResearch#15111 (which preserved the obtained_at timestamps through seeding). Partial salvage of NousResearch#10160 by @konsisumer — only the agent/credential_pool.py changes + the 3 Nous-specific regression tests. The PR also touched 10 unrelated files (Dockerfile, tips.py, various tool tests) which were dropped as scope creep. Regression tests: - test_sync_nous_entry_from_auth_store_adopts_newer_tokens - test_sync_nous_entry_noop_when_tokens_match - test_nous_exhausted_entry_recovers_via_auth_store_sync

… RT reuse (NousResearch#15111) Two narrow fixes motivated by NousResearch#15099. 1. _seed_from_singletons() was dropping obtained_at, agent_key_obtained_at, expires_in, and friends when seeding device_code pool entries from the providers.nous singleton. Fresh credentials showed up with obtained_at=None, which broke downstream freshness-sensitive consumers (self-heal hooks, pool pruning by age) — they treated just-minted credentials as older than they actually were and evicted them. 2. When the Nous Portal OAuth 2.1 server returns invalid_grant with 'Refresh token reuse detected' in the error_description, rewrite the message to explain the likely cause (an external process consumed the rotated RT without persisting it back) and the mitigation. The generic reuse message led users to report this as a Hermes persistence bug when the actual trigger was typically a third-party monitoring script calling /api/oauth/token directly. Non-reuse errors keep their original server description untouched. Closes NousResearch#15099. Regression tests: - tests/agent/test_credential_pool.py::test_nous_seed_from_singletons_preserves_obtained_at_timestamps - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_token_reuse_detection_surfaces_actionable_message - tests/hermes_cli/test_auth_nous_provider.py::test_refresh_non_reuse_error_keeps_original_description

@konsisumer

Concurrent Hermes processes (e.g. cron jobs) refreshing a Nous OAuth token via resolve_nous_runtime_credentials() write the rotated tokens to auth.json. The calling process's pool entry becomes stale, and the next refresh against the already-rotated token triggers a 'refresh token reuse' revocation on the Nous Portal. _sync_nous_entry_from_auth_store() reads auth.json under the same lock used by resolve_nous_runtime_credentials, and adopts the newer token pair before refreshing the pool entry. This complements NousResearch#15111 (which preserved the obtained_at timestamps through seeding). Partial salvage of NousResearch#10160 by @konsisumer — only the agent/credential_pool.py changes + the 3 Nous-specific regression tests. The PR also touched 10 unrelated files (Dockerfile, tips.py, various tool tests) which were dropped as scope creep. Regression tests: - test_sync_nous_entry_from_auth_store_adopts_newer_tokens - test_sync_nous_entry_noop_when_tokens_match - test_nous_exhausted_entry_recovers_via_auth_store_sync

teknium1 merged commit 78450c4 into main Apr 24, 2026
11 of 12 checks passed

teknium1 deleted the hermes/hermes-37dfb080 branch April 24, 2026 12:08

teknium1 mentioned this pull request Apr 24, 2026

fix(credential-pool): correctness + rotation + cross-process sync #15120

Merged

teknium1 mentioned this pull request Apr 24, 2026

[Bug]: Nous OAuth refresh in credential pool lacks cross-process sync — concurrent crons revoke session #10147

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(nous-oauth): preserve obtained_at in pool + actionable message on RT reuse#15111

fix(nous-oauth): preserve obtained_at in pool + actionable message on RT reuse#15111
teknium1 merged 1 commit into
mainfrom
hermes/hermes-37dfb080

teknium1 commented Apr 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

teknium1 commented Apr 24, 2026

What this PR does

Fix 1: _seed_from_singletons() preserves mint/refresh timestamps

Fix 2: Actionable message on invalid_grant: refresh token reuse detected

Note on the reporter's primary diagnosis

Validation

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Fix 1: `_seed_from_singletons()` preserves mint/refresh timestamps

Fix 2: Actionable message on `invalid_grant: refresh token reuse detected`