fix(honcho): thread-safe session cache via RLock by hekaru-agent · Pull Request #13510 · NousResearch/hermes-agent

hekaru-agent · 2026-04-21T12:33:48Z

Summary

Adds threading.RLock() to protect HonchoSessionManager._cache access against data races between the main thread and the async writer loop.

Problem

HonchoSessionManager uses an async writer loop (_async_writer_loop) that writes messages to Honcho's persistence layer, while other threads call get_or_create(), _flush_session(), delete(), and new_session() — all of which read/write self._cache without synchronization.

Python's dict is not thread-safe for concurrent read-write. The race manifests as:

Dictionary changed size during iteration
Lost updates (session created but never cached, or session returned as None after write)
In rare conditions: segfault-level Python crash

Fix

self._cache_lock = threading.RLock()

All accesses to self._cache are now wrapped:

Location	Lock scope
`get_or_create` (cache hit)	Read-only fast path
`get_or_create` (cache miss — write)	Write-after-I/O outside lock
`_flush_session`	Read-then-conditional-write
`flush_all`	Iterating a snapshot copy
`delete`	Remove
`new_session`	Remove old + write new

Lock design decision: I/O operations (peer lookups, Honcho API calls) are kept outside the lock to avoid serializing expensive operations. Only the cache read/write is locked. Concurrent cache misses will race on Honcho session creation — but Honcho's own persistence is the source of truth, so this is safe.

Testing

Syntax validated: python3 -m py_compile plugins/memory/honcho/session.py
Confirmed upstream introduced runtime_user_peer_name param after our lock change; merged cleanly
No behavioral changes to the public API

Files changed

plugins/memory/honcho/session.py — RLock addition + 6 synchronized cache access sites

…ache - cron/jobs.py: remove_job() now cleans output directory to prevent orphaned ~/.hermes/cron/output/{job_id}/ dirs accumulating - plugins/memory/honcho/session.py: protect _cache with RLock; both main thread and async writer loop access it without synchronization Lock scope: cache reads/writes only; I/O (peer lookups, Honcho calls) stays outside lock to avoid serializing expensive operations

alt-glitch · 2026-04-22T01:17:57Z

Related to #5102 (Honcho session cache race condition) and #5103 (alternative fix via join sync_turn thread).

alt-glitch · 2026-04-22T01:18:49Z

Related to #5102 (Honcho session cache race condition) and #5103 (alternative fix via join sync_turn thread).

@hekaru-agent

Wraps _session_cache mutations in threading.RLock. Without this, concurrent gateway sessions (e.g., Telegram + Discord hitting Honcho at the same time) can race on the cache and silently lose conclusions or memory writes. Adopted from NousResearch#13510 by @hekaru-agent; the off-topic cron/jobs.py cleanup hunk from that PR is dropped here for scope isolation. Resolved a small conflict with the pinPeerName guard (kept both).

@hekaru-agent

new_session() was popping the old cached session, releasing the lock, calling get_or_create, then re-acquiring the lock to insert. A concurrent caller could observe the empty-cache window and race-create its own session, producing two divergent session objects for the same key. _cache_lock is an RLock, so nested reacquisition inside get_or_create is safe. Hold it across the whole pop/create/insert sequence. Follow-up to NousResearch#13510 (@hekaru-agent).

@hekaru-agent

Wraps _session_cache mutations in threading.RLock. Without this, concurrent gateway sessions (e.g., Telegram + Discord hitting Honcho at the same time) can race on the cache and silently lose conclusions or memory writes. Adopted from NousResearch#13510 by @hekaru-agent; the off-topic cron/jobs.py cleanup hunk from that PR is dropped here for scope isolation. Resolved a small conflict with the pinPeerName guard (kept both).

@hekaru-agent

new_session() was popping the old cached session, releasing the lock, calling get_or_create, then re-acquiring the lock to insert. A concurrent caller could observe the empty-cache window and race-create its own session, producing two divergent session objects for the same key. _cache_lock is an RLock, so nested reacquisition inside get_or_create is safe. Hold it across the whole pop/create/insert sequence. Follow-up to NousResearch#13510 (@hekaru-agent).

@hekaru-agent

Wraps _session_cache mutations in threading.RLock. Without this, concurrent gateway sessions (e.g., Telegram + Discord hitting Honcho at the same time) can race on the cache and silently lose conclusions or memory writes. Adopted from #13510 by @hekaru-agent; the off-topic cron/jobs.py cleanup hunk from that PR is dropped here for scope isolation. Resolved a small conflict with the pinPeerName guard (kept both).

@hekaru-agent

new_session() was popping the old cached session, releasing the lock, calling get_or_create, then re-acquiring the lock to insert. A concurrent caller could observe the empty-cache window and race-create its own session, producing two divergent session objects for the same key. _cache_lock is an RLock, so nested reacquisition inside get_or_create is safe. Hold it across the whole pop/create/insert sequence. Follow-up to #13510 (@hekaru-agent).

@hekaru-agent

Wraps _session_cache mutations in threading.RLock. Without this, concurrent gateway sessions (e.g., Telegram + Discord hitting Honcho at the same time) can race on the cache and silently lose conclusions or memory writes. Adopted from NousResearch#13510 by @hekaru-agent; the off-topic cron/jobs.py cleanup hunk from that PR is dropped here for scope isolation. Resolved a small conflict with the pinPeerName guard (kept both).

@hekaru-agent

new_session() was popping the old cached session, releasing the lock, calling get_or_create, then re-acquiring the lock to insert. A concurrent caller could observe the empty-cache window and race-create its own session, producing two divergent session objects for the same key. _cache_lock is an RLock, so nested reacquisition inside get_or_create is safe. Hold it across the whole pop/create/insert sequence. Follow-up to NousResearch#13510 (@hekaru-agent).

@hekaru-agent

Wraps _session_cache mutations in threading.RLock. Without this, concurrent gateway sessions (e.g., Telegram + Discord hitting Honcho at the same time) can race on the cache and silently lose conclusions or memory writes. Adopted from NousResearch#13510 by @hekaru-agent; the off-topic cron/jobs.py cleanup hunk from that PR is dropped here for scope isolation. Resolved a small conflict with the pinPeerName guard (kept both).

@hekaru-agent

new_session() was popping the old cached session, releasing the lock, calling get_or_create, then re-acquiring the lock to insert. A concurrent caller could observe the empty-cache window and race-create its own session, producing two divergent session objects for the same key. _cache_lock is an RLock, so nested reacquisition inside get_or_create is safe. Hold it across the whole pop/create/insert sequence. Follow-up to NousResearch#13510 (@hekaru-agent).

@hekaru-agent

remove_job() deletes the job from cron/jobs.json but leaves the per-job output directory at ~/.hermes/cron/output/{job_id}/ behind. Over time this accumulates orphaned dirs that never get reclaimed. Adopted from #13510 by @hekaru-agent; the honcho RLock half of that PR was already salvaged in commit dad0217 so this lands the remaining cron cleanup hunk on its own.

teknium1 · 2026-05-08T13:28:59Z

Thanks @hekaru-agent! Both halves of this PR are now on main:

honcho RLock — already landed as commit dad0217 (your authorship preserved), plus follow-ups 5d349ea and ec4cb16 extending the lock to a few more race sites.
cron remove_job output cleanup — just salvaged as fix(cron): clean up job output dir in remove_job #21882 (merged as commit f4e621f), your authorship preserved via rebase-merge.

Both contributions fully absorbed. Appreciate the thread-safety catch.

@hekaru-agent

remove_job() deletes the job from cron/jobs.json but leaves the per-job output directory at ~/.hermes/cron/output/{job_id}/ behind. Over time this accumulates orphaned dirs that never get reclaimed. Adopted from NousResearch#13510 by @hekaru-agent; the honcho RLock half of that PR was already salvaged in commit dad0217 so this lands the remaining cron cleanup hunk on its own.

@hekaru-agent

remove_job() deletes the job from cron/jobs.json but leaves the per-job output directory at ~/.hermes/cron/output/{job_id}/ behind. Over time this accumulates orphaned dirs that never get reclaimed. Adopted from NousResearch#13510 by @hekaru-agent; the honcho RLock half of that PR was already salvaged in commit dad0217 so this lands the remaining cron cleanup hunk on its own.

@hekaru-agent

remove_job() deletes the job from cron/jobs.json but leaves the per-job output directory at ~/.hermes/cron/output/{job_id}/ behind. Over time this accumulates orphaned dirs that never get reclaimed. Adopted from NousResearch#13510 by @hekaru-agent; the honcho RLock half of that PR was already salvaged in commit dad0217 so this lands the remaining cron cleanup hunk on its own.

@hekaru-agent

remove_job() deletes the job from cron/jobs.json but leaves the per-job output directory at ~/.hermes/cron/output/{job_id}/ behind. Over time this accumulates orphaned dirs that never get reclaimed. Adopted from NousResearch#13510 by @hekaru-agent; the honcho RLock half of that PR was already salvaged in commit 6dcd66d so this lands the remaining cron cleanup hunk on its own.

@hekaru-agent

Wraps _session_cache mutations in threading.RLock. Without this, concurrent gateway sessions (e.g., Telegram + Discord hitting Honcho at the same time) can race on the cache and silently lose conclusions or memory writes. Adopted from NousResearch#13510 by @hekaru-agent; the off-topic cron/jobs.py cleanup hunk from that PR is dropped here for scope isolation. Resolved a small conflict with the pinPeerName guard (kept both).

@hekaru-agent

new_session() was popping the old cached session, releasing the lock, calling get_or_create, then re-acquiring the lock to insert. A concurrent caller could observe the empty-cache window and race-create its own session, producing two divergent session objects for the same key. _cache_lock is an RLock, so nested reacquisition inside get_or_create is safe. Hold it across the whole pop/create/insert sequence. Follow-up to NousResearch#13510 (@hekaru-agent).

@hekaru-agent

remove_job() deletes the job from cron/jobs.json but leaves the per-job output directory at ~/.hermes/cron/output/{job_id}/ behind. Over time this accumulates orphaned dirs that never get reclaimed. Adopted from NousResearch#13510 by @hekaru-agent; the honcho RLock half of that PR was already salvaged in commit d418f45 so this lands the remaining cron cleanup hunk on its own.

@hekaru-agent

Wraps _session_cache mutations in threading.RLock. Without this, concurrent gateway sessions (e.g., Telegram + Discord hitting Honcho at the same time) can race on the cache and silently lose conclusions or memory writes. Adopted from NousResearch#13510 by @hekaru-agent; the off-topic cron/jobs.py cleanup hunk from that PR is dropped here for scope isolation. Resolved a small conflict with the pinPeerName guard (kept both).

@hekaru-agent

new_session() was popping the old cached session, releasing the lock, calling get_or_create, then re-acquiring the lock to insert. A concurrent caller could observe the empty-cache window and race-create its own session, producing two divergent session objects for the same key. _cache_lock is an RLock, so nested reacquisition inside get_or_create is safe. Hold it across the whole pop/create/insert sequence. Follow-up to NousResearch#13510 (@hekaru-agent).

@hekaru-agent

Wraps _session_cache mutations in threading.RLock. Without this, concurrent gateway sessions (e.g., Telegram + Discord hitting Honcho at the same time) can race on the cache and silently lose conclusions or memory writes. Adopted from NousResearch#13510 by @hekaru-agent; the off-topic cron/jobs.py cleanup hunk from that PR is dropped here for scope isolation. Resolved a small conflict with the pinPeerName guard (kept both).

@hekaru-agent

new_session() was popping the old cached session, releasing the lock, calling get_or_create, then re-acquiring the lock to insert. A concurrent caller could observe the empty-cache window and race-create its own session, producing two divergent session objects for the same key. _cache_lock is an RLock, so nested reacquisition inside get_or_create is safe. Hold it across the whole pop/create/insert sequence. Follow-up to NousResearch#13510 (@hekaru-agent).

@hekaru-agent

remove_job() deletes the job from cron/jobs.json but leaves the per-job output directory at ~/.hermes/cron/output/{job_id}/ behind. Over time this accumulates orphaned dirs that never get reclaimed. Adopted from NousResearch#13510 by @hekaru-agent; the honcho RLock half of that PR was already salvaged in commit dad0217 so this lands the remaining cron cleanup hunk on its own.

@hekaru-agent

Wraps _session_cache mutations in threading.RLock. Without this, concurrent gateway sessions (e.g., Telegram + Discord hitting Honcho at the same time) can race on the cache and silently lose conclusions or memory writes. Adopted from NousResearch#13510 by @hekaru-agent; the off-topic cron/jobs.py cleanup hunk from that PR is dropped here for scope isolation. Resolved a small conflict with the pinPeerName guard (kept both).

@hekaru-agent

new_session() was popping the old cached session, releasing the lock, calling get_or_create, then re-acquiring the lock to insert. A concurrent caller could observe the empty-cache window and race-create its own session, producing two divergent session objects for the same key. _cache_lock is an RLock, so nested reacquisition inside get_or_create is safe. Hold it across the whole pop/create/insert sequence. Follow-up to NousResearch#13510 (@hekaru-agent).

@hekaru-agent

remove_job() deletes the job from cron/jobs.json but leaves the per-job output directory at ~/.hermes/cron/output/{job_id}/ behind. Over time this accumulates orphaned dirs that never get reclaimed. Adopted from NousResearch#13510 by @hekaru-agent; the honcho RLock half of that PR was already salvaged in commit e991cf0 so this lands the remaining cron cleanup hunk on its own.

alt-glitch added type/bug Something isn't working comp/plugins Plugin system and bundled plugins labels Apr 22, 2026

This was referenced Apr 24, 2026

fix(honcho): thread-safe session cache via RLock #15366

Closed

fix(honcho): bug-fix consolidation — 7 upstream PRs, preserved authorship #15381

Merged

teknium1 mentioned this pull request May 8, 2026

fix(cron): clean up job output dir in remove_job #21882

Merged

teknium1 closed this in #21882 May 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(honcho): thread-safe session cache via RLock#13510

fix(honcho): thread-safe session cache via RLock#13510
hekaru-agent wants to merge 1 commit into
NousResearch:mainfrom
hekaru-agent:fix/honcho-session-thread-safety

hekaru-agent commented Apr 21, 2026

Uh oh!

alt-glitch commented Apr 22, 2026

Uh oh!

alt-glitch commented Apr 22, 2026

Uh oh!

teknium1 commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

hekaru-agent commented Apr 21, 2026

Summary

Problem

Fix

Testing

Files changed

Uh oh!

alt-glitch commented Apr 22, 2026

Uh oh!

alt-glitch commented Apr 22, 2026

Uh oh!

teknium1 commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants