Skip to content

fix: prevent Hindsight retain work during interpreter shutdown#15512

Open
vominh1919 wants to merge 1 commit into
NousResearch:mainfrom
vominh1919:fix/hindsight-shutdown-guard
Open

fix: prevent Hindsight retain work during interpreter shutdown#15512
vominh1919 wants to merge 1 commit into
NousResearch:mainfrom
vominh1919:fix/hindsight-shutdown-guard

Conversation

@vominh1919

Copy link
Copy Markdown
Contributor

Fixes #15497

Problem

The HindsightMemoryProvider can still submit async retain work after provider shutdown has started. During interpreter teardown, asyncio.run_coroutine_threadsafe() fails with RuntimeError: cannot schedule new futures after interpreter shutdown.

Root Cause

shutdown() joins existing threads but has no guard flag to prevent sync_turn() from accepting new work.

Fix

  1. Added self._shutting_down = False in __init__() — initializes the guard flag
  2. Added early-return guard in sync_turn() — checks self._shutting_down before doing any work
  3. Set self._shutting_down = True at top of shutdown() — marks provider as shutting down before joining threads
  4. Defense-in-depth in _run_sync() — checks loop.is_closed() before calling asyncio.run_coroutine_threadsafe()

Files Changed

  • plugins/memory/hindsight/__init__.py — 7 lines added, 0 removed

Add shutdown guard to sync_turn() and defense-in-depth loop.is_closed()
check in _run_sync() to prevent RuntimeError during interpreter teardown.

Fixes NousResearch#15497
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/plugins Plugin system and bundled plugins labels Apr 25, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Likely duplicate of #15507 — both fix Hindsight retain-during-shutdown race for #15497. #15507 has a more comprehensive approach (lifecycle state gating, extracted _submit_retain helper, shutdown flush).

@nicoloboschi

Copy link
Copy Markdown
Contributor

Superseded by #17005 (merged) which adopts a single-writer queue + atexit drain — a more comprehensive fix for the same shutdown race. Could you close?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hindsight provider can submit retain work during interpreter shutdown

3 participants