fix(gateway): prevent re-flushing expired sessions on restart#2510
Closed
dlkakbs wants to merge 2 commits into
Closed
fix(gateway): prevent re-flushing expired sessions on restart#2510dlkakbs wants to merge 2 commits into
dlkakbs wants to merge 2 commits into
Conversation
Previously, flushed session IDs were tracked in an in-memory set (_pre_flushed_sessions) that reset on gateway restart. Expired sessions remained in sessions.json, causing redundant LLM API calls for memory flush on every restart. Fix: After successful memory flush, remove the session entry from _entries and persist to disk. This eliminates the re-flush storm since expired sessions no longer exist in storage after being processed. Cleanup: Remove the now-unused _pre_flushed_sessions set and related code from run.py, session.py, and update tests accordingly. Fixes NousResearch#2506
bdff95c to
355e257
Compare
Contributor
|
Thanks @dlkakbs — you were the first to identify this bug and propose the entry-removal approach. The fix landed in #4481, which preserves session entries in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2506
The background
_session_expiry_watcherrepeatedly flushes memories for the same expired sessions after every gateway restart, causing a flood of LLM API calls and log noise.Root Cause
_pre_flushed_sessionsis an in-memoryset()that resets on gateway restart. Flushed sessions remain insessions.json, so every expired session is re-flushed on restart → O(N) LLM API calls.Fix
After successful memory flush, remove the session entry from
_entriesand persist to disk. Expired sessions no longer exist in storage after being processed.Cleanup
Removed the now-unused
_pre_flushed_sessionsset and related code from:gateway/run.py- skip check and add() callgateway/session.py- set definition and discard() calltests/gateway/test_async_memory_flush.py- updated testsTesting
test_async_memory_flush.py- 7/7 PASSED