fix: prevent session leaks and exhaustion in smart router and consumer#2158
Merged
Conversation
6b8c935 to
ee5f0d7
Compare
There was a problem hiding this comment.
This PR is being reviewed by Cursor Bugbot
Details
You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
c00ae8c to
af7f82a
Compare
Test Results 6 files ± 0 129 suites ±0 33m 35s ⏱️ - 1m 19s For more details on these failures and errors, see this check. Results for commit 42872f0. ± Comparison against base commit ac492c9. ♻️ This comment has been updated with latest results. |
1c7759d to
eb63eb6
Compare
c406f55 to
5bd7018
Compare
…d (1000+ r/s) with a single provider in smart router mode. Root causes fixed: 1. Sessions held for 3 seconds during backoff (rpcconsumer + rpcsmartrouter) - Sessions now released immediately on relay failure - Backoff no longer holds session locked 2. OnSessionFailure early return without Free() (consumer_session_manager) - When session was already blocklisted, returned without unlocking - Session stayed locked forever causing exhaustion - Now calls Free() before returning in blocklisted case 3. Data reliability failures with single provider - DR needs at least 2 providers to compare results - Added early exit when < 2 providers available 4. Defer-based cleanup for all exit paths - sessionHandled flag prevents double-free - Panic recovery ensures session release 5. Configurable max sessions per provider - Added --max-sessions-per-provider flag (default: 1000) 6. Configurable max streams per connection - Added --maximum-streams-per-connection flag (default: 100) Files modified: - protocol/lavasession/consumer_session_manager.go - protocol/lavasession/consumer_types.go - protocol/lavasession/common.go - protocol/common/cobra_common.go - protocol/rpcconsumer/rpcconsumer.go - protocol/rpcconsumer/rpcconsumer_server.go - protocol/rpcsmartrouter/rpcsmartrouter.go - protocol/rpcsmartrouter/rpcsmartrouter_server.go Tests added: - TestSessionLeakPrevention_* in rpcconsumer
0acd93e to
f6887da
Compare
…safety in RPC session management Updated the session handling mechanism in both rpcconsumer and rpcsmartrouter servers to use atomic.Bool instead of a regular boolean. This change ensures thread-safe access to the sessionHandled flag, preventing potential race conditions during session cleanup and management. The updates include initializing the atomic variable and modifying all relevant checks and updates to use the atomic methods.
avitenzer
approved these changes
Dec 31, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit fixes session exhaustion issues that occur under high load (1000+ r/s) with a single provider in smart router mode.
Root causes fixed:
Sessions held for 3 seconds during backoff (rpcconsumer + rpcsmartrouter)
OnSessionFailure early return without Free() (consumer_session_manager)
Data reliability failures with single provider
Defer-based cleanup for all exit paths
Configurable max sessions per provider
Files modified:
Tests added:
Description
Closes: #XXXX
Author Checklist
All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.
I have...
!in the type prefix if API or client breaking changemainbranchReviewers Checklist
All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.
I have...
Note
Resolves session exhaustion under load by making session handling robust and configurable.
sessionHandledand panic recovery insendRelayToProvider(consumer + smart router)OnSessionFailureimmediately on relay errors (removed backoff-held sessions)OnSessionFailure, free session when already blocklisted to avoid deadlocks--max-sessions-per-provider(backsMaxSessionsAllowedPerProvider) and--maximum-streams-per-connectionGetMaxAllowedBlockListedSessionPerProvider; compute blocked-session thresholds using itMaximumStreamsOverASingleConnectionin connection reuse checksrpcconsumerandrpcsmartrouter; centralize flag names incommonTestSessionLeakPrevention_*suites for consumer and smart router (panic, early-return, concurrency, subscriptions)Written by Cursor Bugbot for commit 3779858. This will update automatically on new commits. Configure here.