Skip to content

Conversation

@lhotari
Copy link
Member

@lhotari lhotari commented Jul 16, 2025

Motivation

"PIP-165 Auto release client useless connections" added a solution that closes unused idling connections. The idle check doesn't consider topic watchers created by a topics pattern (regex) consumer. The impact of this is that connections would get frequently closed and immediately reconnected when topics pattern consumers are used and when the topic watcher is the only resource using the connection.

While reviewing the code, another possible race condition was observed in the solution due lack of usage of volatile for ClientCnxIdleState's idleMarkTime field. The field gets mutated and read in different threads in certain cases.
There was also a concern that a currently used connection could get released under a race condition. That's why this PR adds another check before the ClientCnxIdleState can move from IDLE to RELEASING.

Modifications

  • fix the ClientCnx idleCheck method
    • check topic list watchers
  • make ClientCnxIdleState's idleMarkTime field volatile
  • add extra idleCheck in ClientCnxIdleState's doIdleDetect before the state can change from IDLE to RELEASING, switch to USING if the connection is getting used.
  • adds test coverage at different levels. There are also tests for the usage of Pulsar Proxy + multiple brokers.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@lhotari lhotari added this to the 4.1.0 milestone Jul 16, 2025
@lhotari lhotari self-assigned this Jul 16, 2025
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jul 16, 2025
@lhotari lhotari requested review from dao-jun and nodece July 16, 2025 18:47
@codecov-commenter
Copy link

codecov-commenter commented Jul 16, 2025

Codecov Report

❌ Patch coverage is 80.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 74.31%. Comparing base (bbc6224) to head (2a4d454).
⚠️ Report is 1311 commits behind head on master.

Files with missing lines Patch % Lines
.../apache/pulsar/client/impl/ClientCnxIdleState.java 66.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #24528      +/-   ##
============================================
+ Coverage     73.57%   74.31%   +0.74%     
- Complexity    32624    32872     +248     
============================================
  Files          1877     1868       -9     
  Lines        139502   145946    +6444     
  Branches      15299    16737    +1438     
============================================
+ Hits         102638   108465    +5827     
+ Misses        28908    28866      -42     
- Partials       7956     8615     +659     
Flag Coverage Δ
inttests 26.72% <0.00%> (+2.14%) ⬆️
systests 23.33% <0.00%> (-0.99%) ⬇️
unittests 73.80% <80.00%> (+0.95%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
.../java/org/apache/pulsar/client/impl/ClientCnx.java 69.55% <100.00%> (-2.22%) ⬇️
.../apache/pulsar/client/impl/ClientCnxIdleState.java 68.75% <66.66%> (-2.37%) ⬇️

... and 1096 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lhotari lhotari merged commit 8be127f into apache:master Jul 17, 2025
54 checks passed
lhotari added a commit that referenced this pull request Jul 17, 2025
…ics pattern consumer (#24528)

(cherry picked from commit 8be127f)
lhotari added a commit that referenced this pull request Jul 17, 2025
…ics pattern consumer (#24528)

(cherry picked from commit 8be127f)
BewareMyPower pushed a commit that referenced this pull request Jul 17, 2025
priyanshu-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 22, 2025
…ics pattern consumer (apache#24528)

(cherry picked from commit 8be127f)
(cherry picked from commit f16b757)
priyanshu-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 22, 2025
…ics pattern consumer (apache#24528)

(cherry picked from commit 8be127f)
(cherry picked from commit 1c1bf66)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 23, 2025
…ics pattern consumer (apache#24528)

(cherry picked from commit 8be127f)
(cherry picked from commit f16b757)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 24, 2025
…ics pattern consumer (apache#24528)

(cherry picked from commit 8be127f)
(cherry picked from commit 1c1bf66)
nodece pushed a commit to ascentstream/pulsar that referenced this pull request Jul 28, 2025
nodece pushed a commit to ascentstream/pulsar that referenced this pull request Jul 28, 2025
KannarFr pushed a commit to CleverCloud/pulsar that referenced this pull request Sep 22, 2025
walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants