[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads #24542

Apurva007 · 2025-07-21T21:21:39Z

Motivation

BucketDelayedDeliveryTracker had thread safety issues in frequently called methods like containsMessage() and nextDeliveryTime() that could lead to race conditions, incorrect duplicate detection, and scheduling inconsistencies under high concurrency loads.

The issue manifested as:

Race conditions in containsMessage() leading to incorrect duplicate detection
Concurrent access issues in nextDeliveryTime() causing scheduling inconsistencies
Potential data corruption under high concurrency scenarios

Modifications

Added StampedLock for high-performance concurrency control
- Implemented optimistic read pattern for frequently called read operations
- Provides lock-free fast path when no concurrent writes are occurring
- Falls back gracefully to read locks when validation fails
Applied optimistic reads to critical methods:
- containsMessage() - Used for duplicate message detection
- nextDeliveryTime() - Called frequently for message scheduling
Maintained existing write operation synchronization
- Write operations continue to use synchronized for simplicity and safety
- Mixed approach optimal for typical read-heavy delayed delivery workloads
Removed unused data structure
- Eliminated unused immutableBucketsMap field to reduce memory overhead
- All bucket operations use the existing immutableBuckets RangeMap

Performance Improvements

Benchmark results show excellent performance across various concurrency scenarios:

Single-threaded reads: ~305 million ops/s
High concurrency (16 threads): ~2.6 billion ops/s
Mixed read/write ratios: Consistent performance from 10:90 to 90:10
Optimistic read success rate: Very high under typical read-heavy workloads

Thread Safety Strategy

Read operations: Use StampedLock optimistic reads for maximum performance
Write operations: Continue using synchronized for safety and simplicity
Data structures: Leverage existing thread-safe collections (ConcurrentHashMap, etc.)

Verifying this change

Added comprehensive thread safety test: BucketDelayedDeliveryTrackerThreadSafetyTest
Created performance benchmark: BucketDelayedDeliveryTrackerSimpleBenchmark
All existing tests pass
No functional changes - maintains full backward compatibility

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes:

Dependencies (add or upgrade a dependency)
The public API
The schema
The default behavior
The cluster topology
The ARM (kafka compatibility, producer/consumer compatibility)

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: Link

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

github-actions · 2025-07-21T21:22:05Z

@Apurva007 Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

codecov-commenter · 2025-07-21T22:57:49Z

Codecov Report

Attention: Patch coverage is 78.37838% with 8 lines in your changes missing coverage. Please review.

Project coverage is 74.26%. Comparing base (bbc6224) to head (975ff1f).
Report is 1215 commits behind head on master.

Files with missing lines	Patch %	Lines
...r/delayed/bucket/BucketDelayedDeliveryTracker.java	78.37%	6 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #24542      +/-   ##
============================================
+ Coverage     73.57%   74.26%   +0.69%     
- Complexity    32624    32898     +274     
============================================
  Files          1877     1869       -8     
  Lines        139502   146113    +6611     
  Branches      15299    16763    +1464     
============================================
+ Hits         102638   108518    +5880     
- Misses        28908    28964      +56     
- Partials       7956     8631     +675

Flag	Coverage Δ
inttests	`26.74% <0.00%> (+2.15%)`	⬆️
systests	`23.31% <0.00%> (-1.01%)`	⬇️
unittests	`73.77% <78.37%> (+0.92%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...r/delayed/bucket/BucketDelayedDeliveryTracker.java	`81.83% <78.37%> (-1.87%)`	⬇️

... and 1101 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Technoboy-

👍

lhotari

LGTM. Good work @Apurva007. Thanks for the contribution!

coderzc

LGTM

… with StampedLock optimistic reads (#24542) (cherry picked from commit 86eb3b8)

… with StampedLock optimistic reads (apache#24542) (cherry picked from commit 86eb3b8) (cherry picked from commit 455a052)

… with StampedLock optimistic reads (#24542) (cherry picked from commit 86eb3b8)

… with StampedLock optimistic reads (apache#24542) (cherry picked from commit 86eb3b8)

lhotari · 2025-07-29T15:52:39Z

Reopened #23190

…yTracker with StampedLock optimistic reads (#24542)" This reverts commit 455a052.

…yTracker with StampedLock optimistic reads (#24542)" This reverts commit b08b26d.

…yTracker with StampedLock optimistic reads (#24542)" This reverts commit dfb3ca5.

…yTracker with StampedLock optimistic reads (apache#24542)" This reverts commit dfb3ca5. (cherry picked from commit 00f2433)

…yTracker with StampedLock optimistic reads (apache#24542)" This reverts commit dfb3ca5.

… with StampedLock optimistic reads (apache#24542) (cherry picked from commit 86eb3b8) (cherry picked from commit dfb3ca5)

…yTracker with StampedLock optimistic reads (apache#24542)" This reverts commit dfb3ca5. (cherry picked from commit 00f2433)

… with StampedLock optimistic reads (#24542)

…yTracker with StampedLock optimistic reads (apache#24542)" This reverts commit 455a052.

… with StampedLock optimistic reads (apache#24542)

Apurva Telang added 2 commits July 18, 2025 15:44

Fix Thread Safety Issues in BucketDelayedDeliveryTracker implementation

b427297

Make the snapshotkey a local variable

975ff1f

github-actions bot added the doc-label-missing label Jul 21, 2025

hangc0276 assigned Apurva007 Jul 21, 2025

hangc0276 added ready-to-test doc-not-needed Your PR changes do not impact docs labels Jul 21, 2025

github-actions bot removed the doc-label-missing label Jul 21, 2025

codelipenghui requested review from codelipenghui, coderzc and lhotari July 21, 2025 22:22

codelipenghui added this to the 4.1.0 milestone Jul 21, 2025

codelipenghui added the release/4.0.6 label Jul 21, 2025

Technoboy- approved these changes Jul 22, 2025

View reviewed changes

lhotari approved these changes Jul 22, 2025

View reviewed changes

coderzc approved these changes Jul 22, 2025

View reviewed changes

codelipenghui merged commit 86eb3b8 into apache:master Jul 22, 2025
72 of 81 checks passed

lhotari pushed a commit that referenced this pull request Jul 22, 2025

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker…

455a052

… with StampedLock optimistic reads (#24542) (cherry picked from commit 86eb3b8)

lhotari added the cherry-picked/branch-4.0 label Jul 22, 2025

lhotari added release/3.0.13 release/3.3.8 labels Jul 25, 2025

lhotari pushed a commit that referenced this pull request Jul 25, 2025

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker…

b08b26d

… with StampedLock optimistic reads (#24542) (cherry picked from commit 86eb3b8)

lhotari pushed a commit that referenced this pull request Jul 25, 2025

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker…

dfb3ca5

… with StampedLock optimistic reads (#24542) (cherry picked from commit 86eb3b8)

lhotari added cherry-picked/branch-3.0 cherry-picked/branch-3.3 labels Jul 25, 2025

nodece pushed a commit to ascentstream/pulsar that referenced this pull request Jul 28, 2025

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker…

c2f1533

… with StampedLock optimistic reads (apache#24542) (cherry picked from commit 86eb3b8)

nodece pushed a commit to ascentstream/pulsar that referenced this pull request Jul 28, 2025

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker…

72ca596

… with StampedLock optimistic reads (apache#24542) (cherry picked from commit 86eb3b8)

lhotari mentioned this pull request Jul 29, 2025

[Bug] BucketDelayedDeliveryTracker.containsMessage is not thread-safe, but it's called from another thread #23190

Open

3 tasks

lhotari added release/4.0.7 release/3.0.14 release/3.3.9 and removed cherry-picked/branch-3.0 cherry-picked/branch-3.3 cherry-picked/branch-4.0 release/3.0.13 release/4.0.6 release/3.3.8 labels Jul 29, 2025

lhotari added a commit that referenced this pull request Jul 29, 2025

Revert "[fix][broker]Fix thread safety issues in BucketDelayedDeliver…

0508c51

…yTracker with StampedLock optimistic reads (#24542)" This reverts commit 455a052.

lhotari added a commit that referenced this pull request Jul 29, 2025

Revert "[fix][broker]Fix thread safety issues in BucketDelayedDeliver…

0fa78d9

…yTracker with StampedLock optimistic reads (#24542)" This reverts commit b08b26d.

lhotari added a commit that referenced this pull request Jul 29, 2025

Revert "[fix][broker]Fix thread safety issues in BucketDelayedDeliver…

00f2433

…yTracker with StampedLock optimistic reads (#24542)" This reverts commit dfb3ca5.

nodece pushed a commit to ascentstream/pulsar that referenced this pull request Jul 31, 2025

Revert "[fix][broker]Fix thread safety issues in BucketDelayedDeliver…

dbbc4ec

…yTracker with StampedLock optimistic reads (apache#24542)" This reverts commit dfb3ca5.

Technoboy- added the cherry-picked/branch-4.0 label Jul 31, 2025

Technoboy- pushed a commit that referenced this pull request Jul 31, 2025

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker…

9eb8986

… with StampedLock optimistic reads (#24542)

Denovo1998 mentioned this pull request Aug 4, 2025

[Enhancement] Future optimization and defect fixing of the delayed message module #24600

Open

2 tasks

nborisov pushed a commit to nborisov/pulsar that referenced this pull request Sep 12, 2025

Revert "[fix][broker]Fix thread safety issues in BucketDelayedDeliver…

e9e5ee8

…yTracker with StampedLock optimistic reads (apache#24542)" This reverts commit 455a052.

KannarFr pushed a commit to CleverCloud/pulsar that referenced this pull request Sep 22, 2025

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker…

be328c0

… with StampedLock optimistic reads (apache#24542)

lhotari added release/3.0.15 release/3.3.10 and removed release/3.0.14 release/3.3.9 labels Sep 27, 2025

walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker…

53e9b3d

… with StampedLock optimistic reads (apache#24542)

lhotari added release/3.0.16 and removed release/3.0.15 labels Nov 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads #24542

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads #24542

Apurva007 commented Jul 21, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 21, 2025

Uh oh!

codecov-commenter commented Jul 21, 2025 •

edited

Loading

Uh oh!

Technoboy- left a comment

Uh oh!

lhotari left a comment

Uh oh!

coderzc left a comment

Uh oh!

Uh oh!

lhotari commented Jul 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads #24542

[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads #24542

Conversation

Apurva007 commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Performance Improvements

Thread Safety Strategy

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

Uh oh!

github-actions bot commented Jul 21, 2025

Uh oh!

codecov-commenter commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Technoboy- left a comment

Choose a reason for hiding this comment

Uh oh!

lhotari left a comment

Choose a reason for hiding this comment

Uh oh!

coderzc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lhotari commented Jul 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Apurva007 commented Jul 21, 2025 •

edited

Loading

codecov-commenter commented Jul 21, 2025 •

edited

Loading