Skip to content

[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue #24338

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dao-jun merged 4 commits into from
May 23, 2025
Merged

[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue #24338

dao-jun merged 4 commits into from
May 23, 2025

Conversation

@dao-jun
Copy link
Member

@dao-jun dao-jun commented May 22, 2025

Motivation

Fix ManagedCursorImpl.individualDeletedMessages concurrent issue.

2025-05-22T13:16:55,588 - INFO  - [PulsarTestContext-executor-OrderedExecutor-0-0:PulsarMockBookKeeper] - Creating ledger 6
2025-05-22T13:16:55,592 - WARN  - [PulsarTestContext-executor-OrderedExecutor-0-0:ManagedCursorImpl] - [my-property/my-ns/persistent/my-topic1]-my-subscriber-name Failed to serialize individualDeletedMessages
java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2
	at org.roaringbitmap.ArrayContainer.copyBitmapTo(ArrayContainer.java:1136) ~[RoaringBitmap-1.2.0.jar:?]
	at org.roaringbitmap.BitSetUtil.toLongArray(BitSetUtil.java:94) ~[RoaringBitmap-1.2.0.jar:?]
	at org.roaringbitmap.RoaringBitSet.toLongArray(RoaringBitSet.java:216) ~[RoaringBitmap-1.2.0.jar:?]
	at org.apache.pulsar.common.util.collections.OpenLongPairRangeSet.lambda$toRanges$5(OpenLongPairRangeSet.java:266) ~[classes/:?]
	at java.base/java.util.concurrent.ConcurrentSkipListMap.forEach(ConcurrentSkipListMap.java:3030) ~[?:?]
	at org.apache.pulsar.common.util.collections.OpenLongPairRangeSet.toRanges(OpenLongPairRangeSet.java:262) ~[classes/:?]
	at org.apache.bookkeeper.mledger.impl.RangeSetWrapper.toRanges(RangeSetWrapper.java:151) ~[classes/:?]
	at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.persistPositionToLedger(ManagedCursorImpl.java:3263) ~[classes/:?]
	at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.lambda$createNewMetadataLedger$37(ManagedCursorImpl.java:3067) ~[classes/:?]
	at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire$$$capture(CompletableFuture.java:718) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
	at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.lambda$doCreateNewMetadataLedger$39(ManagedCursorImpl.java:3115) ~[classes/:?]

Modifications

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@dao-jun dao-jun requested a review from lhotari May 22, 2025 05:34
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label May 22, 2025
@dao-jun dao-jun changed the title [fix][ML] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue [fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue May 22, 2025
@dao-jun dao-jun self-assigned this May 22, 2025
@dao-jun dao-jun added type/bug The PR fixed a bug or issue reported a bug area/ML ready-to-test labels May 22, 2025
@dao-jun dao-jun closed this May 22, 2025
@dao-jun dao-jun reopened this May 22, 2025
@codecov-commenter
Copy link

codecov-commenter commented May 22, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.24%. Comparing base (bbc6224) to head (ec7277d).
Report is 1116 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #24338      +/-   ##
============================================
+ Coverage     73.57%   74.24%   +0.67%     
+ Complexity    32624    32257     -367     
============================================
  Files          1877     1866      -11     
  Lines        139502   145126    +5624     
  Branches      15299    16593    +1294     
============================================
+ Hits         102638   107753    +5115     
+ Misses        28908    28854      -54     
- Partials       7956     8519     +563
Flag Coverage Δ
inttests 26.79% <50.00%> (+2.21%) ⬆️
systests 23.23% <50.00%> (-1.09%) ⬇️
unittests 73.72% <100.00%> (+0.87%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...che/bookkeeper/mledger/impl/ManagedCursorImpl.java 79.12% <100.00%> (-0.17%) ⬇️

... and 1086 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lhotari
Copy link
Member

lhotari commented May 22, 2025

@dao-jun It looks like there are multiple other concurrency issues introduced by #9292 changes. Would you like to cover all locations where individualDeletedMessages is accessed. Since the class isn't thread safe, either the read lock or write lock should be used for all accesses.

@lhotari
Copy link
Member

lhotari commented May 22, 2025

@dao-jun it seems that the only other location is in recoverIndividualDeletedMessages method.

lhotari
lhotari reviewed May 22, 2025
View reviewed changes
Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please also handle the concurrency issue in recoverIndividualDeletedMessages method. A write lock would be needed there.

@dao-jun
Copy link
Member Author

dao-jun commented May 22, 2025

please also handle the concurrency issue in recoverIndividualDeletedMessages method. A write lock would be needed there.

fixed

@dao-jun
Copy link
Member Author

dao-jun commented May 22, 2025

related to #22966

lhotari
lhotari approved these changes May 22, 2025
View reviewed changes
Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dao-jun dao-jun merged commit 376ae57 into apache:master May 23, 2025
53 checks passed
@dao-jun dao-jun deleted the fix/individual_concurrent_issue branch May 23, 2025 01:38
@lhotari lhotari added this to the 4.1.0 milestone May 23, 2025
lhotari pushed a commit that referenced this pull request Jun 2, 2025
@dao-jun @lhotari
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
fac0c16 
…issue (#24338)

(cherry picked from commit 376ae57)
lhotari pushed a commit that referenced this pull request Jun 2, 2025
@dao-jun @lhotari
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
c909926 
…issue (#24338)

(cherry picked from commit 376ae57)
lhotari pushed a commit that referenced this pull request Jun 2, 2025
@dao-jun @lhotari
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
448d5d0 
…issue (#24338)

(cherry picked from commit 376ae57)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 4, 2025
@dao-jun @ganesh-ctds
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
b47bce9 
…issue (apache#24338)

(cherry picked from commit 376ae57)
(cherry picked from commit c909926)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 4, 2025
@dao-jun @manas-ctds
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
3456617 
…issue (apache#24338)

(cherry picked from commit 376ae57)
(cherry picked from commit fac0c16)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 4, 2025
@dao-jun @ganesh-ctds
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
f857482 
…issue (apache#24338)

(cherry picked from commit 376ae57)
(cherry picked from commit c909926)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 4, 2025
@dao-jun @ganesh-ctds
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
6d9dbca 
…issue (apache#24338)

(cherry picked from commit 376ae57)
(cherry picked from commit c909926)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
@dao-jun @srinath-ctds
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
9d89526 
…issue (apache#24338)

(cherry picked from commit 376ae57)
(cherry picked from commit fac0c16)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
@dao-jun @ganesh-ctds
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
bc8b7e3 
…issue (apache#24338)

(cherry picked from commit 376ae57)
(cherry picked from commit c909926)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 10, 2025
@dao-jun @srinath-ctds
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
a4219c2 
…issue (apache#24338)

(cherry picked from commit 376ae57)
(cherry picked from commit c909926)
nodece pushed a commit to nodece/pulsar that referenced this pull request Jun 18, 2025
@dao-jun @nodece
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
0b35a9f 
…issue (apache#24338)

(cherry picked from commit 376ae57)
KannarFr pushed a commit to CleverCloud/pulsar that referenced this pull request Sep 22, 2025
@dao-jun @KannarFr
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
0b01818 
…issue (apache#24338)
walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025
@dao-jun @walkinggo
[fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent …
35f4cda 
…issue (apache#24338)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lhotari lhotari lhotari approved these changes

Assignees

@dao-jun dao-jun

Labels

area/ML cherry-picked/branch-3.0 cherry-picked/branch-3.3 cherry-picked/branch-4.0 doc-not-needed Your PR changes do not impact docs ready-to-test release/3.0.13 release/3.3.8 release/4.0.6 type/bug The PR fixed a bug or issue reported a bug

Projects

None yet

Milestone

4.1.0

Development

Successfully merging this pull request may close these issues.

3 participants

@dao-jun @codecov-commenter @lhotari