Skip to content

Conversation

@lhotari
Copy link
Member

@lhotari lhotari commented Jun 11, 2024

Motivation

The Pulsar Managed Ledger module contains an interface org.apache.bookkeeper.mledger.Position.
The general recommendation of using interfaces is to depend on the interface and not on implementation details.
However, this principle isn't followed in the code and there are a lot of casting to PositionImpl in the code and tight coupling to the implementation.

Another problem with the current code base is that batch index acknowledgement state ("ack set") is also coupled to the Position implementation. This PR provides a solution where "ack set" handling is decoupled from the Position interface without changing the architecture. It will continue to be possible to create a Position instance that carries and holds the "ack set" state for those execution paths that depend on this.

The motivation of this PR is to cleanup code before Pulsar 4.0 which is scheduled for October. The current master branch will most likely become 4.0. Since 4.0 will be the next LTS release, we would like to complete larger changes in the code base before the release.

The Position interface isn't exposed in Pulsar end user applications. Therefore this change can be considered as internal cleanup and refactoring which doesn't necessary require a PIP decision before proceeding. I'll bring this PR up to discussion on the mailing list regardless so this change doesn't come as a surprise to other contributors.

Third party Pulsar broker plugins depending on managed ledger interfaces will need to be adapted. However, this is a simple process. It's a simple replacement of PositionImpl with Position and using PositionFactory for the creation of Position instances.

After this PR, I'm planning to continue cleaning Managed Ledger the ManagedLedger and ManagedCursor interfaces. This PR is a pre-requisite for that work. Decoupling Pulsar core from Managed Ledger implementations will open up ways to add new Managed Ledger implementations in the future and make it pluggable.

Modifications

  • replace all usages of PositionImpl with Position and remove PositionImpl completely to remove any confusion in the future.
  • add PositionFactory for creating new instances so that there's no need for coupling to PositionImpl
  • decouple the "ack set" batch index acknowledgement state from Position with a solution (AckSetState) that doesn't change the current architecture.
    • it will continue to be possible to create a Position instance that carries and holds the "ack set" state for those execution paths that depend on this.
  • make the default implementation of Position immutable so that thread safety issues are avoided with the instance state
    • there are separate implementations to support object recycling and carrying "ack set" state

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: lhotari#193

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jun 11, 2024
@lhotari lhotari force-pushed the lh-positionimpl-to-position branch from a6baa92 to 5586660 Compare June 11, 2024 16:03
@lhotari lhotari self-assigned this Jun 11, 2024
@lhotari lhotari added type/cleanup Code or doc cleanups e.g. remove the outdated documentation or remove the code no longer in use area/ML labels Jun 11, 2024
@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 86.93467% with 78 lines in your changes missing coverage. Please review.

Project coverage is 73.27%. Comparing base (bbc6224) to head (5586660).
Report is 372 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #22891      +/-   ##
============================================
- Coverage     73.57%   73.27%   -0.31%     
- Complexity    32624    33028     +404     
============================================
  Files          1877     1896      +19     
  Lines        139502   142008    +2506     
  Branches      15299    15565     +266     
============================================
+ Hits         102638   104050    +1412     
- Misses        28908    29942    +1034     
- Partials       7956     8016      +60     
Flag Coverage Δ
inttests 27.49% <41.77%> (+2.91%) ⬆️
systests 24.76% <30.03%> (+0.43%) ⬆️
unittests 72.29% <86.93%> (-0.56%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...a/org/apache/bookkeeper/mledger/ManagedCursor.java 33.33% <ø> (-1.97%) ⬇️
...org/apache/bookkeeper/mledger/PositionFactory.java 100.00% <100.00%> (ø)
...rg/apache/bookkeeper/mledger/impl/AckSetState.java 100.00% <100.00%> (ø)
.../org/apache/bookkeeper/mledger/impl/EntryImpl.java 78.31% <100.00%> (-0.17%) ⬇️
...bookkeeper/mledger/impl/ImmutablePositionImpl.java 100.00% <100.00%> (ø)
...ookkeeper/mledger/impl/ManagedCursorContainer.java 96.10% <100.00%> (+0.64%) ⬆️
...kkeeper/mledger/impl/ManagedLedgerFactoryImpl.java 82.60% <100.00%> (+1.22%) ⬆️
.../bookkeeper/mledger/impl/NonDurableCursorImpl.java 81.25% <100.00%> (ø)
...org/apache/bookkeeper/mledger/impl/OpAddEntry.java 73.84% <100.00%> (+1.02%) ⬆️
...g/apache/bookkeeper/mledger/impl/OpFindNewest.java 89.85% <100.00%> (+0.30%) ⬆️
... and 64 more

... and 365 files with indirect coverage changes

@dave2wave
Copy link
Member

This change is so massive that I think docs are in fact required to explain all of the changes in a TL/DR manner.

Copy link
Contributor

@merlimat merlimat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nicely done!

@lhotari
Copy link
Member Author

lhotari commented Jun 13, 2024

This change is so massive that I think docs are in fact required to explain all of the changes in a TL/DR manner.

@dave2wave The reason why this change is massive is mainly due to the PositionImpl -> Position/PositionFactory changes. The PR description already contains the explanation in a TL/DR manner in "Modifications".
Anything missing?

Copy link
Contributor

@hangc0276 hangc0276 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job!

@lhotari lhotari merged commit 411f697 into apache:master Jun 13, 2024
@lhotari lhotari added this to the 4.0.0 milestone Oct 14, 2024
hanmz pushed a commit to hanmz/pulsar that referenced this pull request Feb 12, 2025
…terface (apache#22891)

Co-authored-by: Matteo Merli <mmerli@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ML doc-not-needed Your PR changes do not impact docs ready-to-test type/cleanup Code or doc cleanups e.g. remove the outdated documentation or remove the code no longer in use

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants