Allow to configure the managed ledger cache eviction frequency #4066
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4f12504 to
7facb33
Compare
rdhabalia
reviewed
Apr 23, 2019
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
Outdated
Show resolved
Hide resolved
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
Outdated
Show resolved
Hide resolved
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryImpl.java
Outdated
Show resolved
Hide resolved
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
Outdated
Show resolved
Hide resolved
22eaac4 to
60cda05
Compare
rdhabalia
approved these changes
Apr 24, 2019
Contributor
rdhabalia
left a comment
There was a problem hiding this comment.
minor comment regarding test.
| // activate caught up cursors | ||
| cursors.forEach(cursor -> { | ||
| if (cursor.getNumberOfEntries() < maxActiveCursorBacklogEntries) { | ||
| if (cursor.getNumberOfEntries() < backloggedCursorThresholdEntries) { |
pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
Show resolved
Hide resolved
Contributor
Author
|
@rdhabalia Updated with an additional time threshold and check for slowest active consumer. Please take another look. |
rdhabalia
reviewed
Apr 24, 2019
Contributor
rdhabalia
left a comment
There was a problem hiding this comment.
LGTM.. few minor comments..
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/RangeCache.java
Outdated
Show resolved
Hide resolved
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
Outdated
Show resolved
Hide resolved
| } | ||
| // this task: helps to activate inactive-backlog-cursors which have caught up and | ||
| // connected, also deactivate active-backlog-cursors which has backlog | ||
| ((PersistentTopic) topic).getManagedLedger().checkBackloggedCursors(); |
Contributor
There was a problem hiding this comment.
if we have removed from here then who triggers checkBackloggedCursors(..)?
Contributor
|
@merlimat we may also want to add config to standalone-config file and site-documentation file. |
Contributor
Author
Which one are you referring? I added into broker.conf |
Contributor
may be we can add configs to : |
Contributor
Author
|
Good point, Added. |
ivankelly
approved these changes
Apr 26, 2019
Contributor
ivankelly
left a comment
There was a problem hiding this comment.
Is there any testing for the time based eviction? I don't see it
Contributor
Author
|
@ivankelly Good point. Added explicit test for time-based eviction. |
Contributor
Author
|
run java8 tests |
This was referenced Jan 13, 2020
Closed
sijie
pushed a commit
that referenced
this pull request
Jan 18, 2020
### Motivation Some parameters are added in the `broker.conf` and `standalone.conf` files. However, those parameters are not updated in the docs. See the following PRs for details: #4150, #4066, #4197, #3819, #4261, #4273, #4320. ### Modifications Add those parameter info, and sync docs with the code. Does not update the description quite much, there are two reasons for this: 1. Keep doc content consistent with code. We need to update the description for those parameters in the code first, and then sync them in docs. 2. Will adopt a generator to generate those content automatically in the near future.
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Aug 24, 2020
### Motivation Some parameters are added in the `broker.conf` and `standalone.conf` files. However, those parameters are not updated in the docs. See the following PRs for details: apache#4150, apache#4066, apache#4197, apache#3819, apache#4261, apache#4273, apache#4320. ### Modifications Add those parameter info, and sync docs with the code. Does not update the description quite much, there are two reasons for this: 1. Keep doc content consistent with code. We need to update the description for those parameters in the code first, and then sync them in docs. 2. Will adopt a generator to generate those content automatically in the near future.
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Currently there's a rate limiter that avoid evicting from cache each time the read position is updated but rather only does that every 1 sec at max. The primary reason is to avoid mutex contention across different cursors trying to trigger the eviction in parallel.
The frequency should be higher though, to make sure we promptly discard all entries once all active cursors have moved past a particular point.
Modifications
broker.conf