Add memory limiter for the add entry request to avoid OOM #3139
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
The test come with next commit |
Contributor
|
@zymap Have you tried configuring backpressure for the BK client and server? I think it covers your case. |
Member
Author
|
@dlg99 Thanks for your information. But looks like the maxAddsInProgressLimit only applied on the v3 request? |
Member
Author
|
I tried the backpressure with V2 protocol. It doesn't work. Because it is a server limitation, when there has a server slow to respond, the other server will respond successfully, but the client will hold the entry buffer until the slowest response is returned. Same issue: https://lists.apache.org/thread/r2yh6gxgnhzzz35o63db040wt2t6o108 |
eolivelli
requested changes
Jun 12, 2022
| private final BookieAddressResolver bookieAddressResolver; | ||
|
|
||
| private final long bookieErrorThresholdPerInterval; | ||
| private Optional<MemoryLimitController> memoryLimitController; |
| } | ||
| } | ||
|
|
||
| private static class WriteAndFlushCallbackImpl implements WriteAndFlushCallback { |
Contributor
There was a problem hiding this comment.
This should be probably recycled
| private Optional<WriteAndFlushCallback> setMemoryLimit(final long entryId, final long entrySize) throws InterruptedException { | ||
| if (getMemoryLimitController().isPresent()) { | ||
| MemoryLimitController mlc = getMemoryLimitController().get(); | ||
| mlc.reserveMemory(entrySize); |
Contributor
There was a problem hiding this comment.
We should handle this with a timeout otherwise we may block forever?
| "clientConnectBookieUnavailableLogThrottling"; | ||
|
|
||
| // client memory limit options | ||
| protected static final String CLIENT_MEMORY_LIMIT_ENABLED = "clientMemoryLimitEnabled"; |
Contributor
There was a problem hiding this comment.
It looks like this is only for writes.
We should reflect this in the name
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieClientImpl.java
Outdated
Show resolved
Hide resolved
Contributor
Contributor
|
@hangc0276 I don't think that the issue is resolved. @zymap what do you think ? |
--- *Motivation* When there has a bookie is slowly to respond the add entry request, and the AQ is smaller than WQ, the client will hole the entry buffer and that will cause the memory won't be released. More context: apache/pulsar#14861 *Modifications* Add the memory limit for the client to send add request.
fa285bf to
a55b91b
Compare
Member
Author
|
The test output looks like works as expected, I will try to add more tests and clean the code. logs |
hangc0276
reviewed
Oct 18, 2022
| } | ||
|
|
||
| public long getWriteMemoryLowWaterMark() { | ||
| return getInt(WRITE_MEMORY_LOW_WATER_MARK, 64 * 1024 * 1024); |
| } | ||
|
|
||
| public long getWriteMemoryHighWaterMark() { | ||
| return getInt(WRITE_MEMORY_HIGH_WATER_MARK, 256 * 1024 * 1024); |
| */ | ||
| public class WriteWaterMark { | ||
| private static final long DEFAULT_LOW_WATER_MARK = 1610612736L; // 1.5GB | ||
| private static final long DEFAULT_HIGH_WATER_MARK = 2147483648L; // 2GB |
Contributor
There was a problem hiding this comment.
Why the default lowWaterMark=64MB and HighWaterMark=256MB in ClientConfiguration, but default lowWaterMark=1.5GB and HighWaterMark=2GB in WriteWaterMark?
hangc0276
reviewed
Oct 18, 2022
| @Slf4j | ||
| public class WriteMemoryCounter { | ||
| private final WriteWaterMark writeWaterMark; | ||
| private AtomicLong sizeCounter = new AtomicLong(0); |
|
|
||
| public void decrementPendingWriteBytes(long size) { | ||
| long usage = sizeCounter.addAndGet(-size); | ||
| log.info("decrement the size to {}", usage); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Master issue: #3231
Motivation
When there has a bookie is slow to respond the add entry request,
and the AQ is smaller than WQ, the client will hole the entry buffer
and that will cause the memory won't be released.
More context:
apache/pulsar#14861
Modifications
Add the memory limit for the client to the send add request.