Originates from #16059 - it was covering some unrelated topics, that got resolved in #16058 (comment).
Opening an issue for reproducible part of the question.
I have simple test that writes some bytes to a direct buffer.
Idea is to use allocator.metric().usedDirectMemory() and try to short circuit the operation if used direct memory is about to exceed the limit.
Netty version: 4.2.9.Final
// Run it with -Xmx80m (direct will be similar, Runtime.getRuntime().maxMemory())
// PooledByteBufAllocator allocator = new PooledByteBufAllocator(true);
AdaptiveByteBufAllocator allocator = new AdaptiveByteBufAllocator(true);
ByteBuf buf = allocator.buffer();
assertThat(buf.isDirect()).isTrue();
// I run this with MaxDirectMemorySize not specified,
// see https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/jdk/internal/misc/VM.java#L138
long directMemoryLimit = Runtime.getRuntime().maxMemory();
byte[] data = new byte[5000];
Arrays.fill(data, (byte) 1);
while (true) {
if (allocator.metric().usedDirectMemory() + data.length > directMemoryLimit) {
throw new RuntimeException("OOM has been prevented, used memory reporting is precise.");
}
logger.info(
"used direct :{}, used heap: {}, writing: {}, limit: {}",
allocator.metric().usedDirectMemory(),
allocator.metric().usedHeapMemory(),
data.length,
directMemoryLimit
);
buf.writeBytes(data);
}
I'm getting OOM-s with both allocators
PooledByteBufAllocator
java.lang.OutOfMemoryError: Cannot reserve 37748736 bytes of direct buffer memory (allocated: 50331649, limit: 81133568)
at __randomizedtesting.SeedInfo.seed([4EA583D51E5D8AD3:9A07F15EAC7E0677]:0)
at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:108)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:367)
at io.netty.util.internal.CleanerJava9.allocate(CleanerJava9.java:86)
at io.netty.util.internal.PlatformDependent.allocateDirect(PlatformDependent.java:600)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:767)
at io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:754)
at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:230)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:146)
at io.netty.buffer.PoolArena.reallocate(PoolArena.java:328)
at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:126)
at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:305)
at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:280)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1080)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1088)
at io.netty.buffer.WrappedByteBuf.writeBytes(WrappedByteBuf.java:803)
at io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:618)
PooledByteBufAllocator logs
used direct :8388608, used heap: 0, writing: 5000, limit: 81133568
.. many same entries
used direct :12582912, used heap: 0, writing: 5000, limit: 81133568
...many similar entries with "used direct" increasing to 4MB from time to time.
...
// last reported before OOM
used direct :37748736, used heap: 0, writing: 5000, limit: 81133568
Note that allocated: 50331649 in the OOM error is 12MB higher then last reported in logs allocator.metric().usedDirectMemory()
AdaptiveByteBufAllocator
java.lang.OutOfMemoryError: Cannot reserve 37748736 bytes of direct buffer memory (allocated: 50200577, limit: 81133568)
at __randomizedtesting.SeedInfo.seed([4EA583D51E5D8AD3:9A07F15EAC7E0677]:0)
at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:108)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:367)
at io.netty.util.internal.CleanerJava9.allocate(CleanerJava9.java:86)
at io.netty.util.internal.PlatformDependent.allocateDirect(PlatformDependent.java:600)
at io.netty.buffer.UnpooledDirectByteBuf.allocateDirectBuffer(UnpooledDirectByteBuf.java:138)
at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:81)
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:48)
at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:698)
at io.netty.buffer.AdaptiveByteBufAllocator$DirectChunkAllocator.allocate(AdaptiveByteBufAllocator.java:115)
at io.netty.buffer.AdaptivePoolingAllocator.allocateFallback(AdaptivePoolingAllocator.java:309)
at io.netty.buffer.AdaptivePoolingAllocator.allocate(AdaptivePoolingAllocator.java:273)
at io.netty.buffer.AdaptivePoolingAllocator.reallocate(AdaptivePoolingAllocator.java:332)
at io.netty.buffer.AdaptivePoolingAllocator$AdaptiveByteBuf.capacity(AdaptivePoolingAllocator.java:1471)
at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:305)
at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:280)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1080)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1088)
at io.netty.buffer.WrappedByteBuf.writeBytes(WrappedByteBuf.java:803)
at io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:618)
Logs - same picture, 4MB increment, heap is also 0 (expected, just logged it for completeness), last reported before OOM is
37617664 which is also around 12MB lower then allocated: 50200577 from the error message
Originates from #16059 - it was covering some unrelated topics, that got resolved in #16058 (comment).
Opening an issue for reproducible part of the question.
I have simple test that writes some bytes to a direct buffer.
Idea is to use
allocator.metric().usedDirectMemory()and try to short circuit the operation if used direct memory is about to exceed the limit.Netty version: 4.2.9.Final
I'm getting OOM-s with both allocators
PooledByteBufAllocator
PooledByteBufAllocator logs
Note that
allocated: 50331649in the OOM error is 12MB higher then last reported in logsallocator.metric().usedDirectMemory()AdaptiveByteBufAllocator
Logs - same picture, 4MB increment, heap is also 0 (expected, just logged it for completeness), last reported before OOM is
37617664 which is also around 12MB lower then
allocated: 50200577from the error message