CompositeByteBuf#newComponent() failed to resolve source index#9398
CompositeByteBuf#newComponent() failed to resolve source index#9398
Conversation
Merge the latest version
|
Can one of the admins verify this patch? |
|
@jingene please provide a unit test as well |
|
@normanmaurer Oh sorry, I added a unit test just now. 184d750 |
|
@netty-bot test this please |
|
|
||
| @Test | ||
| public void testAddComponentWithLeakAwareByteBuf() { | ||
| ResourceLeakDetector.setLevel(ResourceLeakDetector.Level.PARANOID); |
There was a problem hiding this comment.
@jingene we must reset this to the previous level once this test completes.
There was a problem hiding this comment.
Didn't you also adjust the sampling interval before ?
There was a problem hiding this comment.
Also I wonder if we couldn't construct the buffer directly and so not need all of the above. Please check our *LeakAwareByteBufTest classes.
There was a problem hiding this comment.
@normanmaurer Thank you for your kind guide. AdvancedLeakAwareByteBufTest#wrap guarantees return AdvancedLeakAwareByteBuf instance so I removed ResourceLeakDetector manipulation codes (c479b10)
| @@ -0,0 +1,24 @@ | |||
| package io.netty.buffer; | |||
There was a problem hiding this comment.
Please add a license header.
| public void tearDown() throws Exception { | ||
| System.clearProperty("io.netty.leakDetection.level"); | ||
| System.clearProperty("io.netty.leakDetection.samplingInterval"); | ||
| } |
There was a problem hiding this comment.
better do all of this in the test itself. That said you should grab the previous value and use it later to reset it. This should happen in a finally block.
| public class CompositeByteBufTest extends AdvancedLeakAwareByteBufTest { | ||
|
|
||
| @Test | ||
| public void testAddComponentWithLeakAwareByteBuf() { |
There was a problem hiding this comment.
just merge this test into AdvancedLeakAwareByteBufTest. There is no need to introduce a new sub-class
There was a problem hiding this comment.
@normanmaurer Make sense. TestCase moved(5484583).
|
|
||
| @Test | ||
| public void testAddComponentWithLeakAwareByteBuf() { | ||
| ByteBuf buffer = newBuffer("hello world".length()); |
There was a problem hiding this comment.
are you sure this is correct ? It seems like this will only allocate an "empty" buffer with the length of "hello world".
| public void testAddComponentWithLeakAwareByteBuf() { | ||
| NoopResourceLeakTracker<ByteBuf> tracker = new NoopResourceLeakTracker<ByteBuf>(); | ||
|
|
||
| ByteBuf buffer = wrappedBuffer("hello world".getBytes()).slice(6, 5); |
There was a problem hiding this comment.
nit: getBytes(CharsetUtil.US_ASCII);
| composite.addComponent(true, leakAwareBuf); | ||
| byte[] result = new byte[5]; | ||
| composite.component(0).readBytes(result); | ||
| assertArrayEquals("world".getBytes(), result); |
There was a problem hiding this comment.
nit: getBytes(CharsetUtil.US_ASCII);
| byte[] result = new byte[5]; | ||
| composite.component(0).readBytes(result); | ||
| assertArrayEquals("world".getBytes(), result); | ||
| } |
|
@netty-bot test this please |
|
I fixed to do unwrap() if |
|
@netty-bot test it please |
|
@netty-bot test this please |
|
@jingene there seems to be a leak... please check with |
|
@jingene just to understand, this is not a fix for a leak or functional error, is that correct? The unwrapping and index adjustment done in If we do it then it seems we should do it generally for And it would also cover |
|
@njhill I wrote a testcase to prove https://github.com/netty/netty/pull/9398/files#diff-88454e0c917aafd9378476d4cf035969R38 Please tell me if there is wrong with the verification. Thanks. |
|
Thanks @jingene, I only read the PR description and should have looked closer at your unit test. So the problem is really that the behaviour of I think it was caused by my mis-assumption about the behaviour of As mentioned, unwrapping sliced buffers before they're added was really just an optimization and shouldn't be required for correctness. It just happens that it circumvented the bug for the most common cases. I will open a PR with a general fix and your unit test (which looks great) and make sure you are included as a co-author. We could separately consider extending the pre-unwrapping optimization to include some of these other buffer variants like wrapped slices, and possibly duplicates. |
|
@njhill Thank you for your explanation :) I'll close this PR and I hope it will be discussed further in the PR that you will open. |
Motivation The way that Components are added/stored in CompositeByteBuf was changed in netty#8437 to minimize slicing and runtime access indirection, with the obvious intention to preserve the effective behaviour of public methods. @jingene discovered a discrepancy, reported in netty#9398, where the component(int) and componentAtOffset(int) methods can return an incorrect ByteBuf, i.e. not equivalent to a slice of the added buffer at the time it was added (the previous behaviour). This can happen in particular if the added ByteBuf is a wrapped slice, e.g. a leak aware or unreleasable buffer. Upon further scrutiny another subtle deviation was also noticed: When the added `ByteBuf is a slice whose readable region does not cover the whole buffer internalComponent(int) returns the original slice (with capacity != readableBytes) rather than a sliced slice. Modifications Unfortunately to fix this robustly required slightly more invasive change than first expected. - A final ByteBuf srcBuf field has been added to the Component class, which always holds the originally-added buffer. This simplifies some of the existing fragile logic where we need access to this (e.g. when releasing) - Component has been made abstract with two final impls. Which of these is used depends on whether the srcBuf is already a slice of the component's range (or equivalent to one) - Correctly handle the various places where access to the pre-unwrapped buffer is important, including the problem component(int) methods - Extend the pre-unwrapping optimization to cover WrappedByteBufs, SwappedByteBufs, and duplicates in addition to slices - Unit test for the original bug provided by @jingene Result - Correct behaviour of the (internal)component(AtOffset) methods in all cases - Reduced indirection when accessing components that were added from more kinds of ByteBufs (duplicates in particular) - Less fragile logic related to lazy-slicing components I don't expect there to be any noticeable performance impact of splitting Component into two subclasses since most of the methods are still final in the superclass and the ones that aren't will benefit from bimorphic inlining. But will aim to benchmark nonetheless. There could be slight increase in mem use due to two additional fields in _one_ of the two Component subtypes, but I think this is needed for correctness and still much better than slicing components unconditionally when adding them. Co-authored-by: jingene <jingene0206@gmail.com>
Motivation This is a "simpler" alternative to netty#9416 which fixes the same CompositeByteBuf bugs described there, originally reported by @jingene in netty#9398. Modifications - Add fields to Component class for the original buffer along with its adjustment, which may be different to the already-stored unwrapped buffer. Use it in appropriate places to ensure correctness and equivalent behaviour to that prior to the earlier optimizations - Add comments explaining purpose of each of the Component fields - Unwrap more kinds of buffers in newComponent method to extend scope of the existing indirection-reduction optimization - De-duplicate common buffer consolidation logic - Unit test for the original bug provided by @jingene Result - Correct behaviour / fixed bugs - Some code deduplication / simplification - Unwrapping optimization applied to more types of buffers The downside is increased mem footprint from the two new fields, and additional allocations in some specific cases, though those should be rare. Co-authored-by: jingene <jingene0206@gmail.com>
Motivation This is a "simpler" alternative to #9416 which fixes the same CompositeByteBuf bugs described there, originally reported by @jingene in #9398. Modifications - Add fields to Component class for the original buffer along with its adjustment, which may be different to the already-stored unwrapped buffer. Use it in appropriate places to ensure correctness and equivalent behaviour to that prior to the earlier optimizations - Add comments explaining purpose of each of the Component fields - Unwrap more kinds of buffers in newComponent method to extend scope of the existing indirection-reduction optimization - De-duplicate common buffer consolidation logic - Unit test for the original bug provided by @jingene Result - Correct behaviour / fixed bugs - Some code deduplication / simplification - Unwrapping optimization applied to more types of buffers The downside is increased mem footprint from the two new fields, and additional allocations in some specific cases, though those should be rare. Co-authored-by: jingene <jingene0206@gmail.com>
Motivation This is a "simpler" alternative to #9416 which fixes the same CompositeByteBuf bugs described there, originally reported by @jingene in #9398. Modifications - Add fields to Component class for the original buffer along with its adjustment, which may be different to the already-stored unwrapped buffer. Use it in appropriate places to ensure correctness and equivalent behaviour to that prior to the earlier optimizations - Add comments explaining purpose of each of the Component fields - Unwrap more kinds of buffers in newComponent method to extend scope of the existing indirection-reduction optimization - De-duplicate common buffer consolidation logic - Unit test for the original bug provided by @jingene Result - Correct behaviour / fixed bugs - Some code deduplication / simplification - Unwrapping optimization applied to more types of buffers The downside is increased mem footprint from the two new fields, and additional allocations in some specific cases, though those should be rare. Co-authored-by: jingene <jingene0206@gmail.com>
Motivation:
CompositeByteBuf#newComponent(ByteBuf buf, int offset)checks whetherbufis instance of (AbstractUnpooledSlicedByteBuf,PooledSlicedByteBuf) to adjustsrcIndex.-Dio.netty.leakDetection.level=SIMPLE, PARANOID, .., to find out memory leakage,(Simple|Advanced)LeakAwareByteBufwrapsByteBufs mentioned above based on samplingInterval. soByteBufchecking logic skips to unintentionally.Modification:
bufis instance ofWrappedByteBuf, unwrap it first.Result:
srcIndexis resolved correctly.