Skip to content

CompositeByteBuf#newComponent() failed to resolve source index#9398

Closed
jingene wants to merge 9 commits intonetty:4.1from
jingene:feature/unwrap_first_wrappedbytebuf_in_compositebytebuf
Closed

CompositeByteBuf#newComponent() failed to resolve source index#9398
jingene wants to merge 9 commits intonetty:4.1from
jingene:feature/unwrap_first_wrappedbytebuf_in_compositebytebuf

Conversation

@jingene
Copy link
Copy Markdown
Contributor

@jingene jingene commented Jul 24, 2019

Motivation:

  • CompositeByteBuf#newComponent(ByteBuf buf, int offset) checks whether buf is instance of ( AbstractUnpooledSlicedByteBuf, PooledSlicedByteBuf) to adjust srcIndex.
  • If we use -Dio.netty.leakDetection.level=SIMPLE, PARANOID, .., to find out memory leakage, (Simple|Advanced)LeakAwareByteBuf wraps ByteBufs mentioned above based on samplingInterval. so ByteBuf checking logic skips to unintentionally.

Modification:

  • Add more instance check logic: if buf is instance of WrappedByteBuf, unwrap it first.

Result:

  • srcIndex is resolved correctly.

@netty-bot
Copy link
Copy Markdown

Can one of the admins verify this patch?

@normanmaurer
Copy link
Copy Markdown
Member

@jingene please provide a unit test as well

@jingene
Copy link
Copy Markdown
Contributor Author

jingene commented Jul 24, 2019

@normanmaurer Oh sorry, I added a unit test just now. 184d750

@normanmaurer
Copy link
Copy Markdown
Member

@netty-bot test this please


@Test
public void testAddComponentWithLeakAwareByteBuf() {
ResourceLeakDetector.setLevel(ResourceLeakDetector.Level.PARANOID);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jingene we must reset this to the previous level once this test completes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. fixed(8da1276). Thanks.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't you also adjust the sampling interval before ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also I wonder if we couldn't construct the buffer directly and so not need all of the above. Please check our *LeakAwareByteBufTest classes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@normanmaurer Thank you for your kind guide. AdvancedLeakAwareByteBufTest#wrap guarantees return AdvancedLeakAwareByteBuf instance so I removed ResourceLeakDetector manipulation codes (c479b10)

@@ -0,0 +1,24 @@
package io.netty.buffer;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a license header.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LICENSE header added(8da1276).

public void tearDown() throws Exception {
System.clearProperty("io.netty.leakDetection.level");
System.clearProperty("io.netty.leakDetection.samplingInterval");
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better do all of this in the test itself. That said you should grab the previous value and use it later to reset it. This should happen in a finally block.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

public class CompositeByteBufTest extends AdvancedLeakAwareByteBufTest {

@Test
public void testAddComponentWithLeakAwareByteBuf() {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just merge this test into AdvancedLeakAwareByteBufTest. There is no need to introduce a new sub-class

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@normanmaurer Make sense. TestCase moved(5484583).


@Test
public void testAddComponentWithLeakAwareByteBuf() {
ByteBuf buffer = newBuffer("hello world".length());
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you sure this is correct ? It seems like this will only allocate an "empty" buffer with the length of "hello world".

public void testAddComponentWithLeakAwareByteBuf() {
NoopResourceLeakTracker<ByteBuf> tracker = new NoopResourceLeakTracker<ByteBuf>();

ByteBuf buffer = wrappedBuffer("hello world".getBytes()).slice(6, 5);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: getBytes(CharsetUtil.US_ASCII);

composite.addComponent(true, leakAwareBuf);
byte[] result = new byte[5];
composite.component(0).readBytes(result);
assertArrayEquals("world".getBytes(), result);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: getBytes(CharsetUtil.US_ASCII);

byte[] result = new byte[5];
composite.component(0).readBytes(result);
assertArrayEquals("world".getBytes(), result);
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

composite.release();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. thanks. added(dc5b077)

@normanmaurer
Copy link
Copy Markdown
Member

@netty-bot test this please

@jingene
Copy link
Copy Markdown
Contributor Author

jingene commented Jul 24, 2019

I fixed to do unwrap() if buf is an instance of SimpleLeakAwareByteBuf and unwrapped buf is not type of sliced buffer(AbstractUnpooledSlicedByteBuf or PooledSlicedByteBuf),
create Component instance using initial LeakAwareByteBuf.
(31c0d2f)

@normanmaurer
Copy link
Copy Markdown
Member

@netty-bot test it please

@njhill njhill self-requested a review July 24, 2019 16:31
@normanmaurer
Copy link
Copy Markdown
Member

@netty-bot test this please

@normanmaurer
Copy link
Copy Markdown
Member

@jingene there seems to be a leak... please check with ./mvnw -Pleak clean package and fix.

@njhill
Copy link
Copy Markdown
Member

njhill commented Jul 25, 2019

@jingene just to understand, this is not a fix for a leak or functional error, is that correct? The unwrapping and index adjustment done in newComponent is really only an optimization to reduce subsequent indirection. So I'm not sure this is really needed given that leak aware buffers are rare by design.

If we do it then it seems we should do it generally for WrappedBuffers, what was the reason for the latest commit to make it more specific? The important thing is that when buf is set to an unwrapped buffer, the slice field is set to the original buffer. This will ensure existing ref counting semantics are preserved.

And it would also cover UnreleasableBuffers which could be more prevalent/significant than the leak-aware ones in certain use cases.

@jingene
Copy link
Copy Markdown
Contributor Author

jingene commented Jul 29, 2019

@njhill
Actually, I don't have deep understanding of ByteBuf family concepts. sorry.
The import thing is the behavior of CompositeByteBuf#duplicate() changed after the netty v4.1.32.Final released(related PR; https://github.com/netty/netty/pull/8437/files).

I wrote a testcase to prove CompositeByteBuf#component() works as intended if added component is (*SlicedByteBuf wrapped with *LeakAwareByteBuf)

https://github.com/netty/netty/pull/9398/files#diff-88454e0c917aafd9378476d4cf035969R38
Using before the v4.1.31.Final: TC PASS
Using after the v4.1.32.Final : TC FAILED

Please tell me if there is wrong with the verification. Thanks.

@njhill
Copy link
Copy Markdown
Member

njhill commented Jul 29, 2019

Thanks @jingene, I only read the PR description and should have looked closer at your unit test. So the problem is really that the behaviour of CompositeByteBuf#component(int) changed, right? This does indeed look like a bug, so thank you for raising it.

I think it was caused by my mis-assumption about the behaviour of ByteBuf#duplicate() - specifically that duplicating a slice does not result in another slice and doesn't preserve the reader/writer indices.

As mentioned, unwrapping sliced buffers before they're added was really just an optimization and shouldn't be required for correctness. It just happens that it circumvented the bug for the most common cases. I will open a PR with a general fix and your unit test (which looks great) and make sure you are included as a co-author.

We could separately consider extending the pre-unwrapping optimization to include some of these other buffer variants like wrapped slices, and possibly duplicates.

@jingene
Copy link
Copy Markdown
Contributor Author

jingene commented Jul 30, 2019

@njhill Thank you for your explanation :) I'll close this PR and I hope it will be discussed further in the PR that you will open.

@njhill
Copy link
Copy Markdown
Member

njhill commented Jul 31, 2019

Thanks @jingene, I've now opened #9416, PTAL!

njhill added a commit to njhill/netty that referenced this pull request Aug 28, 2019
Motivation

The way that Components are added/stored in CompositeByteBuf was changed
in netty#8437 to minimize slicing and runtime access indirection, with the
obvious intention to preserve the effective behaviour of public methods.

@jingene discovered a discrepancy, reported in netty#9398, where the
component(int) and componentAtOffset(int) methods can return an
incorrect ByteBuf, i.e. not equivalent to a slice of the added buffer at
the time it was added (the previous behaviour). This can happen in
particular if the added ByteBuf is a wrapped slice, e.g. a leak aware or
unreleasable buffer.

Upon further scrutiny another subtle deviation was also noticed: When
the added `ByteBuf is a slice whose readable region does not cover the
whole buffer internalComponent(int) returns the original slice (with
capacity != readableBytes) rather than a sliced slice.

Modifications

Unfortunately to fix this robustly required slightly more invasive
change than first expected.
- A final ByteBuf srcBuf field has been added to the Component class,
which always holds the originally-added buffer. This simplifies some of
the existing fragile logic where we need access to this (e.g. when
releasing)
- Component has been made abstract with two final impls. Which of these
is used depends on whether the srcBuf is already a slice of the
component's range (or equivalent to one)
- Correctly handle the various places where access to the pre-unwrapped
buffer is important, including the problem component(int) methods
- Extend the pre-unwrapping optimization to cover WrappedByteBufs,
SwappedByteBufs, and duplicates in addition to slices
- Unit test for the original bug provided by @jingene

Result

- Correct behaviour of the (internal)component(AtOffset) methods in all
cases
- Reduced indirection when accessing components that were added from
more kinds of ByteBufs (duplicates in particular)
- Less fragile logic related to lazy-slicing components

I don't expect there to be any noticeable performance impact of
splitting Component into two subclasses since most of the methods are
still final in the superclass and the ones that aren't will benefit from
bimorphic inlining. But will aim to benchmark nonetheless.

There could be slight increase in mem use due to two additional fields
in _one_ of the two Component subtypes, but I think this is needed for
correctness and still much better than slicing components
unconditionally when adding them.


Co-authored-by: jingene <jingene0206@gmail.com>
njhill added a commit to njhill/netty that referenced this pull request Aug 30, 2019
Motivation

This is a "simpler" alternative to netty#9416 which fixes the same
CompositeByteBuf bugs described there, originally reported by @jingene
in netty#9398.

Modifications
- Add fields to Component class for the original buffer along with its
adjustment, which may be different to the already-stored unwrapped
buffer. Use it in appropriate places to ensure correctness and
equivalent behaviour to that prior to the earlier optimizations
- Add comments explaining purpose of each of the Component fields
- Unwrap more kinds of buffers in newComponent method to extend scope of
the existing indirection-reduction optimization
- De-duplicate common buffer consolidation logic
- Unit test for the original bug provided by @jingene

Result
- Correct behaviour / fixed bugs
- Some code deduplication / simplification
- Unwrapping optimization applied to more types of buffers

The downside is increased mem footprint from the two new fields, and
additional allocations in some specific cases, though those should be
rare.


Co-authored-by: jingene <jingene0206@gmail.com>
normanmaurer pushed a commit that referenced this pull request Sep 2, 2019
Motivation

This is a "simpler" alternative to #9416 which fixes the same
CompositeByteBuf bugs described there, originally reported by @jingene
in #9398.

Modifications
- Add fields to Component class for the original buffer along with its
adjustment, which may be different to the already-stored unwrapped
buffer. Use it in appropriate places to ensure correctness and
equivalent behaviour to that prior to the earlier optimizations
- Add comments explaining purpose of each of the Component fields
- Unwrap more kinds of buffers in newComponent method to extend scope of
the existing indirection-reduction optimization
- De-duplicate common buffer consolidation logic
- Unit test for the original bug provided by @jingene

Result
- Correct behaviour / fixed bugs
- Some code deduplication / simplification
- Unwrapping optimization applied to more types of buffers

The downside is increased mem footprint from the two new fields, and
additional allocations in some specific cases, though those should be
rare.


Co-authored-by: jingene <jingene0206@gmail.com>
normanmaurer pushed a commit that referenced this pull request Sep 2, 2019
Motivation

This is a "simpler" alternative to #9416 which fixes the same
CompositeByteBuf bugs described there, originally reported by @jingene
in #9398.

Modifications
- Add fields to Component class for the original buffer along with its
adjustment, which may be different to the already-stored unwrapped
buffer. Use it in appropriate places to ensure correctness and
equivalent behaviour to that prior to the earlier optimizations
- Add comments explaining purpose of each of the Component fields
- Unwrap more kinds of buffers in newComponent method to extend scope of
the existing indirection-reduction optimization
- De-duplicate common buffer consolidation logic
- Unit test for the original bug provided by @jingene

Result
- Correct behaviour / fixed bugs
- Some code deduplication / simplification
- Unwrapping optimization applied to more types of buffers

The downside is increased mem footprint from the two new fields, and
additional allocations in some specific cases, though those should be
rare.


Co-authored-by: jingene <jingene0206@gmail.com>
@normanmaurer normanmaurer added this to the 4.1.40.Final milestone Sep 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants