Skip to content

availableSharedCapacity will be slowly exhausted#9394

Merged
normanmaurer merged 1 commit intonetty:4.1from
YuanHuCoding:4.1
Jul 21, 2019
Merged

availableSharedCapacity will be slowly exhausted#9394
normanmaurer merged 1 commit intonetty:4.1from
YuanHuCoding:4.1

Conversation

@YuanHuCoding
Copy link
Copy Markdown
Contributor

Without this line of code(Line390):this.head.reclaimSpace(LINK_CAPACITY);
availableSharedCapacity will gradually be exhausted, eventually leading to reserveSpace (int space) returning false and Recycler will completely invalid.

Without this line of code(Line390):this.head.reclaimSpace(LINK_CAPACITY);
availableSharedCapacity will gradually be exhausted, eventually leading to reserveSpace (int space) returning false and Recycler will completely invalid.
@netty-bot
Copy link
Copy Markdown

Can one of the admins verify this patch?

@normanmaurer
Copy link
Copy Markdown
Member

@YuanHuCoding would it be possible to add a unit test ?

@YuanHuCoding
Copy link
Copy Markdown
Contributor Author

My test scenario is as follows:

1)Construct Recycler without discarding any objects:

new Recycler < T > (4096, 1, 1, Integer. MAX_VALUE)

2)Submit the recovery operation through a single thread pool:

RECYCLE_THREAD_POOL.submit(() -> {
    xxx.recycle();
}); 

3)Statistics of all the creation of new objects, objects recycled, objects discarded. You'll find out availableSharedCapacity will gradually be exhausted, eventually leading to reserveSpace (int space) returning false and Recycler will completely invalid.

@normanmaurer
Copy link
Copy Markdown
Member

@netty-bot test this please

@normanmaurer
Copy link
Copy Markdown
Member

@YuanHuCoding can you also sign our icla and let me know once done:

https://netty.io/s/icla

@YuanHuCoding
Copy link
Copy Markdown
Contributor Author

YuanHuCoding commented Jul 20, 2019

`

public class Test_Recycler_availableSharedCapacity {

public static void  main(String[] args) {
    WorkTask workTask = new WorkTask();
    for (int i=0;i<10000;i++) {
        WORK_THREAD_POOL.submit(workTask);
    }
}


private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(Test_Recycler_availableSharedCapacity.class);
private static final ThreadPoolExecutor WORK_THREAD_POOL = new ThreadPoolExecutor(1, 1,
        0L, TimeUnit.MILLISECONDS,
        new LinkedBlockingQueue<Runnable>(102400),new DefaultThreadFactory("WorkThreadPool",false,Thread.NORM_PRIORITY));


private static final ThreadPoolExecutor RECYCLE_THREAD_POOL = new ThreadPoolExecutor(1, 1,
        0L, TimeUnit.MILLISECONDS,
        new LinkedBlockingQueue<Runnable>(102400),new DefaultThreadFactory("RecycleThreadPool",false,Thread.NORM_PRIORITY));

private static LongAdder NEW_OBJECT_NUM=new LongAdder();

//If you can't reproduce it, you can try it several times more, or you can reduce the parameter maxCapacityPerThread to make it easier to reproduce.
private static final Recycler<Cycler> CyclerRecycler = new Recycler<Cycler>(256,1,1,Integer.MAX_VALUE) {
    @Override
    protected Cycler newObject(Handle<Cycler> handle) {
        NEW_OBJECT_NUM.increment();
        logger.info("NEW_OBJECT_NUM:"+NEW_OBJECT_NUM.longValue());
        return new Cycler(handle);
    }
};
static final class Cycler {
    private String value;
    public void setValue(String value) {
        this.value = value;
    }
    private Recycler.Handle<Cycler> handle;
    public Cycler(Recycler.Handle<Cycler> handle) {
        this.handle = handle;
    }
    public void recycle() {
        handle.recycle(this);
    }
}

static class WorkTask implements Runnable{
    public WorkTask(){}
    @Override
    public void run() {
        Cycler cycler1 = CyclerRecycler.get();
        cycler1.setValue("hello,netty");
        RECYCLE_THREAD_POOL.submit(() -> {
            cycler1.recycle();
        });
    }
}}

`
Test result:
2019-07-20 23:17:22,256[INFO ]me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity-NEW_OBJECT_NUM:5501
2019-07-20 23:17:22,256[INFO ]me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity-NEW_OBJECT_NUM:5502
2019-07-20 23:17:22,256[INFO ]me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity-NEW_OBJECT_NUM:5503
2019-07-20 23:17:22,256[INFO ]me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity-NEW_OBJECT_NUM:5504

If you add that line of code:this.head.reclaimSpace(LINK_CAPACITY);,The number of new objects created has been greatly reduced.
Test result:
2019-07-20 23:17:42,982[INFO ]me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity-NEW_OBJECT_NUM:30
2019-07-20 23:17:42,986[INFO ]me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity-NEW_OBJECT_NUM:31
2019-07-20 23:17:43,004[INFO ]me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity-NEW_OBJECT_NUM:32
2019-07-20 23:17:43,014[INFO ]me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity-NEW_OBJECT_NUM:33

You can also use the version I added statistics to make it easier to see the problem.
Recycler_statistics.txt

The test results are as follows:
2019-07-20 23:14:15,980[INFO ]io.netty.util.Recycler-RECYCLER_STATISTICS_NEW_OBJECT:{class me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity$Cycler=9472}
2019-07-20 23:14:15,980[INFO ]io.netty.util.Recycler-RECYCLER_STATISTICS_RECYCLE:{class me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity$Cycler=528}
2019-07-20 23:14:15,980[INFO ]io.netty.util.Recycler-RECYCLER_STATISTICS_DROP:{class me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity$Cycler=9471}
2019-07-20 23:14:15,980[INFO ]io.netty.util.Recycler-reclaimSpaceNum:17
2019-07-20 23:14:15,980[INFO ]io.netty.util.Recycler-reserveSpaceNum:32

If you add that line of code:this.head.reclaimSpace(LINK_CAPACITY);
Test result:
2019-07-20 23:13:58,711[INFO ]io.netty.util.Recycler-RECYCLER_STATISTICS_NEW_OBJECT:{class me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity$Cycler=30}
2019-07-20 23:13:58,711[INFO ]io.netty.util.Recycler-RECYCLER_STATISTICS_RECYCLE:{class me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity$Cycler=9999}
2019-07-20 23:13:58,711[INFO ]io.netty.util.Recycler-RECYCLER_STATISTICS_DROP:{class me.yuanhu.core.io.test.Unit_Recycler_availableSharedCapacity$Cycler=0}
2019-07-20 23:13:58,711[INFO ]io.netty.util.Recycler-reclaimSpaceNum:624
2019-07-20 23:13:58,711[INFO ]io.netty.util.Recycler-reserveSpaceNum:624

@normanmaurer normanmaurer merged commit 94f3930 into netty:4.1 Jul 21, 2019
@normanmaurer
Copy link
Copy Markdown
Member

@YuanHuCoding thanks a lot...great catch!

@normanmaurer normanmaurer added this to the 4.1.38.Final milestone Jul 21, 2019
normanmaurer pushed a commit that referenced this pull request Jul 21, 2019
… reclaimSpace(...) call (#9394)

Motivation:

We did miss to call reclaimSpace(...) in one case which can lead to the situation of having the Recycler to not correctly reclaim space and so just create new objects when not needed.

Modifications:

Correctly call reclaimSpace(...)

Result:

Recycler correctly reclaims space in all situations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants