EPollArrayWrapper.epollWait 100% CPU Usage #327
Description
Hi,
I believe I have an issue similar to #302 but on Linux (Ubuntu 10.04) with JDK (1.6.0u30) and JDK(1.7.0u4) using Netty-4.0.0 (Revision: 52a7d28)
The app is proxying connections to backend systems. The proxy has a pool of channels that it can use to send requests to the backend systems. If the pool is low on channels, new channels are spawned and put into the pool so that requests sent to the proxy can be serviced. The pools get populated on app startup, so that is why it doesn't take long at all for the CPU to spike through the roof (22 seconds into the app lifecycle).
The test box has two CPUs, the output from 'top' is below:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8220 root 20 0 2281m 741m 10m R 50.2 18.7 0:22.57 java
8218 root 20 0 2281m 741m 10m R 49.9 18.7 0:22.65 java
8219 root 20 0 2281m 741m 10m R 49.2 18.7 0:22.86 java
8221 root 20 0 2281m 741m 10m R 49.2 18.7 0:22.20 java
Thread Dump for the four NioClient based Worker Threads that are chewing up all the CPU.
"backend-worker-pool-7-thread-1" prio=10 tid=0x00007f5918015800 nid=0x201a runnable [0x00007f5924ba3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000008be93580> (a sun.nio.ch.Util$2)
- locked <0x000000008be93570> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000008be92548> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at io.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:55)
at io.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
at io.netty.channel.socket.nio.NioWorker.run(NioWorker.java:37)
at io.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:43)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000008be00748> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"backend-worker-pool-7-thread-2" prio=10 tid=0x00007f5918012000 nid=0x201b runnable [0x00007f5924b82000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000008be94a28> (a sun.nio.ch.Util$2)
- locked <0x000000008be94a18> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000008be90648> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at io.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:55)
at io.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
at io.netty.channel.socket.nio.NioWorker.run(NioWorker.java:37)
at io.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:43)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000008be904c8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"backend-worker-pool-7-thread-3" prio=10 tid=0x00007f5918007800 nid=0x201c runnable [0x00007f5924b61000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000008be952e0> (a sun.nio.ch.Util$2)
- locked <0x000000008be952d0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000008be8f858> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at io.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:55)
at io.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
at io.netty.channel.socket.nio.NioWorker.run(NioWorker.java:37)
at io.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:43)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000008be8f618> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"backend-worker-pool-7-thread-4" prio=10 tid=0x00007f5918019000 nid=0x201d runnable [0x00007f5924b40000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000008be003f8> (a sun.nio.ch.Util$2)
- locked <0x000000008be003e8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000008be00408> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at io.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:55)
at io.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
at io.netty.channel.socket.nio.NioWorker.run(NioWorker.java:37)
at io.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:43)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000008be004e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)