Expected behavior

A task scheduled on an EpollEventLoop is expected to fire after the specified delay.

Actual behavior

Starting from Netty-4.1.76, scheduled tasks may be delayed or missed in applications that are regularly interrupted by signals.

Steps to reproduce

It is hard to reproduce in isolation. I'll describe the behavior we've seen in our test environment, and will try to come up with minimal code example demonstrating the problem.

Our server application is using scheduled tasks to track different types of timeouts (connection/stream idle timeouts, connection time-to-live, etc). After upgrading Netty from 4.1.68 to 4.1.111 the tests verifying server timeouts started failing because the scheduled tasks were delayed by a few seconds or never triggered. I was able to track the problem down to a change in Netty's epollWait behavior - when a scheduled task is submitted to an EpollEventLoop, and there are no other I/O activity on the interest list, it blocks for a longer time than specified timeout. By adding some debug diagnostics around syscall to epoll_wait I could see that when the syscall is interrupted by a signal, netty_epoll_native_epollWait would retry it with the same timeout value, effectively resetting the timeout duration.

In our case the application uses a profiler library that triggers SIGVTALRM signal every second, and in our test environment, the signal was consistently landing on the thread serving an EpollEventLoop, and interrupted epoll_wait calls. It goes like this:

• Application schedules a task with 2 second delay
• No other scheduled tasks on the event loop, so the poll deadline is now + 2000ms
• netty_epoll_native_epollWait calls epoll_wait with timeout=2000ms in a loop:

    do {
        result = epoll_wait(efd, ev, len, timeout);
        if (result >= 0) {
            return result;
        }
    } while((err = errno) == EINTR);

• One second later a SIGVTALRM signal interrupts the thread, and epoll_wait call fails with EINTR error
• The call is restarted with a fresh timeout of 2000ms, resetting the wait
• One second later another SIGVTALRM signal interrupts the thread, etc ...

As far as I understood, this has not been a problem with older versions of Netty because prior to Epoll timer optimization introduced in 4.1.76, netty_epoll_native_epollWait0 has always armed a timer descriptor via timerfd_settime, and called epoll_wait with indefinite (-1) timeout. The timer descriptor would keep the target notification deadline, and does not need to be adjusted for interrupted and retried system calls.

One way to fix this would be to track the time spent in a failed epoll_wait call, and cut down the timeout value by that amount when retrying interrupted calls. I didn't get a chance to try it on newer Linux kernels, but I think the same applies to epoll_pwait2 retry loop in netty_epoll_native_epollWait0

In fact, when I disable Epoll timer optimization by setting io.netty.channel.epoll.epollWaitThreshold system property to 0, it falls back to old behavior, and our tests are passing again.

Minimal yet complete reproducer code (or URL to code)

WIP. It is hard to predict which thread will pick up a signal in a multi-threaded application (decided by kernel and implementation/version dependent). So far could only reproduce this in our test environment, where signal lands on the EpollEventLoop thread serving a connection. In a similar minimal example app, the same signal is picked up by other random threads.

Netty version

The issue originally discovered in 4.1.111, but it seems to go back to 4.1.76 when Epoll timer optimization was introduced - 86004b7

JVM version (e.g. java -version)

openjdk version "17.0.12" 2024-07-16 LTS
OpenJDK Runtime Environment 1.0.2048.0 (build 17.0.12+8-LTS)
OpenJDK 64-Bit Server VM 1.0.2048.0 (build 17.0.12+8-LTS, mixed mode, sharing)

OS version (e.g. uname -a)

Linux [...].us-west-2.compute.internal 4.14.352-267.564.amzn2.aarch64 #1 SMP Tue Aug 27 09:56:36 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux