Recently, we've added a test to the Vert.x core test suite that requires some memory resources (a few dozen MB). The build passes with all transports except io_uring.

We captured a heap dump on OutOfMemoryError. Here's what we found.

When Vert.x creates a NetClient, it registers a Java cleaner task (in case the user does not close the client properly).

This task holds a reference to the corresponding Vert.x instance, even when the client is closed properly.
eclipse-vertx/vert.x#6109 is going to address the problem.

Regardless of this Vert.x specific issue, we noticed that after Vert.x / Netty are closed, each IoUringIoHandler still holds a MsgHdrMemoryArray of 1024 entries that retains about 600kB.

When the ioHandler is destroyed, the array is released:

Each entry is released as well:

But that doesn't do anything since all entries are created with null cleanables:

And the release method is:

So, for each closed Vert.x // Netty instance, about 8 event loops retain their own ioHandler, which retains about 600kB. In our test suite, when the OOM happens, we have about 200 closed Vert.x // Netty instances hold by the global Vert.x cleaner. That adds up to about a gigabyte 😅

Again, this is going to be addressed in Vert.x by eclipse-vertx/vert.x#6109, but perhaps could we improve things on the Netty side as well?

How about filling the array with nulls when MsgHdrMemoryArray.release() is invoked?

    void release() {
        assert !released;
        released = true;
        Arrays.fill(hdrs, null);
        msgHdrMemoryArrayMemoryCleanable.clean();
    }

Wouldn't the GC get a chance to do its job earlier?

Perhaps this happens because we run the test suite with JDK11, that doesn't have the offsetSlice method?