I'm looking into ways that may improve SocketAsyncEngine.Unix performance.

Based on kestrel-linux-transport implementation, this is what I'm thinking of:

• Batch receives and sends on the epoll thread using Linux AIO. Note that this isn't the POSIX AIO library. The functions called here map directly to syscalls.

and

class SocketAsyncEventArgs
{
  public bool RunContinuationsAsynchronously { get; set; } = true;
  public bool PreferSynchronousCompletion { get; set; } = true;
}

• Setting RunContinuationsAsynchronously to false allows SocketAsyncEngine to invoke callbacks directly from the epoll thread. This avoid context switching cost.

• Setting PreferSynchronousCompletion to false allows SocketAsyncEngine to not try a synchronous attempt and go async immediately. An implementation could set this to true for sends, and false for receives. Or for receives it could be changed based on whether the last receive filled up the buffer completely. Going async immediately saves an attempt syscall and allows batching.

The defaults match the current behavior.

io_uring is also an interesting option to explore. I'm not looking into that atm because the features we need are in 5.5 kernel which was released only a couple of days ago.

I'm going to do a poc in a separate repo for benchmarking. Anyone interested is free to review PRs, or fork and do some experimentation. When I have a working implementation, I'll need some help with benchmarking and see what is the change in perf.

cc @stephentoub @geoffkizer @davidfowl @halter73 @adamsitnik @benaadams @VSadov @damageboy @lpereira @dotnet/ncl