In the old code when adding or removing data from a socket buffer, a socket.c function would update the file state (called the "status" in the C code), then a tcp.c function might also update the file state. For example tcp_receiveUserData would call legacysocket_removeFromInputBuffer which would update the state, and then tcp_receiveUserData would later also update the state. This would lead to situations where socket.c would remove the READABLE flag, and then tcp.c would immediately add it back. While the end result is that the flag would effectively appear unchanged, this would cause edge-triggered epoll to think that the file's state changed twice even though it effectively didn't, leading to #3274.

Since the TCP code is the only C socket code left, now the socket.c code no longer updates the file state (with an exception of legacysocket_pullOutPacket) and the tcp.c code is responsible for updating the file state.

The added test was authored by @ppoth in #3274.

We should also be testing other socket/file types, but I think this will be easier once the new test_epoll_edge.rs tests in #3243 are merged. I did test udp and unix sockets manually and the new test passes for them as well.

Closes #3274.

Edit: The runtime performance is equivalent, but the simulation results change, so I want to take another look at this. https://github.com/shadow/benchmark-results/blob/master/tor/2024-01-07-T22-11-55/README.md

Edit 2: I think it's expected that the simulation results would change. Before we would remove the READABLE flag and then add it back again, which would reset the epoll entry's priority each time a packet was pushed to a TCP socket. Now the priority is reset only when the event is collected. This means that epoll events will be processed in a different order than they previously were.