When Tendermint is ran out of process, the communication goes over sockets. This is a costly overhead, since sockets are running the full TCP protocol. This make sense if the consensus engine and the state machine were running on different physical machines. However I don't see a genuine use case for that / a need to optimize the bulk of software for that, primarily because that further increases the latency of IPC and decreases throughput.

Instead I propose we use Unix named pipes. These are still first in first out, so messages are delivered in order as we would want. Due to the removal of the entire TCP stack, named pipes are empirically faster. See figures 1 and 2 of this study: http://pages.cs.wisc.edu/~adityav/Evaluation_of_Inter_Process_Communication_Mechanisms.pdf

From figure 1, the latency difference for IPC between sockets and pipes is substantial. It appears to me that at small message sizes (less than 1kb), the average pipe latency is 25% that of the socket latency. At the maximum TCP packet size, 64kb, the average pipe latency is 73% that of the socket latency. The former has large performance implications for IPC of small messages (our current deliver txs one by one model), and the latter indicates there are still significant performance gains even if we switched to delivering a full block at once.

Figure 2 shows that there is always a higher throughput for pipes than sockets as well. This is quite significant at small message sizes, (Appears to be a roughly 2x improvement?) and at large message sizes (512kb) the througput improvement is significant again. (roughly 2x as well)