After some performance analysis, we've discovered that UNIX sockets and semaphores have very high overhead per context switch between shadow and the plugins it is running. Most of the time is spent waiting for the other side of the channel to signal a wakeup.

Some initial rough tests show that shared memory and spinlocks can improve performance as much as 100x.

Of course, we don't want to spin forever and burn CPU cycles; we want an intelligent algorithm that each side of the IPC channel first tries the spinlock for some amount of time, and if the channel isn't ready by that time, it falls back to using semaphores.

It probably makes sense to do this in multiple steps:

• (merged in Use shared memory + spinlock & semaphore to perform IPC message passing in preload mode. #924) Change our shadow-to-plugin channel from sockets to shared memory with spinlocks
• (merged in Use shared memory + spinlock & semaphore to perform IPC message passing in preload mode. #924) Implement the part of the algorithm where we eventually give up on the spinlock and fall back to the semaphore.
• Support dynamic computation of the number of spins before falling back to semaphore
• Pick some reaonable upper bound on the spin time, and print a warning when that time is reached and we fall back to the semaphore

Useful links:

• https://github.com/goldsborough/ipc-bench
• https://stackoverflow.com/questions/2854004/ipc-speed-and-compare