Under load the compiler server was failing to accept client connections faster than the client connect timeout period (one second). This lead clients to abandon the server and shell out to csc / vbc directly. This is a net negative for customers, even when the compiler server is under extreme load, because the new csc / vbc processes and the compiler server will essentially be competing for the same CPU resources. Part of the benefit of the compiler server is to avoid this competition.

The most immediate cause of this problem is that the server only creates a single NamedPipeServerStream for accepting connections and there was a significant amount of processing that had to occur between accepting one connection and creating a new NamedPipeServerStream for accepting new connections. Particularly problematic is that the work required several thread transitions. Under normal processing that was fast enough but under load that was not.

The solution here is essentially two fold:

1. The code between accepting a new connection and creating a new listen pipe is trivial and requires no thread transitions.
2. The server now spins up multiple NamedPipeServerStream instances in parallel to ensure that we can accept connections even when a single thread gets stalled under load.

This change also tightens the tolerance level we have in our dogfood builds for failed connections to zero. This should help us catch regressions in this area in the future.

Note: This change is best read commit by commit.