@CyrusNajmabadi @davidwengier I had to add another workaround (extra commit). PTAL.

It's surprising to me that you have to write all of this code yourself. Is that just to avoid allocations that would happen if you just passed everything through System.Encoding like you do in EncodeMarker?

I was wondering the same thing. There are 2 operations:

1. finding the marker in the buffer
   The outer loop needs to be (memory) efficient since the REPL can pump a lot of output to the Interactive Window. So we don't want to allocate too much there. That's why we reuse the buffer. Once we find the marker we don't care about memory since the marker indicates process crash, so we don't expect any more output than what CLR prints for stack overflow. When we find the first character of the marker in the buffer we need to match the whole marker in the buffer - that still needs to be efficient since it could be a valid input. We need to fill the buffer from the reader with the length of the marker because it might not have been all read in yet. ReadAll does that. Then we compare the buffer content with the marker (ArraysEqual).

2. When we hit the marker we need to transcode the rest of the text coming out of the process. .NET Framework just prints out Process is terminated due to StackOverflowException but Core CLR prints out more info (including stack trace). All this output needs to be transcoded. The transcoding itself can't be done with existing Encoding class since no encoding exists that decodes 2 ASCII bytes stuffed into a UTF16 codepoint to an ASCII character. That's why we need custom Transcode impl.