Dotnet 6.0 Websocket Deflate Segfault On Ubuntu #62422
Description
Description
I run a service that uses a lot of WebSockets and was very excited to enable the new WebSocket deflation feature in dotnet 6.0. I enabled it and was very happy with the performance gain from the compression of the large json based messages I have to send.
However, after deploying my new dotnet 6.0 build I started getting about 1 server segfault crash a day across all of my servers. I set up lldb, attached to the process, and was able to get a call stack of the segfault. The call stack indicates there was a segfaul in the WebSocket stack while it was trying to compress an outgoing message.
I don't know what else would be helpful for debugging, I'm happy to attach lldb to a server process again and capture more information if needed.
(lldb) continue
Process 830510 resuming
Process 830510 stopped
* thread #51, name = '.NET ThreadPool', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
frame #0: 0x00007f01d1f4cacf libz.so.1`___lldb_unnamed_symbol10$$libz.so.1 + 1535
libz.so.1`___lldb_unnamed_symbol10$$libz.so.1:
-> 0x7f01d1f4cacf <+1535>: movq 0x38(%rbp), %r12
0x7f01d1f4cad3 <+1539>: movq %rax, 0x98(%rbx)
0x7f01d1f4cada <+1546>: movq %r12, %rdi
0x7f01d1f4cadd <+1549>: callq 0x7f01d1f57470 ; ___lldb_unnamed_symbol23$$libz.so.1
(lldb) clrstack
OS Thread Id: 0xcd8e7 (51)
Child SP IP Call Site
00007F01D37F9D10 00007f01d1f4cacf [InlinedCallFrame: 00007f01d37f9d10] Interop+zlib.Deflate(ZStream*, FlushCode)
00007F01D37F9D10 00007f04323e7f17 [InlinedCallFrame: 00007f01d37f9d10] Interop+zlib.Deflate(ZStream*, FlushCode)
00007F01D37F9CE0 00007F04323E7F17 System.Net.WebSockets.Compression.WebSocketDeflater.Deflate(ZLibStreamHandle, FlushCode) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/Compression/WebSocketDeflater.cs @ 191]
00007F01D37F9D90 00007F04323FEEA7 System.Net.WebSockets.Compression.WebSocketDeflater.UnsafeFlush(System.Span`1<Byte>, Boolean ByRef) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/Compression/WebSocketDeflater.cs @ 162]
00007F01D37F9DC0 00007F04323FECE5 System.Net.WebSockets.Compression.WebSocketDeflater.DeflatePrivate(System.ReadOnlySpan`1<Byte>, System.Span`1<Byte>, Boolean, Int32 ByRef, Int32 ByRef, Boolean ByRef) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/Compression/WebSocketDeflater.cs @ 104]
00007F01D37F9E10 00007F04323FEA76 System.Net.WebSockets.Compression.WebSocketDeflater.Deflate(System.ReadOnlySpan`1<Byte>, Boolean)
00007F01D37F9E90 00007F0432026F33 System.Net.WebSockets.ManagedWebSocket.WriteFrameToSendBuffer(MessageOpcode, Boolean, Boolean, System.ReadOnlySpan`1<Byte>) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/ManagedWebSocket.cs @ 501]
00007F01D37F9EE0 00007F043202674C System.Net.WebSockets.ManagedWebSocket+<SendFrameFallbackAsync>d__58.MoveNext() [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/ManagedWebSocket.cs @ 477]
00007F01D37F9FF0 00007F0432026438 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.WebSockets.ManagedWebSocket+<SendFrameFallbackAsync>d__58, System.Net.WebSockets]](<SendFrameFallbackAsync>d__58 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
00007F01D37FA040 00007F0432026363 System.Net.WebSockets.ManagedWebSocket.SendFrameFallbackAsync(MessageOpcode, Boolean, Boolean, System.ReadOnlyMemory`1<Byte>, System.Threading.Tasks.Task, System.Threading.CancellationToken)
00007F01D37FA0F0 00007F0432046CB4 System.Net.WebSockets.ManagedWebSocket.SendAsync(System.ReadOnlyMemory`1<Byte>, System.Net.WebSockets.WebSocketMessageType, System.Net.WebSockets.WebSocketMessageFlags, System.Threading.CancellationToken) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/ManagedWebSocket.cs @ 288]
00007F01D37FA1B0 00007F0432046B10 System.Net.WebSockets.ManagedWebSocket.SendAsync(System.ReadOnlyMemory`1<Byte>, System.Net.WebSockets.WebSocketMessageType, Boolean, System.Threading.CancellationToken) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/ManagedWebSocket.cs @ 250]
00007F01D37FA1F0 00007F0432046617 OctoStream.BetterWebSocket+<Send>d__15.MoveNext() [D:\a\1\s\OctoStream\Util\BetterWebSocket.cs @ 78]
00007F01D37FA2A0 00007F0432046368 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[OctoStream.BetterWebSocket+<Send>d__15, OctoStream]](<Send>d__15 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
00007F01D37FA2F0 00007F04320462B5 OctoStream.BetterWebSocket.Send(System.ReadOnlyMemory`1<Byte>, System.Net.WebSockets.WebSocketMessageType)
Reproduction Steps
Setup a server using WebSocket and WebSocket deflation. Open and close WebSockets, send and receive messages on the WebSockets frequently.
Expected behavior
The server should not segfault.
Actual behavior
After some time of WebSockets opening, closing, sending and receiving messages, my server segfaults. My servers handles about 200-500 concurrent WebSockets that come and go at any given time. It probably processes ~50-100 messages a second across all of the WebSockets connected to the server.
Regression?
No, deflation is a new feature to dotnet 6.0.
Known Workarounds
Disable deflation for WebSockets
Configuration
No response
Other information
No response