Frequent WebSocket Compression Segfault #86960
Description
Description
I run a service that heavily uses WebSockets, with anywhere from 3-8k concurrent WebSockets connections per server. These are long-lived connections; most are from 12 hours to a few days. I enabled websocket compression because size and latency is essential for my application, but I have been getting a super frequent crash due to it.
I'm running Ubuntu server 22.04 with the most up-to-date MS package repo versions of dotnet 7.0, aspnet, etc. My service uses the WebSocket class directly; I'm not using any middleware for handling the streams. I have around eight server running, and I see about 1-3 crashes per server per day due to this bug.
Please let me know if I can provide any more helpful debugging information. I'm able to attach lldb and wait for a segfault, so I can gather any other info needed.
Reproduction Steps
Set up an aspnet server that accepts websockets and enable web socket compression. Run the server as normal, allowing messages to be sent on the WebSocket and sockets to come and go. Eventually, the dotnet process will segfault, with this stack:
OS Thread Id: 0x4242a (212)
Child SP IP Call Site
00007F2BDD7F9038 00007f2c4d07f401 [InlinedCallFrame: 00007f2bdd7f9038] Interop+ZLib.Deflate(ZStream*, FlushCode)
00007F2BDD7F9038 00007f6c8109b576 [InlinedCallFrame: 00007f2bdd7f9038] Interop+ZLib.Deflate(ZStream*, FlushCode)
00007F2BDD7F9030 00007F6C8109B576 System.Net.WebSockets.Compression.WebSocketDeflater.Deflate(ZLibStreamHandle, FlushCode)
00007F2BDD7F90C0 00007F6C810F4458 System.Net.WebSockets.Compression.WebSocketDeflater.UnsafeFlush(System.Span`1<Byte>, Boolean ByRef)
00007F2BDD7F90F0 00007F6C810F4268 System.Net.WebSockets.Compression.WebSocketDeflater.DeflatePrivate(System.ReadOnlySpan`1<Byte>, System.Span`1<Byte>, Boolean, Int32 ByRef, Int32 ByRef, Boolean ByRef)
00007F2BDD7F9140 00007F6C810F4012 System.Net.WebSockets.Compression.WebSocketDeflater.Deflate(System.ReadOnlySpan`1<Byte>, Boolean)
00007F2BDD7F91C0 00007F6C80BCBD17 System.Net.WebSockets.ManagedWebSocket.WriteFrameToSendBuffer(MessageOpcode, Boolean, Boolean, System.ReadOnlySpan`1<Byte>)
00007F2BDD7F9210 00007F6C80BCB3C8 System.Net.WebSockets.ManagedWebSocket+<SendFrameFallbackAsync>d__58.MoveNext()
00007F2BDD7F9310 00007F6C80BCB0C7 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.WebSockets.ManagedWebSocket+<SendFrameFallbackAsync>d__58, System.Net.WebSockets]](<SendFrameFallbackAsync>d__58 ByRef)
00007F2BDD7F9350 00007F6C80BCAFE4 System.Net.WebSockets.ManagedWebSocket.SendFrameFallbackAsync(MessageOpcode, Boolean, Boolean, System.ReadOnlyMemory`1<Byte>, System.Threading.Tasks.Task, System.Threading.CancellationToken)
00007F2BDD7F9400 00007F6C80BCAB52 System.Net.WebSockets.ManagedWebSocket.SendAsync(System.ReadOnlyMemory`1<Byte>, System.Net.WebSockets.WebSocketMessageType, System.Net.WebSockets.WebSocketMessageFlags, System.Threading.CancellationToken)
...
<service specific code that's writing to the websocket>
Expected behavior
The dotnet process shouldn't crash; at a minimum, it should throw a managed exception.
Actual behavior
After some time of my service running, the process segfaults with the stack above.
Regression?
This has been happening for a while, I'm not sure when it started, but it's been at least a few months.
Known Workarounds
None. :(
Configuration
Latest dotnet and aspnet packages from the Microsoft package repo for Ubunut Jammy. I also have the most up-to-date system packages.
dotnet sdk 7.0.302
dotnet runtime 7.0.5
aspnet 7.0.5
Ubuntu 22.04.2 LTS (Jammy Jellyfish), x64.
Other information
No response