Skip to content

Stream protocol: fix a crash in parse_command/1 on zero-size frames#15703

Merged
michaelklishin merged 2 commits intorabbitmq:mainfrom
amazon-mq:fix/stream-parser-zero-bytes
Mar 11, 2026
Merged

Stream protocol: fix a crash in parse_command/1 on zero-size frames#15703
michaelklishin merged 2 commits intorabbitmq:mainfrom
amazon-mq:fix/stream-parser-zero-bytes

Conversation

@lukebakken
Copy link
Copy Markdown
Collaborator

@lukebakken lukebakken commented Mar 11, 2026

When a client sends a zero-size frame — for example, a port scanner sending <<0,0,0,0,0,0,0,0>> to the TLS-enabled stream listeners port, parse_command/1 is called with an empty binary <<>>. No existing clause matches, causing a function_clause crash that terminates the stream connection process. This has been observed consistently on EC2-hosted brokers.

Add a catch-all clause that returns {unknown, Data} for any binary that is not a valid request or response frame. {unknown, binary()} is already a valid command() type and is handled gracefully by all three frame handlers in rabbit_stream_reader: pre-auth closes the connection, post-auth sends a protocol-level close frame with RESPONSE_CODE_UNKNOWN_FRAME, and post-close ignores it.

The fix is accompanied by a test that reproduces the exact crash using the 8-byte payload observed in production logs.

@the-mikedavis
Copy link
Copy Markdown
Collaborator

I copied this patch over to an EC2 I have running with rabbitmq_stream enabled and the logs look much cleaner now:

Warning logs...
[ec2-user@ip-<ip-addr> rabbitmq]$ tail -f ~/data/rabbit\@ip-<ip-addr>/log/rabbit\@ip-<ip-addr>.log | grep '\[warning\]'
2026-03-11 18:09:30.332997+00:00 [warning] <0.22557.0> Closing socket #Port<0.3391>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.336423+00:00 [warning] <0.22608.0> Closing socket #Port<0.3399>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.339745+00:00 [warning] <0.22611.0> Closing socket #Port<0.3400>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.343200+00:00 [warning] <0.22614.0> Closing socket #Port<0.3401>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.346358+00:00 [warning] <0.22617.0> Closing socket #Port<0.3402>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.349939+00:00 [warning] <0.22620.0> Closing socket #Port<0.3403>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.357156+00:00 [warning] <0.22623.0> Closing socket #Port<0.3404>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.360390+00:00 [warning] <0.22626.0> Closing socket #Port<0.3405>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.363733+00:00 [warning] <0.22629.0> Closing socket #Port<0.3406>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.367245+00:00 [warning] <0.22632.0> Closing socket #Port<0.3407>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.370684+00:00 [warning] <0.22635.0> Closing socket #Port<0.3408>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.374073+00:00 [warning] <0.22638.0> unknown command {unknown,<<106,129,110,48,129,107,161,3,2,1,5,162,3,2,1,10,164,129,94,48,92,160,7,3,5,0,80,128,0,16,162,4,27,2,78,77,163,23,48,21,160,3,2,1,0,161,14,48,12,27,6,107,114,98,116,103,116,27,2,78,77,165,17,24,15,49,57,55,48,48,49,48,49,48,48,48,48,48,48,90,167,6,2,4,31,30,185,217,168,23,48,21,2,1,18,2,1,17,2,1,16,2,1,23,2,1,1,2,1,3,2,1,2>>}, closing connection.
2026-03-11 18:09:30.374346+00:00 [warning] <0.22638.0> Closing socket #Port<0.3409>. Invalid transition from tcp_connected to failure.
2026-03-11 18:09:30.377875+00:00 [warning] <0.22641.0> unknown command {unknown,<<255,83,77,66,114,0,0,0,0,8,1,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,64,6,0,0,1,0,0,129,0,2,80,67,32,78,69,84,87,79,82,75,32,80,82,79,71,82,65,77,32,49,46,48,0,2,77,73,67,82,79,83,79,70,84,32,78,69,84,87,79,82,75,83,32,49,46,48,51,0,2,77,73,67,82,79,83,79,70,84,32,78,69,84,87,79,82,75,83,32,51,46,48,0,2,76,65,78,77,65,78,49,46,48,0,2,76,77,49,46,50,88,48,48,50,0,2,83,97,109,98,97,0,2,78,84,32,76,65,78,77,65,78,32,49,46,48,0,2,78,84,32,76,77,32,48,46,49,50,0>>}, closing connection.
2026-03-11 18:09:30.378012+00:00 [warning] <0.22641.0> Closing socket #Port<0.3410>. Invalid transition from tcp_connected to failure.
2026-03-11 18:09:30.381482+00:00 [warning] <0.22644.0> Closing socket #Port<0.3411>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.384927+00:00 [warning] <0.22647.0> Closing socket #Port<0.3412>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.389007+00:00 [warning] <0.22650.0> Closing socket #Port<0.3413>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.392264+00:00 [warning] <0.22653.0> Closing socket #Port<0.3414>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.395933+00:00 [warning] <0.22656.0> Closing socket #Port<0.3415>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.399644+00:00 [warning] <0.22659.0> Closing socket #Port<0.3416>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.403163+00:00 [warning] <0.22662.0> Closing socket #Port<0.3417>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.406940+00:00 [warning] <0.22665.0> Closing socket #Port<0.3418>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.410676+00:00 [warning] <0.22668.0> Closing socket #Port<0.3419>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.414849+00:00 [warning] <0.22671.0> Closing socket #Port<0.3420>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.418075+00:00 [warning] <0.22674.0> Closing socket #Port<0.3421>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.421797+00:00 [warning] <0.22677.0> Closing socket #Port<0.3422>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.425036+00:00 [warning] <0.22680.0> Closing socket #Port<0.3423>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.428853+00:00 [warning] <0.22683.0> Closing socket #Port<0.3424>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.432425+00:00 [warning] <0.22686.0> Closing socket #Port<0.3425>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.435867+00:00 [warning] <0.22689.0> Closing socket #Port<0.3426>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.439428+00:00 [warning] <0.22692.0> Closing socket #Port<0.3427>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.442727+00:00 [warning] <0.22695.0> unknown command {unknown,<<0,0,0,96,123,34,99,111,100,101,34,58,50,56,44,34,102,108,97,103,34,58,48,44,34,108,97,110,103,117,97,103,101,34,58,34,74,65,86,65,34,44,34,111,112,97,113,117,101,34,58,48,44,34,115,101,114,105,97,108,105,122,101,84,121,112,101,67,117,114,114,101,110,116,82,80,67,34,58,34,74,83,79,78,34,44,34,118,101,114,115,105,111,110,34,58,52,51,51,125>>}, closing connection.
2026-03-11 18:09:30.443253+00:00 [warning] <0.22695.0> Closing socket #Port<0.3428>. Invalid transition from tcp_connected to failure.
2026-03-11 18:09:30.446668+00:00 [warning] <0.22698.0> Closing socket #Port<0.3429>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.450042+00:00 [warning] <0.22701.0> unknown command {unknown,<<>>}, closing connection.
2026-03-11 18:09:30.450098+00:00 [warning] <0.22701.0> unknown command {unknown,<<>>}, closing connection.
2026-03-11 18:09:30.450179+00:00 [warning] <0.22701.0> Closing socket #Port<0.3430>. Invalid transition from tcp_connected to failure.
2026-03-11 18:09:30.453592+00:00 [warning] <0.22704.0> Closing socket #Port<0.3431>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.456945+00:00 [warning] <0.22707.0> Closing socket #Port<0.3432>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.460704+00:00 [warning] <0.22710.0> Closing socket #Port<0.3433>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.464353+00:00 [warning] <0.22713.0> Closing socket #Port<0.3434>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.467698+00:00 [warning] <0.22716.0> Closing socket #Port<0.3435>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.471145+00:00 [warning] <0.22719.0> Closing socket #Port<0.3436>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.474653+00:00 [warning] <0.22722.0> Closing socket #Port<0.3437>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.478863+00:00 [warning] <0.22725.0> Closing socket #Port<0.3438>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.482301+00:00 [warning] <0.22728.0> Closing socket #Port<0.3439>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.485594+00:00 [warning] <0.22731.0> Closing socket #Port<0.3440>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.491679+00:00 [warning] <0.22734.0> Closing socket #Port<0.3441>. Invalid transition from tcp_connected to tcp_connected.
2026-03-11 18:09:30.500093+00:00 [warning] <0.22737.0> Closing socket #Port<0.3442>. Invalid transition from tcp_connected to tcp_connected.

We could probably also clean up the logs around invalid transitions, but that could be done separately from this fix.

A zero-size frame (a 4-byte header with value 0) causes
`parse_command/1` to be called with an empty binary `<<>>`, for which
no clause matches, crashing the stream connection process.

Add a test case that reproduces the crash by sending the exact 8-byte
payload observed in production logs from EC2-hosted brokers.
When a client sends a zero-size frame (e.g. a port scanner sending
`<<0,0,0,0,0,0,0,0>>`), `parse_command/1` is called with an empty
binary `<<>>`. No existing clause matches, causing a `function_clause`
crash that terminates the stream connection process.

Add a catch-all clause that returns `{unknown, Data}` for any binary
that is not a valid request or response frame. This is already a valid
`command()` type and is handled gracefully by all three frame handlers
in `rabbit_stream_reader`: pre-auth closes the connection, post-auth
sends a protocol-level close frame, and post-close ignores it.
@lukebakken lukebakken force-pushed the fix/stream-parser-zero-bytes branch from 830092f to 3510476 Compare March 11, 2026 18:40
@michaelklishin michaelklishin added this to the 4.3.0 milestone Mar 11, 2026
@michaelklishin michaelklishin changed the title Fix crash in parse_command/1 on zero-size frames Stream protocol: fix a crash in parse_command/1 on zero-size frames Mar 11, 2026
@michaelklishin michaelklishin merged commit 6443b05 into rabbitmq:main Mar 11, 2026
182 checks passed
@michaelklishin
Copy link
Copy Markdown
Collaborator

@lukebakken @the-mikedavis thank you!

@the-mikedavis the-mikedavis deleted the fix/stream-parser-zero-bytes branch March 11, 2026 21:57
the-mikedavis added a commit that referenced this pull request Mar 11, 2026
Stream protocol: fix a crash in `parse_command/1` on zero-size frames (backport #15703)
lukebakken pushed a commit to amazon-mq/upstream-to-rabbitmq-server that referenced this pull request Mar 11, 2026
…ro-bytes

Stream protocol: fix a crash in `parse_command/1` on zero-size frames

(cherry picked from commit 6443b05)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants