Skip to content

Replace catch with try/catch in handle_leader/2#596

Merged
kjnilsson merged 2 commits intomainfrom
handle_leader
Mar 19, 2026
Merged

Replace catch with try/catch in handle_leader/2#596
kjnilsson merged 2 commits intomainfrom
handle_leader

Conversation

@ansd
Copy link
Copy Markdown
Member

@ansd ansd commented Mar 19, 2026

The catch expression captured the full exception term including
the stacktrace, which could contain the entire Ra server state
in its function arguments. Logging this with ~p caused
io_lib_pretty to traverse the massive term, triggering extreme
memory allocation and OOM crashes for servers with large machine
states (e.g. millions of messages).

In this case the node crashed due to OOM because there was a
function_clause error in the state machine implementation where the
state machine had a huge state.
We still need the stack trace to locate where the function_clause
originates from.

Prior to this commit, the following function_clause caused the node to run out of memory. With this commit, we will get the following log printed:

 queue 'my-jms-queue' in vhost '/': handle_leader err error function_clause
 [{rabbit_jms_machine,merge_consumer,7,
                      [{file,"rabbit_jms_machine.erl"},{line,2240}]},
  {rabbit_jms_machine,apply_,3,[{file,"rabbit_jms_machine.erl"},{line,409}]},
  {ra_server,apply_with,2,[{file,"src/ra_server.erl"},{line,3279}]},
  {ra_log_segment,fold0,7,[{file,"src/ra_log_segment.erl"},{line,674}]},
  {ra_log_segments,'-segment_fold/6-fun-0-',5,
                   [{file,"src/ra_log_segments.erl"},{line,592}]},
  {ra_log_segments,'-segment_fold/6-lists^foldl/2-1-',3,
                   [{file,"src/ra_log_segments.erl"},{line,589}]},
  {ra_log_segments,segment_fold,6,
                   [{file,"src/ra_log_segments.erl"},{line,589}]},
  {ra_log,fold,6,[{file,"src/ra_log.erl"},{line,633}]}]

@ansd ansd marked this pull request as ready for review March 19, 2026 10:05
@ansd ansd marked this pull request as draft March 19, 2026 10:07
The catch expression captured the full exception term including
the stacktrace, which could contain the entire Ra server state
in its function arguments. Logging this with ~p caused
io_lib_pretty to traverse the massive term, triggering extreme
memory allocation and OOM crashes for servers with large machine
states (e.g. millions of messages).

In this case the node crashed due to OOM because there was a
function_clause error in the state machine implementation where the
state machine had a huge state.
We still need the stack trace to locate where the function_clause
originates from.
@ansd ansd marked this pull request as ready for review March 19, 2026 11:49
@kjnilsson kjnilsson merged commit 0a99661 into main Mar 19, 2026
12 of 13 checks passed
@dumbbell-rabbitmq dumbbell-rabbitmq deleted the handle_leader branch March 23, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants