Fix race condition in message store when GC deletes file

lukebakken · lukebakken · commit 8f27887ecfa7 · 2026-02-06T20:56:05.000Z
Handle case where garbage collector asynchronously deletes a file from
FileSummaryEts while a write operation is trying to reference it. This
race condition can occur during high load (e.g., maintenance drain with
heavy queue leadership transfers) when:

1. Message exists in index with ref_count=0, file=N
2. File N is marked for deletion (valid_total_size=0)
3. GC asynchronously deletes file N from FileSummaryEts
4. Write request arrives for the message
5. ets:lookup(FileSummaryEts, N) returns []
6. Code crashes with {case_clause, {false, []}}

The fix adds two new case clauses to handle this scenario:

- {false, []} - Delete stale index entry and write fresh copy
- {false_if_increment, []} - Ignore write (client dying)

This follows the same pattern as existing clauses that handle files
being deleted or locked. Since ref_count=0, the old copy is orphaned
and safe to discard.

Observed in production during maintenance window causing 969 queue
process crashes. See ticket V2090892319.
diff --git a/deps/rabbit/src/rabbit_msg_store.erl b/deps/rabbit/src/rabbit_msg_store.erl
@@ -1122,7 +1122,20 @@ write_action({Mask, #msg_location { ref_count = 0, file = File,
         {_Mask, [#file_summary {}]} ->
             ok = index_update_ref_counter(IndexEts, MsgId, +1), %% Effectively set to 1.
             State1 = adjust_valid_total_size(File, TotalSize, State),
-            {confirm, File, State1}
+            {confirm, File, State1};
+        {false, []} ->
+            %% Handle case where GC has deleted the file from the summary table
+            %% before we could look it up. This can occur during high load when
+            %% the GC asynchronously deletes files while write operations are
+            %% in progress. Since ref_count=0, the old copy is orphaned and
+            %% safe to discard. Delete stale index entry and write fresh copy
+            %% to current file.
+            ok = index_delete(IndexEts, MsgId),
+            {write, State};
+        {false_if_increment, []} ->
+            %% File deleted by GC, but client is dying - ignore the write
+            %% since the message will be deleted when client death is processed.
+            {ignore, File, State}
     end;
 write_action({_Mask, #msg_location { file = File }},
              MsgId, State = #msstate{ index_ets = IndexEts }) ->
