Flaky integration test test_storage_rabbitmq #71049
Description
Al rabbitmq tests have failed or thrown an error. Checking https://s3.amazonaws.com/clickhouse-test-reports/66532/ab2380a9f09c0ce89ea31311ab01acd87e677ce8/integration_tests__tsan__[6_6]/integration_run_parallel4_0.tar.zst
I see:
1065:2024-10-24 17:02:20.281556+00:00 [error] <0.1566.0> Channel error on connection <0.1557.0> (172.16.4.4:48356 -> 172.16.4.2:5672, vhost: '/', user: 'root'), channel 1:
1066:2024-10-24 17:02:20.281556+00:00 [error] <0.1566.0> operation exchange.declare caused a channel exception precondition_failed: inequivalent arg 'durable' for exchange 'f' in vhost '/': received 'true' but current is 'false'
1222:2024-10-24 17:04:06.936672+00:00 [error] <0.2256.0> Channel error on connection <0.2247.0> (172.16.4.1:63724 -> 172.16.4.2:5672, vhost: '/', user: 'root'), channel 1:
1223:2024-10-24 17:04:06.936672+00:00 [error] <0.2256.0> operation queue.declare caused a channel exception not_found: no queue 'rabbit_queue_drop' in vhost '/'
1695:2024-10-24 17:06:00.398984+00:00 [error] <0.6314.0> Channel error on connection <0.5754.0> (172.16.4.4:50816 -> 172.16.4.2:5672, vhost: '/', user: 'root'), channel 4:
1696:2024-10-24 17:06:00.398984+00:00 [error] <0.6314.0> operation queue.delete caused a channel exception precondition_failed: queue '1_many_consumers' in vhost '/' in use
1698:2024-10-24 17:06:00.632674+00:00 [error] <0.6328.0> Channel error on connection <0.5798.0> (172.16.4.4:40384 -> 172.16.4.2:5672, vhost: '/', user: 'root'), channel 4:
1699:2024-10-24 17:06:00.632674+00:00 [error] <0.6328.0> operation queue.delete caused a channel exception precondition_failed: queue '1_many_consumers' in vhost '/' in use
1701:2024-10-24 17:06:00.858751+00:00 [error] <0.6342.0> Channel error on connection <0.5828.0> (172.16.4.4:40388 -> 172.16.4.2:5672, vhost: '/', user: 'root'), channel 4:
1702:2024-10-24 17:06:00.858751+00:00 [error] <0.6342.0> operation queue.delete caused a channel exception precondition_failed: queue '1_many_consumers' in vhost '/' in use
1974:2024-10-24 17:07:21.832520+00:00 [error] <0.2646.0> closing AMQP connection <0.2646.0> (172.16.4.1:60108 -> 172.16.4.2:5672, duration: '3M, 0s'):
1975:2024-10-24 17:07:21.832520+00:00 [error] <0.2646.0> missed heartbeats from client, timeout: 60s
2306:2024-10-24 17:10:49.510519+00:00 [error] <0.9585.0> Channel error on connection <0.9148.0> (172.16.4.4:56446 -> 172.16.4.2:5672, vhost: '/', user: 'root'), channel 4:
2307:2024-10-24 17:10:49.510519+00:00 [error] <0.9585.0> operation queue.delete caused a channel exception precondition_failed: queue '1_random' in vhost '/' in use
2353:2024-10-24 17:11:07.793439+00:00 [error] <0.9729.0> Error on AMQP connection <0.9729.0> (172.16.4.1:58558 -> 172.16.4.2:5672, vhost: '/', user: 'root', state: running), channel 0:
2354:2024-10-24 17:11:07.793439+00:00 [error] <0.9729.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2355:2024-10-24 17:11:07.793444+00:00 [error] <0.9701.0> Error on AMQP connection <0.9701.0> (172.16.4.4:53246 -> 172.16.4.2:5672, vhost: '/', user: 'root', state: running), channel 0:
2356:2024-10-24 17:11:07.793444+00:00 [error] <0.9701.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2357:2024-10-24 17:11:07.793451+00:00 [error] <0.9663.0> Error on AMQP connection <0.9663.0> (172.16.4.4:53230 -> 172.16.4.2:5672, vhost: '/', user: 'root', state: running), channel 0:
2358:2024-10-24 17:11:07.793451+00:00 [error] <0.9663.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2671:2024-10-24 17:11:16.140095+00:00 [debug] <0.296.0> ra_log_segment_writer: error sending ra_log_event to: RABBIT6KG0VW9M7ZBN. Error: No Pid
3246:2024-10-24 17:14:21.558924+00:00 [error] <0.737.0> closing AMQP connection <0.737.0> (172.16.4.4:60680 -> 172.16.4.2:5672, duration: '3M, 0s'):
3247:2024-10-24 17:14:21.558924+00:00 [error] <0.737.0> missed heartbeats from client, timeout: 60s
3291:2024-10-24 17:27:20.168729+00:00 [error] <0.803.0> closing AMQP connection <0.803.0> (172.16.4.4:44368 -> 172.16.4.2:5672, duration: '3M, 0s'):
3292:2024-10-24 17:27:20.168729+00:00 [error] <0.803.0> missed heartbeats from client, timeout: 60s
3303:2024-10-24 17:37:36.696945+00:00 [error] <0.1012.0> closing AMQP connection <0.1012.0> (172.16.4.4:37974 -> 172.16.4.2:5672, duration: '3M, 0s'):
3304:2024-10-24 17:37:36.696945+00:00 [error] <0.1012.0> missed heartbeats from client, timeout: 60s
I think it looks a setup/configuration issue for rabbitmq, but I can't pinpoint the root cause.