Possible deadlock on shutdown of StorageDistributed #53408
Description
(it doesn't look like a real deadlock, because connection attempts will eventually fail with timeout, but it's still bad)
2023-08-14 02:14:45 Thread 187 (Thread 0x7fb0e3fb3640 (LWP 34094) "BgDistSchPool"):
2023-08-14 02:14:45 #0 0x00007fb2d4620d7f in poll () from /lib/x86_64-linux-gnu/libc.so.6
2023-08-14 02:14:45 #1 0x000055b1aa453373 in poll ()
2023-08-14 02:14:45 #2 0x000055b1ea88d094 in Poco::Net::SocketImpl::pollImpl (this=<optimized out>, remainingTime=..., mode=<optimized out>) at ./base/poco/Net/src/SocketImpl.cpp:436
2023-08-14 02:14:45 #3 0x000055b1ea7d4ac5 in Poco::Net::SecureSocketImpl::mustRetry (this=this@entry=0x70b00179b0d0, rc=rc@entry=-1, remaining_time=...) at ./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:480
2023-08-14 02:14:45 #4 0x000055b1ea7d74d1 in Poco::Net::SecureSocketImpl::receiveBytes (this=0x70b00179b0d0, buffer=0x7fb025a4f000, length=1048576, flags=<optimized out>) at ./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:350
2023-08-14 02:14:45 #5 0x000055b1ea7df70d in Poco::Net::SecureStreamSocketImpl::receiveBytes (this=<optimized out>, buffer=0x1, length=289778, flags=-731771521) at ./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:139
2023-08-14 02:14:45 #6 0x000055b1c22b3160 in DB::ReadBufferFromPocoSocket::nextImpl (this=0x70d0007e6788) at ./src/IO/ReadBufferFromPocoSocket.cpp:81
2023-08-14 02:14:45 #7 0x000055b1dea0f312 in DB::ReadBuffer::next (this=<optimized out>) at ./src/IO/ReadBuffer.h:71
2023-08-14 02:14:45 #8 DB::ReadBuffer::eof (this=<optimized out>) at ./src/IO/ReadBuffer.h:105
2023-08-14 02:14:45 #9 DB::Connection::receiveHello (this=this@entry=0x716000a6fe18, handshake_timeout=...) at ./src/Client/Connection.cpp:331
2023-08-14 02:14:45 #10 0x000055b1dea0abf2 in DB::Connection::connect (this=this@entry=0x716000a6fe18, timeouts=...) at ./src/Client/Connection.cpp:190
2023-08-14 02:14:45 #11 0x000055b1dea157d8 in DB::Connection::forceConnected (this=0x716000a6fe18, timeouts=...) at ./contrib/llvm-project/libcxx/include/string:785
2023-08-14 02:14:45 #12 0x000055b1c1fefc5a in DB::ConnectionPool::get (this=<optimized out>, timeouts=..., settings=<optimized out>, force_connected=true) at ./src/Client/ConnectionPool.h:93
2023-08-14 02:14:45 #13 0x000055b1dd7ca6ce in DB::DistributedAsyncInsertDirectoryQueue::processFile (this=this@entry=0x716000311700, file_path=...) at ./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:438
2023-08-14 02:14:45 #14 0x000055b1dd7ba633 in DB::DistributedAsyncInsertDirectoryQueue::processFiles (this=this@entry=0x716000311700) at ./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:405
2023-08-14 02:14:45 #15 0x000055b1dd7bb579 in DB::DistributedAsyncInsertDirectoryQueue::run (this=0x716000311700) at ./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:208
2023-08-14 02:14:45 #16 0x000055b1dd7d2a3f in DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::__1::shared_ptr<DB::IDisk> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::IConnectionPool>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0::operator()() const (this=<optimized out>) at ./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:152
2023-08-14 02:14:45 #17 std::__1::__invoke[abi:v15000]<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::__1::shared_ptr<DB::IDisk> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::IConnectionPool>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&>(DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::__1::shared_ptr<DB::IDisk> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::IConnectionPool>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&) (__f=...) at ./contrib/llvm-project/libcxx/include/__functional/invoke.h:394
2023-08-14 02:14:45 #18 std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::__1::shared_ptr<DB::IDisk> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::IConnectionPool>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&>(DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::__1::shared_ptr<DB::IDisk> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::IConnectionPool>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&) (__args=...) at ./contrib/llvm-project/libcxx/include/__functional/invoke.h:479
2023-08-14 02:14:45 #19 std::__1::__function::__default_alloc_func<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::__1::shared_ptr<DB::IDisk> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::IConnectionPool>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0, void ()>::operator()[abi:v15000]() (this=<optimized out>) at ./contrib/llvm-project/libcxx/include/__functional/function.h:235
2023-08-14 02:14:45 #20 std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::__1::shared_ptr<DB::IDisk> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::IConnectionPool>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0, void ()> >(std::__1::__function::__policy_storage const*) (__buf=<optimized out>) at ./contrib/llvm-project/libcxx/include/__functional/function.h:716
2023-08-14 02:14:45 #21 0x000055b1d6ebb7de in std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const (this=0x7fb0e3fa6830) at ./contrib/llvm-project/libcxx/include/__functional/function.h:848
2023-08-14 02:14:45 #22 std::__1::function<void ()>::operator()() const (this=0x7fb0e3fa6830) at ./contrib/llvm-project/libcxx/include/__functional/function.h:1187
2023-08-14 02:14:45 #23 DB::BackgroundSchedulePoolTaskInfo::execute (this=0x70d000506c08) at ./src/Core/BackgroundSchedulePool.cpp:104
2023-08-14 02:14:45 #24 0x000055b1d6ec450e in DB::BackgroundSchedulePool::threadFunction (this=this@entry=0x712000018000) at ./src/Core/BackgroundSchedulePool.cpp:302
2023-08-14 02:14:45 #25 0x000055b1d6ec6132 in DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const (this=<optimized out>) at ./src/Core/BackgroundSchedulePool.cpp:168
2023-08-14 02:14:45 #26 std::__1::__invoke[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&) (__f=...) at ./contrib/llvm-project/libcxx/include/__functional/invoke.h:394
2023-08-14 02:14:45 #27 std::__1::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::__1::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) (__f=..., __t=...) at ./contrib/llvm-project/libcxx/include/tuple:1789
2023-08-14 02:14:45 #28 std::__1::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::__1::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::__1::tuple<>&) (__f=..., __t=...) at ./contrib/llvm-project/libcxx/include/tuple:1798
2023-08-14 02:14:45 #29 ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::{lambda()#1}::operator()() (this=<optimized out>) at ./src/Common/ThreadPool.h:242
2023-08-14 02:14:45 #30 std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::{lambda()#1}&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&) (__f=...) at ./contrib/llvm-project/libcxx/include/__functional/invoke.h:394
2023-08-14 02:14:45 #31 std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::{lambda()#1}&>(ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::{lambda()#1}&) (__args=...) at ./contrib/llvm-project/libcxx/include/__functional/invoke.h:479
2023-08-14 02:14:45 #32 std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::{lambda()#1}, void ()>::operator()[abi:v15000]() (this=<optimized out>) at ./contrib/llvm-project/libcxx/include/__functional/function.h:235
2023-08-14 02:14:45 #33 std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::{lambda()#1}, void ()> >(std::__1::__function::__policy_storage const*) (__buf=<optimized out>) at ./contrib/llvm-project/libcxx/include/__functional/function.h:716
2023-08-14 02:14:45 #34 0x000055b1c1da2dfb in std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const (this=0x7fb0e3faa758) at ./contrib/llvm-project/libcxx/include/__functional/function.h:848
2023-08-14 02:14:45 #35 std::__1::function<void ()>::operator()() const (this=0x7fb0e3faa758) at ./contrib/llvm-project/libcxx/include/__functional/function.h:1187
2023-08-14 02:14:45 #36 ThreadPoolImpl<std::__1::thread>::worker (this=0x711000003980, thread_it=...) at ./src/Common/ThreadPool.cpp:426
2023-08-14 02:14:45 #37 0x000055b1c1db1a0b in ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::{lambda()#2}::operator()() const (this=<optimized out>) at ./src/Common/ThreadPool.cpp:179
2023-08-14 02:14:45 #38 std::__1::__invoke[abi:v15000]<ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::{lambda()#2}>(ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::{lambda()#2}&&) (__f=...) at ./contrib/llvm-project/libcxx/include/__functional/invoke.h:394
2023-08-14 02:14:45 #39 std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::{lambda()#2}>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::{lambda()#2}>&, std::__1::__tuple_indices<>) (__t=...) at ./contrib/llvm-project/libcxx/include/thread:284
2023-08-14 02:14:45 #40 std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::{lambda()#2}> >(void*) (__vp=0x7020000d0060) at ./contrib/llvm-project/libcxx/include/thread:295
2023-08-14 02:14:45 #41 0x00007fb2d459cb43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
2023-08-14 02:14:45 #42 0x00007fb2d462ea00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
2023-08-14 02:15:12 Thread 1 (Thread 0x7fb2d441cd00 (LWP 33887) "clickhouse-serv"):
2023-08-14 02:15:12 #0 0x00007fb2d4599340 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
2023-08-14 02:15:12 #1 0x00007fb2d45a0082 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libc.so.6
2023-08-14 02:15:12 #2 0x000055b1f476e749 in std::__1::__libcpp_mutex_lock[abi:v15000](pthread_mutex_t*) (__m=0x7160002b1018) at ./contrib/llvm-project/libcxx/include/__threading_support:304
2023-08-14 02:15:12 #3 std::__1::mutex::lock (this=0x7160002b1018) at ./contrib/llvm-project/libcxx/src/mutex.cpp:38
2023-08-14 02:15:12 #4 0x000055b1dd7b9f72 in std::__1::lock_guard<std::__1::mutex>::lock_guard[abi:v15000](std::__1::mutex&) (this=0x7ffff5a1db90, __m=...) at ./contrib/llvm-project/libcxx/include/__mutex_base:94
2023-08-14 02:15:12 #5 DB::DistributedAsyncInsertDirectoryQueue::flushAllData (this=0x7160002b0e00) at ./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:171
2023-08-14 02:15:12 #6 0x000055b1dc7c1211 in DB::StorageDistributed::flushClusterNodesAllData (this=<optimized out>, local_context=...) at ./src/Storages/StorageDistributed.cpp:1464
2023-08-14 02:15:12 #7 0x000055b1dc7c0554 in DB::StorageDistributed::flushAndPrepareForShutdown (this=0x717000a81520) at ./src/Storages/StorageDistributed.cpp:1439
2023-08-14 02:15:12 #8 0x000055b1d7e937a3 in DB::DatabaseWithOwnTablesBase::shutdown (this=0x71300007e018) at ./src/Databases/DatabasesCommon.cpp:295
2023-08-14 02:15:12 #9 0x000055b1d874c14f in DB::DatabaseCatalog::shutdownImpl (this=0x71d000000a00) at ./src/Interpreters/DatabaseCatalog.cpp:266
2023-08-14 02:15:12 #10 0x000055b1d87633ef in DB::DatabaseCatalog::shutdown () at ./src/Interpreters/DatabaseCatalog.cpp:859
2023-08-14 02:15:12 #11 0x000055b1d85c60cf in DB::ContextSharedPart::shutdown (this=0x725000000000) at ./src/Interpreters/Context.cpp:501
2023-08-14 02:15:12 #12 0x000055b1d85a4b65 in DB::Context::shutdown (this=<optimized out>) at ./src/Interpreters/Context.cpp:3738
2023-08-14 02:15:12 #13 0x000055b1c1cf0b68 in DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_12::operator()() const (this=0x7ffff5a20f40) at ./programs/server/Server.cpp:1565
2023-08-14 02:15:12 #14 BasicScopeGuard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_12>::invoke() (this=0x7ffff5a20f40) at ./base/base/../base/scope_guard.h:99
2023-08-14 02:15:12 #15 BasicScopeGuard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_12>::~BasicScopeGuard() (this=this@entry=0x7ffff5a20f40) at ./base/base/../base/scope_guard.h:48
2023-08-14 02:15:12 #16 0x000055b1c1ccfe21 in DB::Server::main (this=<optimized out>) at ./programs/server/Server.cpp:1940
2023-08-14 02:15:12 #17 0x000055b1ea8d6fff in Poco::Util::Application::run (this=0x7ffff5a21298) at ./base/poco/Util/src/Application.cpp:315
2023-08-14 02:15:12 #18 0x000055b1c1ca0abd in DB::Server::run (this=0x7ffff5a21298) at ./programs/server/Server.cpp:395
2023-08-14 02:15:12 #19 0x000055b1ea921c60 in Poco::Util::ServerApplication::run (this=0x7ffff5a21298, argc=6, argv=0x703000021630) at ./base/poco/Util/src/ServerApplication.cpp:131
2023-08-14 02:15:12 #20 0x000055b1c1c9992a in mainEntryClickHouseServer (argc=6, argv=0x2) at ./programs/server/Server.cpp:200
2023-08-14 02:15:12 #21 0x000055b1aa4943c0 in main (argc_=<optimized out>, argv_=<optimized out>) at ./programs/main.cpp:492