Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Use nightly version, deploy a multiple machine cluster(tiflash cfg:
logger.level: debug
profiles.default.task_scheduler_thread_hard_limit: 1000
profiles.default.task_scheduler_thread_soft_limit: 600), prepare tpch SF=10, tiflash replica = 2, parrelly starts five process running sequentially tpch sqls for 300 times.

2. What did you expect to see? (Required)

No crash happen.

3. What did you see instead (Required)

Random crash, one of the likely stack info:
[2022/05/22 22:22:01.861 +08:00] [ERROR] [BaseDaemon.cpp:570] ["BaseDaemon:
0x5c75b9 faultSignalHandler [tiflash+6059449]
libs/libdaemon/src/BaseDaemon.cpp:221
0x7fa3d3177630 [libpthread.so.0+63024]
0x511a7b3 grpc_impl::ClientAsyncReadermpp::MPPDataPacket::~ClientAsyncReader() [tiflash+85043123]
contrib/grpc/include/grpcpp/impl/codegen/async_stream_impl.h:197
0x511a855 grpc_impl::ClientAsyncReadermpp::MPPDataPacket::~ClientAsyncReader() [tiflash+85043285]
contrib/grpc/include/grpcpp/impl/codegen/async_stream_impl.h:197
0x511a881 std::default_delete<grpc_impl::ClientAsyncReadermpp::MPPDataPacket >::operator()(grpc_impl::ClientAsyncReadermpp::MPPDataPacket) const [tiflash+85043329]
/usr/local/include/c++/7.3.0/bits/unique_ptr.h:78
0x511bb5d std::unique_ptr<grpc_impl::ClientAsyncReadermpp::MPPDataPacket, std::default_delete<grpc_impl::ClientAsyncReadermpp::MPPDataPacket > >::reset(grpc_impl::ClientAsyncReadermpp::MPPDataPacket) [tiflash+85048157]
/usr/local/include/c++/7.3.0/bits/unique_ptr.h:376
0x511bd13 std::unique_ptr<grpc_impl::ClientAsyncReadermpp::MPPDataPacket, std::default_delete<grpc_impl::ClientAsyncReadermpp::MPPDataPacket > >::operator=(std::unique_ptr<grpc_impl::ClientAsyncReadermpp::MPPDataPacket, std::default_delete<grpc_impl::ClientAsyncReadermpp::MPPDataPacket > >&&) [tiflash+85048595]
/usr/local/include/c++/7.3.0/bits/unique_ptr.h:283
0x511937b init [tiflash+85037947]
dbms/src/Flash/Mpp/GRPCReceiverContext.cpp:102
0x5118ea9 DB::GRPCReceiverContext::makeAsyncReader(DB::ExchangeRecvRequest const&, std::shared_ptrDB::AsyncExchangePacketReader&, DB::UnaryCallback*) const [tiflash+85036713]
dbms/src/Flash/Mpp/GRPCReceiverContext.cpp:256
0x510ca66 start [tiflash+84986470]
dbms/src/Flash/Mpp/ExchangeReceiver.cpp:241
0x510ceb6 retrySucceed [tiflash+84987574]
dbms/src/Flash/Mpp/ExchangeReceiver.cpp:199
0x51136a6 DB::ExchangeReceiverBaseDB::GRPCReceiverContext::reactor(std::vector<DB::ExchangeRecvRequest, std::allocatorDB::ExchangeRecvRequest > const&) [tiflash+85014182]
dbms/src/Flash/Mpp/ExchangeReceiver.cpp:454
0x511376b DB::ExchangeReceiverBaseDB::GRPCReceiverContext::setUpConnection()::{lambda()#2}::operator()() const [tiflash+85014379]
dbms/src/Flash/Mpp/ExchangeReceiver.cpp:394
0x511377e std::_Function_handler<void (), DB::ExchangeReceiverBaseDB::GRPCReceiverContext::setUpConnection()::{lambda()#2}>::_M_invoke(std::_Any_data const&) [tiflash+85014398]

the other one of the less likely stack info:
[2022/05/23 20:21:13.219 +08:00] [ERROR] [BaseDaemon.cpp:570] ["BaseDaemon:
0x1ed24b1 faultSignalHandler(int, siginfo_t*, void*) [tiflash+32318641]
libs/libdaemon/src/BaseDaemon.cpp:221
0x7f0bc811a630 [libpthread.so.0+63024]
0x7f0bc7b5d387 raise [libc.so.6+222087]
0x7f0bc7b5ea78 abort [libc.so.6+227960]
0x86d6daa grpc_chttp2_stream::~grpc_chttp2_stream() [tiflash+141389226]
contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:0
0x85a249c grpc_combiner_continue_exec_ctx() [tiflash+140125340]
contrib/grpc/src/core/lib/iomgr/combiner.cc:236
0x85a4cfd grpc_core::ExecCtx::Flush() [tiflash+140135677]
contrib/grpc/src/core/lib/iomgr/exec_ctx.cc:156
0x85aab2b pollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+140159787]
contrib/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1136
0x85caba1 cq_next(grpc_completion_queue*, gpr_timespec, void*) [tiflash+140290977]
contrib/grpc/src/core/lib/surface/completion_queue.cc:1038
0x85590d0 grpc_impl::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) [tiflash+139825360]
contrib/grpc/src/cpp/common/completion_queue_cc.cc:48
0x1d5683e DB::handleRpcs(grpc_impl::ServerCompletionQueue*, Poco::Logger*) [tiflash+30763070]
dbms/src/Server/Server.cpp:528
0x1db157c auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+31135100]
dbms/src/Common/wrapInvocable.h:36
0x1db16e5 std::__1::packaged_task<void ()>::operator()() [tiflash+31135461]
/usr/local/bin/../include/c++/v1/future:2089
0x1da0cb8 DB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_deleteDB::IExecutableTask >&) [tiflash+31067320]
dbms/src/Common/DynamicThreadPool.cpp:101
0x1da0695 DB::DynamicThreadPool::fixedWork(unsigned long) [tiflash+31065749]
dbms/src/Common/DynamicThreadPool.cpp:115
0x1da1c3e auto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::)(unsigned long), DB::DynamicThreadPool, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, void (DB::DynamicThreadPool::&&)(unsigned long), DB::DynamicThreadPool&&, unsigned long&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, unsigned long>(auto&&...) const [tiflash+31071294]
dbms/src/Common/ThreadFactory.h:47

4. What is your TiFlash version? (Required)

| Release Version: v6.1.0-alpha
Edition: Community
Git Commit Hash: 388b3c79cc3a313aee59334ebe6fc7dc451769c3
Git Branch: heads/refs/tags/v6.1.0-alpha
UTC Build Time: 2022-05-22 15:03:09
GoVersion: go1.18.2
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false |