TiFlash crashes when multiple regions of the same table received concurrently #8217
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
2. What did you expect to see? (Required)
3. What did you see instead (Required)
tiflash crashes with error logging as below
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.568 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=439]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.570 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=278]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.580 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=279]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.581 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=280]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.583 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=546]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.585 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=547]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.587 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=318]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.588 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=328]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.589 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=621]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.591 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=672]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.592 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=673]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.594 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=356]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.595 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=374]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.597 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=429]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.598 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=384]
[2023-10-18T07:22:07.332Z] [2023/10/18 15:21:50.798 +08:00] [WARN] [InterpreterCreateQuery.cpp:584] ["createTable failed with error code is 337, error info is DB::Exception: Table db_2.t_420 is creating or attaching right now, stack_info is
0x41aa8de StackTrace::StackTrace() [tiflash+68856030]
0x4199182 DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
0xc72e3c6 DB::DDLGuard::DDLGuard(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208856006]
0xc73af23 std::__1::__unique_if<DB::DDLGuard>::__unique_single std::__1::make_unique<DB::DDLGuard, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208908067]
0xc72e542 DB::Context::getDDLGuard(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856386]
0xc72e6e4 DB::Context::getDDLGuardIfTableDoesntExist(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856804]
0xc9c6580 DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) [tiflash+211576192]
0xc9c7d37 DB::InterpreterCreateQuery::execute() [tiflash+211582263]
0xdcf52d0 DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyCreatePhysicalTable(std::__1::shared_ptr<TiDB::DBInfo> const&, std::__1::shared_ptr<TiDB::TableInfo> const&) [tiflash+231690960]
0xdcf31e5 DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231682533]
0xdd717d5 DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
0xc8551b8 DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
0xe7c706b DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
0xe6dd0c6 DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
0xe6dc9d2 DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
0xe6b17f1 PreHandleSnapshot [tiflash+241899505]
0x7efea91d2446 proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
0x7efea917105d engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
0x7efea44b7580 engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
0x7efea41bb3e2 _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
0x7efea4776d29 yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
0x7efeaa7219ee yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
0x7efeaa7227a4 _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
0x7efeaa71ce73 yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
0x7efeaa6e3f0d yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
0x7efeaa6d872c std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]
0x7efeaa70fd6d std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h584a80ae5b79b471 [libtiflash_proxy.so+166788461]
0x7efeaa6e7061 _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h11083f11ca183545 [libtiflash_proxy.so+166621281]
0x7efeaa70ff96 std::panicking::try::do_call::h8105d4df961e925b [libtiflash_proxy.so+166789014]
0x7efeaa72323b __rust_try [libtiflash_proxy.so+166867515]
0x7efeaa70febf std::panicking::try::h99fd52b9928d9a80 [libtiflash_proxy.so+166788799]
0x7efeaa70e371 std::panic::catch_unwind::h1258304a0f4464c1 [libtiflash_proxy.so+166781809]"] [source=InterpreterCreateQuery] [thread_id=439]
[2023-10-18T07:22:07.332Z] [2023/10/18 15:21:50.799 +08:00] [WARN] [InterpreterCreateQuery.cpp:584] ["createTable failed with error code is 337, error info is DB::Exception: Table db_2.t_420 is creating or attaching right now, stack_info is
0x41aa8de StackTrace::StackTrace() [tiflash+68856030]
0x4199182 DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
0xc72e3c6 DB::DDLGuard::DDLGuard(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208856006]
0xc73af23 std::__1::__unique_if<DB::DDLGuard>::__unique_single std::__1::make_unique<DB::DDLGuard, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208908067]
0xc72e542 DB::Context::getDDLGuard(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856386]
0xc72e6e4 DB::Context::getDDLGuardIfTableDoesntExist(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856804]
0xc9c6580 DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) [tiflash+211576192]
0xc9c7d37 DB::InterpreterCreateQuery::execute() [tiflash+211582263]
0xdcf52d0 DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyCreatePhysicalTable(std::__1::shared_ptr<TiDB::DBInfo> const&, std::__1::shared_ptr<TiDB::TableInfo> const&) [tiflash+231690960]
0xdcf31e5 DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231682533]
0xdd717d5 DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
0xc8551b8 DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
0xe7c706b DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
0xe6dd0c6 DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
0xe6dc9d2 DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
0xe6b17f1 PreHandleSnapshot [tiflash+241899505]
0x7efea91d2446 proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
0x7efea917105d engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
0x7efea44b7580 engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
0x7efea41bb3e2 _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
0x7efea4776d29 yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
0x7efeaa7219ee yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
0x7efeaa7227a4 _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
0x7efeaa71ce73 yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
0x7efeaa6e3f0d yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
0x7efeaa6d872c std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]
0x7efeaa70fd6d std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h584a80ae5b79b471 [libtiflash_proxy.so+166788461]
0x7efeaa6e7061 _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h11083f11ca183545 [libtiflash_proxy.so+166621281]
0x7efeaa70ff96 std::panicking::try::do_call::h8105d4df961e925b [libtiflash_proxy.so+166789014]
0x7efeaa72323b __rust_try [libtiflash_proxy.so+166867515]
0x7efeaa70febf std::panicking::try::h99fd52b9928d9a80 [libtiflash_proxy.so+166788799]
0x7efeaa70e371 std::panic::catch_unwind::h1258304a0f4464c1 [libtiflash_proxy.so+166781809]"] [source=InterpreterCreateQuery] [thread_id=278]
[2023-10-18T07:22:07.333Z] [2023/10/18 15:21:50.799 +08:00] [WARN] [InterpreterCreateQuery.cpp:584] ["createTable failed with error code is 337, error info is DB::Exception: Table db_2.t_420 is creating or attaching right now, stack_info is
0x41aa8de StackTrace::StackTrace() [tiflash+68856030]
0x4199182 DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
0xc72e3c6 DB::DDLGuard::DDLGuard(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208856006]
0xc73af23 std::__1::__unique_if<DB::DDLGuard>::__unique_single std::__1::make_unique<DB::DDLGuard, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208908067]
0xc72e542 DB::Context::getDDLGuard(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856386]
0xc72e6e4 DB::Context::getDDLGuardIfTableDoesntExist(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856804]
0xc9c6580 DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) [tiflash+211576192]
0xc9c7d37 DB::InterpreterCreateQuery::execute() [tiflash+211582263]
0xdcf52d0 DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyCreatePhysicalTable(std::__1::shared_ptr<TiDB::DBInfo> const&, std::__1::shared_ptr<TiDB::TableInfo> const&) [tiflash+231690960]
0xdcf31e5 DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231682533]
0xdd717d5 DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
0xc8551b8 DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
0xe7c706b DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
0xe6dd0c6 DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
0xe6dc9d2 DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
0xe6b17f1 PreHandleSnapshot [tiflash+241899505]
0x7efea91d2446 proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
0x7efea917105d engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
0x7efea44b7580 engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
0x7efea41bb3e2 _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
0x7efea4776d29 yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
0x7efeaa7219ee yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
0x7efeaa7227a4 _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
0x7efeaa71ce73 yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
0x7efeaa6e3f0d yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
0x7efeaa6d872c std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]
0x7efeaa70fd6d std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h584a80ae5b79b471 [libtiflash_proxy.so+166788461]
0x7efeaa6e7061 _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h11083f11ca183545 [libtiflash_proxy.so+166621281]
0x7efeaa70ff96 std::panicking::try::do_call::h8105d4df961e925b [libtiflash_proxy.so+166789014]
0x7efeaa72323b __rust_try [libtiflash_proxy.so+166867515]
0x7efeaa70febf std::panicking::try::h99fd52b9928d9a80 [libtiflash_proxy.so+166788799]
0x7efeaa70e371 std::panic::catch_unwind::h1258304a0f4464c1 [libtiflash_proxy.so+166781809]"] [source=InterpreterCreateQuery] [thread_id=279]
[2023-10-18T07:22:07.334Z] [2023/10/18 15:21:50.800 +08:00] [ERROR] [Exception.cpp:91] ["Code: 107, e.displayText() = DB::Exception: Cannot open file /tmp/tiflash/data/db/metadata/db_2/t_420.sql, errno: 2, strerror: No such file or directory: keyspace=4294967295 physical_table_id=-1: (while preHandleSnapshot region_id=604, index=7, term=6), e.what() = DB::Exception, Stack trace:
0x41aa8de StackTrace::StackTrace() [tiflash+68856030]
0x4199182 DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
0x41cd3ba DB::ErrnoException::ErrnoException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68998074]
0x41c921e DB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68981278]
0xc6e72dc DB::PosixRandomAccessFile::PosixRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, std::__1::shared_ptr<DB::ReadLimiter> const&, std::__1::shared_ptr<DB::FileSegment> const&) [tiflash+208564956]
0xc6dbb8b DB::PosixRandomAccessFile* std::__1::construct_at<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, DB::PosixRandomAccessFile*>(DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208518027]
0xc6db91b void std::__1::allocator_traits<std::__1::allocator<DB::PosixRandomAccessFile> >::construct<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void, void>(std::__1::allocator<DB::PosixRandomAccessFile>&, DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208517403]
0xc6db61e std::__1::__shared_ptr_emplace<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&>(std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516638]
0xc6db444 std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::allocate_shared<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::allocator<DB::PosixRandomAccessFile> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516164]
0xc6da9e7 std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::make_shared<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208513511]
0xc6d688f DB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::ReadLimiter> const&, int) const [tiflash+208496783]
0xc70342c DB::ReadBufferFromFileProvider::ReadBufferFromFileProvider(std::__1::shared_ptr<DB::FileProvider> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptr<DB::ReadLimiter> const&, int, char*, unsigned long) [tiflash+208679980]
0xc6b2be9 DB::DatabaseTiFlash::alterTable(DB::Context const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::ColumnsDescription const&, std::__1::function<void (DB::IAST&)> const&) [tiflash+208350185]
0xd4eca4f DB::updateDeltaMergeTableCreateStatement(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<DB::SortColumnDescription, std::__1::allocator<DB::SortColumnDescription> > const&, DB::ColumnsDescription const&, DB::OrderedNameSet const&, std::__1::optional<std::__1::reference_wrapper<TiDB::TableInfo const> >, unsigned long, DB::Context const&) [tiflash+223267407]
0xd4ed475 DB::StorageDeltaMerge::alterSchemaChange(std::__1::shared_ptr<DB::RWLock::LockHolderImpl> const&, TiDB::TableInfo&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context const&) [tiflash+223270005]
0xdcf3799 DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231683993]
0xdd717d5 DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
0xc8551b8 DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
0xe7c706b DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
0xe6dd0c6 DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
0xe6dc9d2 DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
0xe6b17f1 PreHandleSnapshot [tiflash+241899505]
0x7efea91d2446 proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
0x7efea917105d engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
0x7efea44b7580 engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
0x7efea41bb3e2 _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
0x7efea4776d29 yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
0x7efeaa7219ee yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
0x7efeaa7227a4 _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
0x7efeaa71ce73 yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
0x7efeaa6e3f0d yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
0x7efeaa6d872c std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=547]
[2023-10-18T07:22:07.334Z] [2023/10/18 15:21:50.800 +08:00] [ERROR] [Exception.cpp:91] ["Code: 107, e.displayText() = DB::Exception: Cannot open file /tmp/tiflash/data/db/metadata/db_2/t_420.sql, errno: 2, strerror: No such file or directory: keyspace=4294967295 physical_table_id=-1: (while preHandleSnapshot region_id=582, index=7, term=6), e.what() = DB::Exception, Stack trace:
0x41aa8de StackTrace::StackTrace() [tiflash+68856030]
0x4199182 DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
0x41cd3ba DB::ErrnoException::ErrnoException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68998074]
0x41c921e DB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68981278]
0xc6e72dc DB::PosixRandomAccessFile::PosixRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, std::__1::shared_ptr<DB::ReadLimiter> const&, std::__1::shared_ptr<DB::FileSegment> const&) [tiflash+208564956]
0xc6dbb8b DB::PosixRandomAccessFile* std::__1::construct_at<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, DB::PosixRandomAccessFile*>(DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208518027]
0xc6db91b void std::__1::allocator_traits<std::__1::allocator<DB::PosixRandomAccessFile> >::construct<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void, void>(std::__1::allocator<DB::PosixRandomAccessFile>&, DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208517403]
0xc6db61e std::__1::__shared_ptr_emplace<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&>(std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516638]
0xc6db444 std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::allocate_shared<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::allocator<DB::PosixRandomAccessFile> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516164]
0xc6da9e7 std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::make_shared<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208513511]
0xc6d688f DB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::ReadLimiter> const&, int) const [tiflash+208496783]
0xc70342c DB::ReadBufferFromFileProvider::ReadBufferFromFileProvider(std::__1::shared_ptr<DB::FileProvider> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptr<DB::ReadLimiter> const&, int, char*, unsigned long) [tiflash+208679980]
0xc6b2be9 DB::DatabaseTiFlash::alterTable(DB::Context const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::ColumnsDescription const&, std::__1::function<void (DB::IAST&)> const&) [tiflash+208350185]
0xd4eca4f DB::updateDeltaMergeTableCreateStatement(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<DB::SortColumnDescription, std::__1::allocator<DB::SortColumnDescription> > const&, DB::ColumnsDescription const&, DB::OrderedNameSet const&, std::__1::optional<std::__1::reference_wrapper<TiDB::TableInfo const> >, unsigned long, DB::Context const&) [tiflash+223267407]
0xd4ed475 DB::StorageDeltaMerge::alterSchemaChange(std::__1::shared_ptr<DB::RWLock::LockHolderImpl> const&, TiDB::TableInfo&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context const&) [tiflash+223270005]
0xdcf3799 DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231683993]
0xdd717d5 DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
0xc8551b8 DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
0xe7c706b DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
0xe6dd0c6 DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
0xe6dc9d2 DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
0xe6b17f1 PreHandleSnapshot [tiflash+241899505]
0x7efea91d2446 proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
0x7efea917105d engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
0x7efea44b7580 engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
0x7efea41bb3e2 _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
0x7efea4776d29 yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
0x7efeaa7219ee yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
0x7efeaa7227a4 _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
0x7efeaa71ce73 yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
0x7efeaa6e3f0d yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
0x7efeaa6d872c std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=328]
[2023-10-18T07:22:07.335Z] [2023/10/18 15:21:50.801 +08:00] [ERROR] [Exception.cpp:91] ["Code: 107, e.displayText() = DB::Exception: Cannot open file /tmp/tiflash/data/db/metadata/db_2/t_420.sql, errno: 2, strerror: No such file or directory: keyspace=4294967295 physical_table_id=-1: (while preHandleSnapshot region_id=602, index=7, term=6), e.what() = DB::Exception, Stack trace:
0x41aa8de StackTrace::StackTrace() [tiflash+68856030]
0x4199182 DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
0x41cd3ba DB::ErrnoException::ErrnoException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68998074]
0x41c921e DB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68981278]
0xc6e72dc DB::PosixRandomAccessFile::PosixRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, std::__1::shared_ptr<DB::ReadLimiter> const&, std::__1::shared_ptr<DB::FileSegment> const&) [tiflash+208564956]
0xc6dbb8b DB::PosixRandomAccessFile* std::__1::construct_at<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, DB::PosixRandomAccessFile*>(DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208518027]
0xc6db91b void std::__1::allocator_traits<std::__1::allocator<DB::PosixRandomAccessFile> >::construct<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void, void>(std::__1::allocator<DB::PosixRandomAccessFile>&, DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208517403]
0xc6db61e std::__1::__shared_ptr_emplace<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&>(std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516638]
0xc6db444 std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::allocate_shared<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::allocator<DB::PosixRandomAccessFile> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516164]
0xc6da9e7 std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::make_shared<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208513511]
0xc6d688f DB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::ReadLimiter> const&, int) const [tiflash+208496783]
0xc70342c DB::ReadBufferFromFileProvider::ReadBufferFromFileProvider(std::__1::shared_ptr<DB::FileProvider> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptr<DB::ReadLimiter> const&, int, char*, unsigned long) [tiflash+208679980]
0xc6b2be9 DB::DatabaseTiFlash::alterTable(DB::Context const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::ColumnsDescription const&, std::__1::function<void (DB::IAST&)> const&) [tiflash+208350185]
0xd4eca4f DB::updateDeltaMergeTableCreateStatement(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<DB::SortColumnDescription, std::__1::allocator<DB::SortColumnDescription> > const&, DB::ColumnsDescription const&, DB::OrderedNameSet const&, std::__1::optional<std::__1::reference_wrapper<TiDB::TableInfo const> >, unsigned long, DB::Context const&) [tiflash+223267407]
0xd4ed475 DB::StorageDeltaMerge::alterSchemaChange(std::__1::shared_ptr<DB::RWLock::LockHolderImpl> const&, TiDB::TableInfo&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context const&) [tiflash+223270005]
0xdcf3799 DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231683993]
0xdd717d5 DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
0xc8551b8 DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
0xe7c706b DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
0xe6dd0c6 DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
0xe6dc9d2 DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
0xe6b17f1 PreHandleSnapshot [tiflash+241899505]
0x7efea91d2446 proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
0x7efea917105d engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
0x7efea44b7580 engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
0x7efea41bb3e2 _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
0x7efea4776d29 yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
0x7efeaa7219ee yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
0x7efeaa7227a4 _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
0x7efeaa71ce73 yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
0x7efeaa6e3f0d yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
0x7efeaa6d872c std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=318]
[2023-10-18T07:22:07.336Z] [2023/10/18 15:21:50.808 +08:00] [ERROR] [BaseDaemon.cpp:370] [########################################] [source=BaseDaemon] [thread_id=762]