Skip to content

TiKV panics: [region 15822] 15824 status Some((15824, true)) is not expected #17992

@overvenus

Description

@overvenus

Bug Report

/opt/tidb/kv.log:[2024/12/12 21:11:46.853 +00:00] [FATAL] [lib.rs:480] ["[region 15822] 15824 status Some((15824, true)) is not expected"] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}
             at /home/stn/tikv/components/tikv_util/src/lib.rs:479:18
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2029:9
      std::panicking::rust_panic_with_hook
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:783:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:657:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:171:18
   4: rust_begin_unwind
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:645:5
   5: core::panicking::panic_fmt
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:72:14
   6: raftstore::store::fsm::peer::PeerFsmDelegate<EK,ER,T>::check_snapshot
             at /home/stn/tikv/components/raftstore/src/store/fsm/peer.rs:3642:17
   7: raftstore::store::fsm::peer::PeerFsmDelegate<EK,ER,T>::on_raft_message
             at /home/stn/tikv/components/raftstore/src/store/fsm/peer.rs:2771:40
      raftstore::store::fsm::peer::PeerFsmDelegate<EK,ER,T>::handle_msgs
             at /home/stn/tikv/components/raftstore/src/store/fsm/peer.rs:653:37
   8: <raftstore::store::fsm::store::RaftPoller<EK,ER,T> as batch_system::batch::PollHandler<raftstore::store::fsm::peer::PeerFsm<EK,ER>,raftstore::store::fsm::store::StoreFsm<EK>>>::handle_normal
             at /home/stn/tikv/components/raftstore/src/store/fsm/store.rs:1098:9
      batch_system::batch::Poller<N,C,Handler>::poll
             at /home/stn/tikv/components/batch-system/src/batch.rs:458:27
   9: batch_system::batch::BatchSystem<N,C>::start_poller::{{closure}}
             at /home/stn/tikv/components/batch-system/src/batch.rs:589:17
      <std::thread::Builder as tikv_util::sys::thread::StdThreadBuildWrapper>::spawn_wrapper::{{closure}}
             at /home/stn/tikv/components/tikv_util/src/sys/thread.rs:438:13
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:155:18
  10: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:529:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9
      std::panicking::try::do_call
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:552:40
      std::panicking::try
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:516:19
      std::panic::catch_unwind
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:142:14
      std::thread::Builder::spawn_unchecked_::{{closure}}
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:528:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
  11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2015:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2015:9
      std::sys::unix::thread::Thread::new::thread_start
             at /home/stn/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:108:17
  12: start_thread
  13: clone
"] [location=/home/stn/tikv/components/raftstore/src/store/fsm/peer.rs:3642] [thread_name=raftstore-8-1] [thread_id=144]

https://tcms.pingcap.net/dashboard/executions/plan/7698101

What version of TiKV are you using?

#17948

Steps to reproduce

Run jepsen tests.

What did you expect?

No panic.

What did happened?

Panic.

Metadata

Metadata

Assignees

Labels

affects-6.1This bug affects the 6.1.x(LTS) versions.affects-6.5This bug affects the 6.5.x(LTS) versions.affects-7.1This bug affects the 7.1.x(LTS) versions.affects-7.5This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.impact/panicseverity/minortype/bugThe issue is confirmed as a bug.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions