"illegal rows_offset" exception in DM::findPack #2023
Description
Reproduce steps:
- 5 tiflash nodes, tiflash replica 2. Keep killing some TiFlash nodes. And use
pd-ctl scheduler add shuffle-region-schedulerto trigger frequent region rebalance. - 16 insert threads
- 16 query threads (select count(*) )
In very rare cases, queries throw the following exceptions.
[2021/05/30 07:55:08.759 +00:00] [ERROR] [<unknown>] ["DAGDriver: void DB::DAGDriver<batch>::execute() [with bool batch = true]: DB Exception: illegal rows_offset(265415), deletes_count(12)
\n0. /tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x3458655]
\n1. /tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x25) [0x344f1e5]
\n2. /tiflash/tiflash(DB::DM::findPack(std::vector<std::shared_ptr<DB::DM::DeltaValueSpace::Pack>, std::allocator<std::shared_ptr<DB::DM::DeltaValueSpace::Pack> > > const&, unsigned long, unsigned long)+0x2a8) [0x69e8fb8]
\n3. /tiflash/tiflash(DB::DM::DeltaValueSpace::Snapshot::getMergeBlocks(unsigned long, unsigned long, unsigned long, unsigned long)+0x60) [0x69ea020]
\n4. /tiflash/tiflash(DB::DM::Segment::ensurePlace(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueSpace::Snapshot>&, std::vector<DB::DM::Range<long>, std::allocator<DB::DM::Range<long> > > const&, unsigned long) const+0x418) [0x697a138]
\n5. /tiflash/tiflash(DB::DM::Segment::getReadInfo(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, std::vector<DB::DM::Range<long>, std::allocator<DB::DM::Range<long> > > const&, unsigned long) const+0x14a) [0x697b50a]
\n6. /tiflash/tiflash(DB::DM::Segment::getInputStream(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, std::vector<DB::DM::Range<long>, std::allocator<DB::DM::Range<long> > > const&, std::shared_ptr<DB::DM::RSOperator> const&, unsigned long, unsigned long)+0x153) [0x697bba3]
\n7. /tiflash/tiflash(DB::DM::DMSegmentThreadInputStream::readImpl(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x605) [0x6964a25]
\n8. /tiflash/tiflash(DB::DM::DMSegmentThreadInputStream::readImpl()+0x1a) [0x695897a]
\n9. /tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x3bc) [0x607ea0c]
\n10. /tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x607ebf6]
\n11. /tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1a) [0x6c5c4da]
\n12. /tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x3bc) [0x607ea0c]
\n13. /tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x607ebf6]
\n14. /tiflash/tiflash(DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler, (DB::StreamUnionMode)0>::loop(unsigned long)+0xea) [0x6f1714a]
\n15. /tiflash/tiflash(DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler, (DB::StreamUnionMode)0>::thread(MemoryTracker*, unsigned long)+0x20c) [0x6f1769c]
\n16. /tiflash/tiflash() [0x7e54cfe]
\n17. /lib64/libpthread.so.0(+0x7dd4) [0x7f66b1be5dd4]
\n18. /lib64/libc.so.6(clone+0x6c) [0x7f66b160ceac]
\n"] [thread_id=4804]