Skip to content

Segmentation fault when PreHandleSnapshot #1944

@lilinghai

Description

@lilinghai

Maybe the Minireproduce

TPCC 10k warehouses data and master tiflash cluster
scale-in tiflash
some ddl sql such as add index and add primary key
some ap sql

The tiflash error log

2021.05.20 21:58:32.274914 [ 176424 ] <Error> BaseDaemon: ########################################
2021.05.20 21:58:32.279488 [ 176424 ] <Error> BaseDaemon: (from thread 1560) Received signal Segmentation fault (11).
2021.05.20 21:58:32.286558 [ 176424 ] <Error> BaseDaemon: Address: 0x38
2021.05.20 21:58:32.286575 [ 176424 ] <Error> BaseDaemon: Access: read.
2021.05.20 21:58:32.286588 [ 176424 ] <Error> BaseDaemon: Address not mapped to object.
2021.05.20 21:58:32.324809 [ 176424 ] <Error> BaseDaemon: 0. bin/tiflash/tiflash(long Poco::Dynamic::Var::convert<long>() const+0x30) [0x73861f0]
2021.05.20 21:58:32.327358 [ 176424 ] <Error> BaseDaemon: 1. bin/tiflash/tiflash(TiDB::ColumnInfo::defaultValueToField() const+0x1d5) [0x7380c85]
2021.05.20 21:58:32.330020 [ 176424 ] <Error> BaseDaemon: 2. bin/tiflash/tiflash(bool DB::setColumnValues<(DB::TMTPKType)0>(DB::ColumnVector<unsigned char>&, DB::ColumnVector<unsigned long>&, std::vector<long, std::allocator<long> >&, std::vector<std::pair<long, unsigned long>, std::allocator<std::pair<long, unsigned long> > > const&, google::dense_hash_map<long, unsigned long, std::hash<long>, std::equal_to<long>, google::libc_allocator_with_realloc<std::pair<long const, unsigned long> > >&, DB::ColumnDataInfoMap&, std::vector<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject<false> const> >, std::allocator<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject<false> const> > > > const&, unsigned long, bool, TiDB::TableInfo const&, bool, std::shared_ptr<DB::RegionScanFilter>)+0x694) [0x7325c14]
2021.05.20 21:58:32.332418 [ 176424 ] <Error> BaseDaemon: 3. bin/tiflash/tiflash(DB::RegionBlockReader::read(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject<false> const> >, std::allocator<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject<false> const> > > >&, bool)+0xf23) [0x731d873]
2021.05.20 21:58:32.334734 [ 176424 ] <Error> BaseDaemon: 4. bin/tiflash/tiflash(DB::GenRegionBlockDatawithSchema(std::shared_ptr<DB::Region> const&, std::shared_ptr<DB::StorageDeltaMerge> const&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, unsigned long, bool, DB::TMTContext&)+0x596) [0x7306f26]
2021.05.20 21:58:32.336801 [ 176424 ] <Error> BaseDaemon: 5. bin/tiflash/tiflash(DB::DM::SSTFilesToBlockInputStream::readCommitedBlock()+0xa7) [0x781f8d7]
2021.05.20 21:58:32.341296 [ 176424 ] <Error> BaseDaemon: 6. bin/tiflash/tiflash(DB::DM::SSTFilesToBlockInputStream::read()+0x16a) [0x782024a]
2021.05.20 21:58:32.343210 [ 176424 ] <Error> BaseDaemon: 7. bin/tiflash/tiflash(DB::DM::PKSquashingBlockInputStream<true>::read()+0xe0) [0x7822220]
2021.05.20 21:58:32.345185 [ 176424 ] <Error> BaseDaemon: 8. bin/tiflash/tiflash(DB::DM::DMVersionFilterBlockInputStream<1>::initNextBlock()+0x31) [0x76ad071]
2021.05.20 21:58:32.345233 [ 176424 ] <Error> BaseDaemon: 9. bin/tiflash/tiflash(DB::DM::DMVersionFilterBlockInputStream<1>::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x1808) [0x76af3a8]
2021.05.20 21:58:32.347290 [ 176424 ] <Error> BaseDaemon: 10. bin/tiflash/tiflash(DB::DM::BoundedSSTFilesToBlockInputStream::read()+0x29) [0x781fab9]
2021.05.20 21:58:32.349704 [ 176424 ] <Error> BaseDaemon: 11. bin/tiflash/tiflash(DB::DM::SSTFilesToDTFilesOutputStream::write()+0x82) [0x7824f42]
2021.05.20 21:58:32.349763 [ 176424 ] <Error> BaseDaemon: 12. bin/tiflash/tiflash(DB::KVStore::preHandleSSTsToDTFiles(std::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&)+0x374) [0x76d8cc4]
2021.05.20 21:58:32.354741 [ 176424 ] <Error> BaseDaemon: 13. bin/tiflash/tiflash(DB::KVStore::preHandleSnapshotToFiles(std::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::TMTContext&)+0x46) [0x76d9416]
2021.05.20 21:58:32.360112 [ 176424 ] <Error> BaseDaemon: 14. bin/tiflash/tiflash(DB::PreHandleSnapshot(DB::EngineStoreServerWrap*, DB::BaseBuffView, unsigned long, DB::SSTViewVec, unsigned long, unsigned long)+0xf4) [0x7310f04]
2021.05.20 21:58:32.368360 [ 176424 ] <Error> BaseDaemon: 15. /data1/tidb-deploy-upgrade0/tiflash-9900/bin/tiflash/libtiflash_proxy.so(+0x9a02b4) [0x7fe552b6c2b4]
2021.05.20 21:58:32.370432 [ 176424 ] <Error> BaseDaemon: 16. /data1/tidb-deploy-upgrade0/tiflash-9900/bin/tiflash/libtiflash_proxy.so(+0x98aa1f) [0x7fe552b56a1f]
2021.05.20 21:58:32.374407 [ 176424 ] <Error> BaseDaemon: 17. /data1/tidb-deploy-upgrade0/tiflash-9900/bin/tiflash/libtiflash_proxy.so(+0x113c080) [0x7fe553308080]
2021.05.20 21:58:32.376398 [ 176424 ] <Error> BaseDaemon: 18. /data1/tidb-deploy-upgrade0/tiflash-9900/bin/tiflash/libtiflash_proxy.so(+0x1c06f36) [0x7fe553dd2f36]
2021.05.20 21:58:32.378312 [ 176424 ] <Error> BaseDaemon: 19. /data1/tidb-deploy-upgrade0/tiflash-9900/bin/tiflash/libtiflash_proxy.so(+0x1bfd968) [0x7fe553dc9968]
2021.05.20 21:58:32.380046 [ 176424 ] <Error> BaseDaemon: 20. /data1/tidb-deploy-upgrade0/tiflash-9900/bin/tiflash/libtiflash_proxy.so(+0x1bff0ab) [0x7fe553dcb0ab]
2021.05.20 21:58:32.381945 [ 176424 ] <Error> BaseDaemon: 21. /data1/tidb-deploy-upgrade0/tiflash-9900/bin/tiflash/libtiflash_proxy.so(+0x1435ecb) [0x7fe553601ecb]
2021.05.20 21:58:32.384078 [ 176424 ] <Error> BaseDaemon: 22. /lib64/libpthread.so.0(+0x7dd5) [0x7fe551991dd5]

call me if more detail log and core file needed

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions