Skip to content

[help] [SOLVED] 所有BE无法启动, 关闭auto_compaction后可以启动 #5192

New issue
New issue
Closed
Closed
[help] [SOLVED] 所有BE无法启动, 关闭auto_compaction后可以启动#5192
@zh0122

Description

@zh0122
zh0122
opened on Jan 5, 2021

问题描述:

  1. 一个表执行delete操作后, 所有BE挂掉, 无法启动, 检查log和源码后, 在be.conf中添加 disable_auto_compaction = true 后, 可以启动BE, 但是此方案只是临时方案, 求助如何在打开 auto compaction情况下, 恢复集群. 谢谢
  2. 打开auto compaction时, 启动BE后, 在30s左右就会挂掉, log如下, BE挂掉之后在top命令中可以看到有abrt-hook-ccpp进程占用大量资源.
  3. 已尝试在be.conf中, 设置有关compaction的参数, 目前发现设置 max_compaction_threads=1, min_compaction_threads=1 可以延长BE时间(启动后1min左右会挂掉, 如果默认值是10,启动不到30s会挂)

集群情况:

  1. 物理机10台, 36C72T, 384G ram
  2. FE 3个Followers, 7个Observer
  3. BE 17个BE, 3台Followers上1个BE, 7台Observer上2个BE

Log

be.out

start time: Tue Jan 5 09:49:16 CST 2021
*** Check failure stack trace: ***
    @          0x1b4ae3d  google::LogMessage::Fail()
    @          0x1b4ccc4  google::LogMessage::SendToLog()
    @          0x1b4a964  google::LogMessage::Flush()
    @          0x1b4d6f9  google::LogMessageFatal::~LogMessageFatal()
    @           0xf2eee4  doris::CondColumn::del_eval()
    @          0x168b024  doris::segment_v2::ColumnReader::_get_filtered_pages()
    @          0x168b698  doris::segment_v2::FileColumnIterator::get_row_ranges_by_zone_map()
    @          0x166159a  doris::segment_v2::SegmentIterator::_get_row_ranges_from_conditions()
    @          0x16628e8  doris::segment_v2::SegmentIterator::_get_row_ranges_by_column_conditions()
    @          0x1662e40  doris::segment_v2::SegmentIterator::_init()
    @          0x16636be  doris::segment_v2::SegmentIterator::next_batch()
    @           0xf803ec  doris::BetaRowsetReader::next_block()
    @           0xefd887  doris::CollectIterator::Level0Iterator::_refresh_current_row()
    @           0xefdbe2  doris::CollectIterator::add_child()
    @           0xedbfee  doris::Reader::_capture_rs_readers()
    @           0xee05ab  doris::Reader::init()
    @           0xecb6d0  doris::Merger::merge_rowsets()
    @           0xeba5f0  doris::Compaction::do_compaction_impl()
    @           0xebcd1c  doris::Compaction::do_compaction()
    @           0xeb16bc  doris::BaseCompaction::execute_compact_impl()
    @           0xeb8ed4  doris::Compaction::execute_compact()
    @           0xe418fb  doris::Tablet::execute_compaction()
    @           0xe1bb8f  _ZNSt17_Function_handlerIFvvEZN5doris13StorageEngine35_compaction_tasks_producer_callbackEvEUlvE0_E9_M_invokeERKSt9_Any_data
    @          0x11a48b2  doris::ThreadPool::dispatch_thread()
    @          0x119c718  doris::Thread::supervise_thread()
    @     0x7f8d24594dd5  start_thread
    @     0x7f8d248a6ead  __clone
start time: Tue Jan 5 09:50:55 CST 2021
*** Check failure stack trace: ***
    @          0x1b4ae3d  google::LogMessage::Fail()
    @          0x1b4ccc4  google::LogMessage::SendToLog()
    @          0x1b4a964  google::LogMessage::Flush()
    @          0x1b4d6f9  google::LogMessageFatal::~LogMessageFatal()
    @           0xf2eee4  doris::CondColumn::del_eval()
    @          0x168b024  doris::segment_v2::ColumnReader::_get_filtered_pages()
    @          0x168b698  doris::segment_v2::FileColumnIterator::get_row_ranges_by_zone_map()
    @          0x166159a  doris::segment_v2::SegmentIterator::_get_row_ranges_from_conditions()
    @          0x16628e8  doris::segment_v2::SegmentIterator::_get_row_ranges_by_column_conditions()
    @          0x1662e40  doris::segment_v2::SegmentIterator::_init()
    @          0x16636be  doris::segment_v2::SegmentIterator::next_batch()
    @           0xf803ec  doris::BetaRowsetReader::next_block()
    @           0xefd887  doris::CollectIterator::Level0Iterator::_refresh_current_row()
    @           0xefdbe2  doris::CollectIterator::add_child()
    @           0xedbfee  doris::Reader::_capture_rs_readers()
    @           0xee05ab  doris::Reader::init()
*** Check failure stack trace: ***
    @           0xecb6d0  doris::Merger::merge_rowsets()
    @          0x1b4ae3d  google::LogMessage::Fail()
    @           0xeba5f0  doris::Compaction::do_compaction_impl()
    @          0x1b4ccc4  google::LogMessage::SendToLog()
    @           0xebcd1c  doris::Compaction::do_compaction()
    @          0x1b4a964  google::LogMessage::Flush()
    @           0xeb16bc  doris::BaseCompaction::execute_compact_impl()
    @          0x1b4d6f9  google::LogMessageFatal::~LogMessageFatal()
    @           0xeb8ed4  doris::Compaction::execute_compact()
    @           0xf2eee4  doris::CondColumn::del_eval()
    @           0xe418fb  doris::Tablet::execute_compaction()
    @           0xe1bb8f  _ZNSt17_Function_handlerIFvvEZN5doris13StorageEngine35_compaction_tasks_producer_callbackEvEUlvE0_E9_M_invokeERKSt9_Any_data
    @          0x168b024  doris::segment_v2::ColumnReader::_get_filtered_pages()
    @          0x11a48b2  doris::ThreadPool::dispatch_thread()
    @          0x119c718  doris::Thread::supervise_thread()
    @          0x168b698  doris::segment_v2::FileColumnIterator::get_row_ranges_by_zone_map()
    @     0x7fcf145b7dd5  start_thread
    @     0x7fcf148c9ead  __clone
    @          0x166159a  doris::segment_v2::SegmentIterator::_get_row_ranges_from_conditions()
start time: Tue Jan 5 09:51:26 CST 2021
*** Check failure stack trace: ***
    @          0x1b4ae3d  google::LogMessage::Fail()
    @          0x1b4ccc4  google::LogMessage::SendToLog()
    @          0x1b4a964  google::LogMessage::Flush()
    @          0x1b4d6f9  google::LogMessageFatal::~LogMessageFatal()
    @           0xf2eee4  doris::CondColumn::del_eval()
    @          0x168b024  doris::segment_v2::ColumnReader::_get_filtered_pages()
    @          0x168b698  doris::segment_v2::FileColumnIterator::get_row_ranges_by_zone_map()
    @          0x166159a  doris::segment_v2::SegmentIterator::_get_row_ranges_from_conditions()
    @          0x16628e8  doris::segment_v2::SegmentIterator::_get_row_ranges_by_column_conditions()
    @          0x1662e40  doris::segment_v2::SegmentIterator::_init()
    @          0x16636be  doris::segment_v2::SegmentIterator::next_batch()
*** Check failure stack trace: ***
    @          0x1b4ae3d  google::LogMessage::Fail()
    @           0xf803ec  doris::BetaRowsetReader::next_block()
    @          0x1b4ccc4  google::LogMessage::SendToLog()
    @           0xefd887  doris::CollectIterator::Level0Iterator::_refresh_current_row()
    @           0xefdbe2  doris::CollectIterator::add_child()
    @          0x1b4a964  google::LogMessage::Flush()
    @           0xedbfee  doris::Reader::_capture_rs_readers()
    @          0x1b4d6f9  google::LogMessageFatal::~LogMessageFatal()
    @           0xf2eee4  doris::CondColumn::del_eval()
    @           0xee05ab  doris::Reader::init()
    @          0x168b024  doris::segment_v2::ColumnReader::_get_filtered_pages()
*** Check failure stack trace: ***
    @          0x168b698  doris::segment_v2::FileColumnIterator::get_row_ranges_by_zone_map()
    @           0xecb6d0  doris::Merger::merge_rowsets()
    @          0x1b4ae3d  google::LogMessage::Fail()
    @          0x166159a  doris::segment_v2::SegmentIterator::_get_row_ranges_from_conditions()
    @          0x1b4ccc4  google::LogMessage::SendToLog()
    @           0xeba5f0  doris::Compaction::do_compaction_impl()
    @          0x16628e8  doris::segment_v2::SegmentIterator::_get_row_ranges_by_column_conditions()
    @          0x1662e40  doris::segment_v2::SegmentIterator::_init()
    @          0x16636be  doris::segment_v2::SegmentIterator::next_batch()
    @          0x1b4a964  google::LogMessage::Flush()
    @           0xebcd1c  doris::Compaction::do_compaction()
    @           0xeb16bc  doris::BaseCompaction::execute_compact_impl()
    @          0x1b4d6f9  google::LogMessageFatal::~LogMessageFatal()
    @           0xf803ec  doris::BetaRowsetReader::next_block()
    @           0xf2eee4  doris::CondColumn::del_eval()
    @           0xeb8ed4  doris::Compaction::execute_compact()
    @           0xefd887  doris::CollectIterator::Level0Iterator::_refresh_current_row()
    @           0xe418fb  doris::Tablet::execute_compaction()
    @          0x168b024  doris::segment_v2::ColumnReader::_get_filtered_pages()
    @           0xe1bb8f  _ZNSt17_Function_handlerIFvvEZN5doris13StorageEngine35_compaction_tasks_producer_callbackEvEUlvE0_E9_M_invokeERKSt9_Any_data
    @           0xefdbe2  doris::CollectIterator::add_child()
    @          0x11a48b2  doris::ThreadPool::dispatch_thread()
    @           0xedbfee  doris::Reader::_capture_rs_readers()
    @          0x168b698  doris::segment_v2::FileColumnIterator::get_row_ranges_by_zone_map()
    @          0x119c718  doris::Thread::supervise_thread()
    @     0x7fc0b24ffdd5  start_thread
    @          0x166159a  doris::segment_v2::SegmentIterator::_get_row_ranges_from_conditions()
    @     0x7fc0b2811ead  __clone
start time: Tue Jan 5 09:53:34 CST 2021
*** Check failure stack trace: ***
    @          0x1b4ae3d  google::LogMessage::Fail()
    @          0x1b4ccc4  google::LogMessage::SendToLog()
    @          0x1b4a964  google::LogMessage::Flush()
    @          0x1b4d6f9  google::LogMessageFatal::~LogMessageFatal()
    @           0xf2eee4  doris::CondColumn::del_eval()
    @          0x168b024  doris::segment_v2::ColumnReader::_get_filtered_pages()
    @          0x168b698  doris::segment_v2::FileColumnIterator::get_row_ranges_by_zone_map()
    @          0x166159a  doris::segment_v2::SegmentIterator::_get_row_ranges_from_conditions()
    @          0x16628e8  doris::segment_v2::SegmentIterator::_get_row_ranges_by_column_conditions()
    @          0x1662e40  doris::segment_v2::SegmentIterator::_init()
    @          0x16636be  doris::segment_v2::SegmentIterator::next_batch()
    @           0xf803ec  doris::BetaRowsetReader::next_block()
    @           0xefd887  doris::CollectIterator::Level0Iterator::_refresh_current_row()
    @           0xefdbe2  doris::CollectIterator::add_child()
    @           0xedbfee  doris::Reader::_capture_rs_readers()
    @           0xee05ab  doris::Reader::init()
    @           0xecb6d0  doris::Merger::merge_rowsets()
    @           0xeba5f0  doris::Compaction::do_compaction_impl()
    @           0xebcd1c  doris::Compaction::do_compaction()
    @           0xeb16bc  doris::BaseCompaction::execute_compact_impl()
    @           0xeb8ed4  doris::Compaction::execute_compact()
    @           0xe418fb  doris::Tablet::execute_compaction()
    @           0xe1bb8f  _ZNSt17_Function_handlerIFvvEZN5doris13StorageEngine35_compaction_tasks_producer_callbackEvEUlvE0_E9_M_invokeERKSt9_Any_data
    @          0x11a48b2  doris::ThreadPool::dispatch_thread()
    @          0x119c718  doris::Thread::supervise_thread()
    @     0x7f7fcd4b7dd5  start_thread
    @     0x7f7fcd7c9ead  __clone

be.INFO

I0105 09:53:39.405167 69543 rowset.cpp:56] rowset is loaded. rowset version:10205-10205, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:532321
I0105 09:53:39.405351 69546 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=497176.1813299096.d64336c4961b1158-6f1cc9d32b60d58e
I0105 09:53:39.405769 69546 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_497176_1813299096 meta_size=67470
I0105 09:53:39.405850 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=413290.1246779604.eb49ad0176e5a613-3086e175ac962bb6
I0105 09:53:39.405898 69545 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=401263.1985520577.cb470b5d0c07e8b6-0b24532bbd849b89
I0105 09:53:39.406111 69555 data_dir.cpp:949] scan data dir path:/u10/data/doris/doris-storage finished. path size:36869
I0105 09:53:39.406183 69556 data_dir.cpp:806] start to path gc by tablet schemahash.
I0105 09:53:39.406301 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_413290_1246779604 meta_size=62709
I0105 09:53:39.406324 69545 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_401263_1985520577 meta_size=63276
I0105 09:53:39.406853 69546 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=498532.1813299096.ee47f3749d84ae8e-05072fe021e611a8
I0105 09:53:39.407063 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=413610.1246779604.a54fd1760d67085c-37fd0103d28908bb
I0105 09:53:39.407332 69546 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_498532_1813299096 meta_size=68979
I0105 09:53:39.407644 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_413610_1246779604 meta_size=62709
I0105 09:53:39.407650 69545 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=497928.1813299096.8d4d83c39838d3d6-a455f092d8b370a5
I0105 09:53:39.408219 69545 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_497928_1813299096 meta_size=67157
I0105 09:53:39.408408 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=412730.1246779604.a547330e99bc7244-2a88899a6e483ebc
I0105 09:53:39.408520 69546 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=498656.1813299096.6d44810886d0786a-ba25adf3fb2fbfbb
I0105 09:53:39.408821 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_412730_1246779604 meta_size=62709
I0105 09:53:39.408860 69556 data_dir.cpp:845] finished one time path gc by tablet.
I0105 09:53:39.408866 69556 olap_server.cpp:262] try to perform path gc by rowsetid!
I0105 09:53:39.408869 69556 data_dir.cpp:856] start to path gc by rowsetid.
I0105 09:53:39.408980 69546 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_498656_1813299096 meta_size=71277
I0105 09:53:39.409356 69545 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=412714.1246779604.7740970a82610b93-d21734f41c2172b7
I0105 09:53:39.409545 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=497832.1813299096.734cb02e0e8e7fb0-58de2d9ffa89f5a7
I0105 09:53:39.409736 69545 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_412714_1246779604 meta_size=62708
I0105 09:53:39.409988 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_497832_1813299096 meta_size=71079
I0105 09:53:39.409991 69546 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=226848.1862174012.5349ee23c1d61516-7daa53b3e80ecb88
I0105 09:53:39.410017 69546 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_226848_1862174012 meta_size=1290
I0105 09:53:39.410068 69546 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=206952.2008120100.484b8e88d0259e22-0885cb8a975a0eb7
I0105 09:53:39.410099 69546 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_206952_2008120100 meta_size=1184
I0105 09:53:39.410166 69546 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=207340.543050668.0348661b44e61bf6-f4c797510c16369f
I0105 09:53:39.410190 69546 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_207340_543050668 meta_size=1632
I0105 09:53:39.410223 69546 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=401127.1985520577.62487043b425f510-956a8c672548fcab
I0105 09:53:39.410619 69546 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_401127_1985520577 meta_size=64410
I0105 09:53:39.410813 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=497636.1813299096.5a4104c785a66d29-ca73f7ed010128a5
I0105 09:53:39.410854 69545 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=497012.1813299096.f44494b8e5ada0b7-78379012312e978d
I0105 09:53:39.411278 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_497636_1813299096 meta_size=72780
I0105 09:53:39.411283 69545 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_497012_1813299096 meta_size=67523
I0105 09:53:39.411691 69546 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=379763.1618909583.6648de57f5ba2ddb-52e3bb08e6927090
I0105 09:53:39.411854 69546 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_379763_1618909583 meta_size=13051
I0105 09:53:39.411943 69546 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=400607.1985520577.6644bfe4a967793f-ac931f02f4654aa0
I0105 09:53:39.411996 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=413658.1246779604.1a4d79e9f860ddab-40b7e617e140f19d
I0105 09:53:39.412395 69546 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_400607_1985520577 meta_size=64158
I0105 09:53:39.412451 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_413658_1246779604 meta_size=62707
I0105 09:53:39.412456 69545 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=497148.1813299096.45475f6107280a12-01370bad911d738b
I0105 09:53:39.412979 69545 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_497148_1813299096 meta_size=73932
I0105 09:53:39.413149 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=413654.1246779604.054f5338718b7804-16aaf679d6ff8fb2
I0105 09:53:39.413427 69546 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=223623.1244624220.a54a8a823778aee7-73c81092493a828f
I0105 09:53:39.413453 69546 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_223623_1244624220 meta_size=897
I0105 09:53:39.413487 69546 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=225788.1862174012.784acd268969915c-af2c59eb3c312998
I0105 09:53:39.413511 69546 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_225788_1862174012 meta_size=1291
I0105 09:53:39.413547 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_413654_1246779604 meta_size=62707
I0105 09:53:39.414052 69545 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=497172.1813299096.3d410c801b65078d-a21805c0ca981ea7
I0105 09:53:39.414258 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=119528.1584148754.1349126ec183f1cd-b9421f79117ae09a
I0105 09:53:39.414311 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_119528_1584148754 meta_size=4497
I0105 09:53:39.414361 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=497688.1813299096.394afa93ae043c03-1e972f78d40843a4
I0105 09:53:39.414434 69545 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_497172_1813299096 meta_size=62061
I0105 09:53:39.414778 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_497688_1813299096 meta_size=68689
I0105 09:53:39.415477 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=401283.1985520577.2942fc68107f2a0c-1bba1d9c32ace7b0
I0105 09:53:39.415694 69545 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=497836.1813299096.984529c001c8f429-6c7cd692fffd228f
I0105 09:53:39.415851 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_401283_1985520577 meta_size=64156
I0105 09:53:39.416131 69545 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_497836_1813299096 meta_size=69820
I0105 09:53:39.416682 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=497540.1813299096.d64310ab42b50830-d375881a97a9c4ad
I0105 09:53:39.417078 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_497540_1813299096 meta_size=67254
I0105 09:53:39.417141 69545 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=497984.1813299096.30434d80472dcbd6-bc7e777d6a1e57a5
I0105 09:53:39.417515 69545 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_497984_1813299096 meta_size=64739
I0105 09:53:39.417891 69547 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=487774.1411653075.504e2535f51c11f2-a66fdff9365827ab
I0105 09:53:39.417917 69547 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_487774_1411653075 meta_size=1630
I0105 09:53:39.418465 69545 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=498068.1813299096.154e62e220cfa84d-9068fdcff18fa39b
I0105 09:53:39.418900 69545 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_498068_1813299096 meta_size=72169
I0105 09:53:39.419875 69545 tablet.cpp:1175] start to do tablet meta checkpoint, tablet=498408.1813299096.124f33154da2704d-fbc4094010019781
I0105 09:53:39.420244 69545 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_498408_1813299096 meta_size=66719
I0105 09:53:39.661070 69552 data_dir.cpp:957] collect garbage dir path: /u03/data/doris/doris-storage/data/452/532321/406360882/02000000000000010c49230dcc10edce9f05c934494d1aa1_0.dat
I0105 09:53:39.661162 69552 data_dir.cpp:957] collect garbage dir path: /u03/data/doris/doris-storage/data/452/532321/406360882/0200000000000007da4a74b01117d87c1e3a5f528c9aaf92_0.dat
I0105 09:53:39.843997 69554 data_dir.cpp:889] finished one time path gc by rowsetid.
I0105 09:53:39.872802 69558 data_dir.cpp:889] finished one time path gc by rowsetid.
I0105 09:53:39.876755 69560 data_dir.cpp:889] finished one time path gc by rowsetid.
I0105 09:53:39.915439 69556 data_dir.cpp:957] collect garbage dir path: /u10/data/doris/doris-storage/data/652/219271/359247826/0200000000000005da4a74b01117d87c1e3a5f528c9aaf92_0.dat
I0105 09:53:39.928979 69552 data_dir.cpp:889] finished one time path gc by rowsetid.
I0105 09:53:40.037642 69556 data_dir.cpp:889] finished one time path gc by rowsetid.
I0105 09:53:40.305531 69595 task_worker_pool.cpp:1101] waiting to receive first heartbeat from frontend
I0105 09:53:40.305598 69596 task_worker_pool.cpp:1159] waiting to receive first heartbeat from frontend
I0105 09:53:41.155467 69768 heartbeat_server.cpp:58] get heartbeat from FE.host:10.181.83.14, port:9020, cluster id:2064593746, counter:1
I0105 09:53:41.155484 69768 heartbeat_server.cpp:120] master change. new master host: 10.181.83.14. port: 9020. epoch: 13
I0105 09:53:41.155488 69768 heartbeat_server.cpp:144] get token. token: 9e7d9584-6e72-4641-8768-28eea6c1aaf2
I0105 09:53:41.155489 69768 heartbeat_server.cpp:166] Master FE is changed or restarted. report tablet and disk info immediately
I0105 09:53:41.155494 69768 task_worker_pool.cpp:238] notify task worker pool: TaskWorkerPool.REPORT_OLAP_TABLE
I0105 09:53:41.155498 69768 task_worker_pool.cpp:238] notify task worker pool: TaskWorkerPool.REPORT_DISK_STATE
I0105 09:53:43.702555 69769 task_worker_pool.cpp:217] submitting task. type=CLEAR_TRANSACTION_TASK, signature=3710023
I0105 09:53:43.702579 69769 task_worker_pool.cpp:229] success to submit task. type=CLEAR_TRANSACTION_TASK, signature=3710023, task_count_in_queue=1
I0105 09:53:43.702587 69769 task_worker_pool.cpp:217] submitting task. type=CLEAR_TRANSACTION_TASK, signature=3710024
I0105 09:53:43.702589 69769 task_worker_pool.cpp:229] success to submit task. type=CLEAR_TRANSACTION_TASK, signature=3710024, task_count_in_queue=2
I0105 09:53:43.702591 69769 task_worker_pool.cpp:217] submitting task. type=CLEAR_TRANSACTION_TASK, signature=3710025
I0105 09:53:43.702620 69769 task_worker_pool.cpp:229] success to submit task. type=CLEAR_TRANSACTION_TASK, signature=3710025, task_count_in_queue=2
I0105 09:53:43.702625 69582 task_worker_pool.cpp:735] get clear transaction task task, signature:3710023, transaction_id: 3710023, partition id size: 0
I0105 09:53:43.702653 69582 storage_engine.cpp:549] begin to clear transaction task. transaction_id=3710023
I0105 09:53:43.702656 69582 storage_engine.cpp:573] finish to clear transaction task. transaction_id=3710023
I0105 09:53:43.702657 69582 task_worker_pool.cpp:755] finish to clear transaction task. signature:3710023, transaction_id: 3710023
I0105 09:53:43.703117 69582 task_worker_pool.cpp:272] finish task success.
I0105 09:53:43.703130 69582 task_worker_pool.cpp:258] remove task info. type=CLEAR_TRANSACTION_TASK, signature=3710023, queue_size=2
I0105 09:53:43.703143 69582 task_worker_pool.cpp:735] get clear transaction task task, signature:3710024, transaction_id: 3710024, partition id size: 0
I0105 09:53:43.703145 69582 storage_engine.cpp:549] begin to clear transaction task. transaction_id=3710024
I0105 09:53:43.703166 69582 storage_engine.cpp:573] finish to clear transaction task. transaction_id=3710024
I0105 09:53:43.703166 69582 task_worker_pool.cpp:755] finish to clear transaction task. signature:3710024, transaction_id: 3710024
I0105 09:53:43.703227 69582 task_worker_pool.cpp:272] finish task success.
I0105 09:53:43.703228 69582 task_worker_pool.cpp:258] remove task info. type=CLEAR_TRANSACTION_TASK, signature=3710024, queue_size=1
I0105 09:53:43.703233 69582 task_worker_pool.cpp:735] get clear transaction task task, signature:3710025, transaction_id: 3710025, partition id size: 0
I0105 09:53:43.703235 69582 storage_engine.cpp:549] begin to clear transaction task. transaction_id=3710025
I0105 09:53:43.703236 69582 storage_engine.cpp:573] finish to clear transaction task. transaction_id=3710025
I0105 09:53:43.703238 69582 task_worker_pool.cpp:755] finish to clear transaction task. signature:3710025, transaction_id: 3710025
I0105 09:53:43.703266 69582 task_worker_pool.cpp:272] finish task success.
I0105 09:53:43.703269 69582 task_worker_pool.cpp:258] remove task info. type=CLEAR_TRANSACTION_TASK, signature=3710025, queue_size=0
I0105 09:53:44.197468 69543 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_532321_406360882 meta_size=4114
I0105 09:53:44.197548 69543 compaction.cpp:134] succeed to do cumulative compaction. tablet=532321.406360882.8246ad011ddd5b45-86622476af84c2b1, output_version=2-10205, segments=6. elapsed time=4.83251s.
I0105 09:53:44.197559 69543 cumulative_compaction.cpp:86] after cumulative compaction, current cumulative point is 2, tablet=532321.406360882.8246ad011ddd5b45-86622476af84c2b1
W0105 09:53:44.197562 69543 tablet.cpp:1391] Trace:
0105 09:53:39.365043 (+     0us) tablet.cpp:1394] execute cumulative compaction
0105 09:53:39.365044 (+     1us) cumulative_compaction.cpp:66] got cumulative compaction lock
0105 09:53:39.365046 (+     2us) compaction.cpp:55] start to do compaction
0105 09:53:39.365137 (+    91us) compaction.cpp:88] prepare finished
0105 09:53:44.197142 (+4832005us) compaction.cpp:102] merge rowsets finished
0105 09:53:44.197323 (+   181us) compaction.cpp:116] output rowset built
0105 09:53:44.197323 (+     0us) compaction.cpp:120] check correctness finished
0105 09:53:44.197548 (+   225us) compaction.cpp:124] modify rowsets finished
0105 09:53:44.197559 (+    11us) cumulative_compaction.cpp:78] compaction finished
0105 09:53:44.197562 (+     3us) cumulative_compaction.cpp:92] save cumulative compaction metrics
Metrics: {"filtered_rows":0,"input_row_num":368246,"input_rowsets_data_size":57372466,"input_segments_num":6,"merge_rowsets_latency_us":4832001,"merged_rows":0,"output_row_num":368246,"output_rowset_data_size":57260781,"output_segments_num":1}
I0105 09:53:44.197726 69543 compaction.cpp:82] start cumulative compaction. tablet=201166.271481667.49405623b8a79fa2-9b589d9050393483, output_version=2-2813, permits: 8
I0105 09:53:44.197860 69543 rowset.cpp:56] rowset is loaded. rowset version:2-2806, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201166
I0105 09:53:44.199677 69543 rowset.cpp:56] rowset is loaded. rowset version:2807-2807, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201166
I0105 09:53:44.199821 69543 rowset.cpp:56] rowset is loaded. rowset version:2808-2808, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201166
I0105 09:53:44.199968 69543 rowset.cpp:56] rowset is loaded. rowset version:2809-2809, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201166
I0105 09:53:44.200122 69543 rowset.cpp:56] rowset is loaded. rowset version:2810-2810, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201166
I0105 09:53:44.200258 69543 rowset.cpp:56] rowset is loaded. rowset version:2811-2811, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201166
I0105 09:53:44.200417 69543 rowset.cpp:56] rowset is loaded. rowset version:2812-2812, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201166
I0105 09:53:44.200562 69543 rowset.cpp:56] rowset is loaded. rowset version:2813-2813, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201166
I0105 09:53:44.211175 69543 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_201166_271481667 meta_size=4029
I0105 09:53:44.211226 69543 compaction.cpp:134] succeed to do cumulative compaction. tablet=201166.271481667.49405623b8a79fa2-9b589d9050393483, output_version=2-2813, segments=1. elapsed time=0.013503s.
I0105 09:53:44.211233 69543 cumulative_compaction.cpp:86] after cumulative compaction, current cumulative point is 2, tablet=201166.271481667.49405623b8a79fa2-9b589d9050393483
I0105 09:53:44.225239 69543 compaction.cpp:82] start cumulative compaction. tablet=205952.207736122.e849f669f0bcd663-c183ecee263221a1, output_version=2-6098, permits: 8
I0105 09:53:44.225313 69543 rowset.cpp:56] rowset is loaded. rowset version:2-6091, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:205952
I0105 09:53:44.225847 69543 rowset.cpp:56] rowset is loaded. rowset version:6092-6092, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:205952
I0105 09:53:44.225934 69543 rowset.cpp:56] rowset is loaded. rowset version:6093-6093, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:205952
I0105 09:53:44.226007 69543 rowset.cpp:56] rowset is loaded. rowset version:6094-6094, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:205952
I0105 09:53:44.226075 69543 rowset.cpp:56] rowset is loaded. rowset version:6095-6095, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:205952
I0105 09:53:44.226145 69543 rowset.cpp:56] rowset is loaded. rowset version:6096-6096, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:205952
I0105 09:53:44.226228 69543 rowset.cpp:56] rowset is loaded. rowset version:6097-6097, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:205952
I0105 09:53:44.226297 69543 rowset.cpp:56] rowset is loaded. rowset version:6098-6098, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:205952
I0105 09:53:44.230173 69543 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_205952_207736122 meta_size=2417
I0105 09:53:44.230208 69543 compaction.cpp:134] succeed to do cumulative compaction. tablet=205952.207736122.e849f669f0bcd663-c183ecee263221a1, output_version=2-6098, segments=1. elapsed time=0.004972s.
I0105 09:53:44.230214 69543 cumulative_compaction.cpp:86] after cumulative compaction, current cumulative point is 2, tablet=205952.207736122.e849f669f0bcd663-c183ecee263221a1
I0105 09:53:44.240698 69543 compaction.cpp:82] start cumulative compaction. tablet=221090.673598560.bd4def9e4329e97f-93e616d9bc4b7e86, output_version=2-88317, permits: 7
I0105 09:53:44.240751 69543 rowset.cpp:56] rowset is loaded. rowset version:2-88311, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:221090
I0105 09:53:44.240900 69543 rowset.cpp:56] rowset is loaded. rowset version:88312-88312, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:221090
I0105 09:53:44.241040 69543 rowset.cpp:56] rowset is loaded. rowset version:88313-88313, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:221090
I0105 09:53:44.241185 69543 rowset.cpp:56] rowset is loaded. rowset version:88314-88314, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:221090
I0105 09:53:44.241307 69543 rowset.cpp:56] rowset is loaded. rowset version:88315-88315, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:221090
I0105 09:53:44.241446 69543 rowset.cpp:56] rowset is loaded. rowset version:88316-88316, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:221090
I0105 09:53:44.241569 69543 rowset.cpp:56] rowset is loaded. rowset version:88317-88317, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:221090
I0105 09:53:44.243818 69543 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_221090_673598560 meta_size=1724
I0105 09:53:44.243844 69543 compaction.cpp:134] succeed to do cumulative compaction. tablet=221090.673598560.bd4def9e4329e97f-93e616d9bc4b7e86, output_version=2-88317, segments=7. elapsed time=0.003149s.
I0105 09:53:44.243850 69543 cumulative_compaction.cpp:86] after cumulative compaction, current cumulative point is 2, tablet=221090.673598560.bd4def9e4329e97f-93e616d9bc4b7e86
I0105 09:53:44.250806 69543 compaction.cpp:82] start cumulative compaction. tablet=531765.406360882.a14a18097cca12f2-81ba6b27b79d76a2, output_version=2-10205, permits: 6
I0105 09:53:44.251129 69543 rowset.cpp:56] rowset is loaded. rowset version:2-10200, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:531765
I0105 09:53:44.279781 69543 rowset.cpp:56] rowset is loaded. rowset version:10201-10201, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:531765
I0105 09:53:44.281635 69543 rowset.cpp:56] rowset is loaded. rowset version:10202-10202, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:531765
I0105 09:53:44.283253 69543 rowset.cpp:56] rowset is loaded. rowset version:10203-10203, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:531765
I0105 09:53:44.284830 69543 rowset.cpp:56] rowset is loaded. rowset version:10204-10204, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:531765
I0105 09:53:44.286429 69543 rowset.cpp:56] rowset is loaded. rowset version:10205-10205, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:531765
I0105 09:53:49.109668 69543 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_531765_406360882 meta_size=4114
I0105 09:53:49.109784 69543 compaction.cpp:134] succeed to do cumulative compaction. tablet=531765.406360882.a14a18097cca12f2-81ba6b27b79d76a2, output_version=2-10205, segments=6. elapsed time=4.85898s.
I0105 09:53:49.109798 69543 cumulative_compaction.cpp:86] after cumulative compaction, current cumulative point is 2, tablet=531765.406360882.a14a18097cca12f2-81ba6b27b79d76a2
W0105 09:53:49.109803 69543 tablet.cpp:1391] Trace:
0105 09:53:44.250804 (+     0us) tablet.cpp:1394] execute cumulative compaction
0105 09:53:44.250805 (+     1us) cumulative_compaction.cpp:66] got cumulative compaction lock
0105 09:53:44.250806 (+     1us) compaction.cpp:55] start to do compaction
0105 09:53:44.250817 (+    11us) compaction.cpp:88] prepare finished
0105 09:53:49.109379 (+4858562us) compaction.cpp:102] merge rowsets finished
0105 09:53:49.109527 (+   148us) compaction.cpp:116] output rowset built
0105 09:53:49.109527 (+     0us) compaction.cpp:120] check correctness finished
0105 09:53:49.109784 (+   257us) compaction.cpp:124] modify rowsets finished
0105 09:53:49.109797 (+    13us) cumulative_compaction.cpp:78] compaction finished
0105 09:53:49.109801 (+     4us) cumulative_compaction.cpp:92] save cumulative compaction metrics
Metrics: {"filtered_rows":0,"input_row_num":372105,"input_rowsets_data_size":57925983,"input_segments_num":6,"merge_rowsets_latency_us":4858558,"merged_rows":0,"output_row_num":372105,"output_rowset_data_size":57817860,"output_segments_num":1}
I0105 09:53:49.109994 69543 compaction.cpp:82] start cumulative compaction. tablet=201038.271481667.a04fe8080e24954e-189c5e8eb416f48d, output_version=2-2813, permits: 5
I0105 09:53:49.110150 69543 rowset.cpp:56] rowset is loaded. rowset version:2-2809, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201038
I0105 09:53:49.111943 69543 rowset.cpp:56] rowset is loaded. rowset version:2810-2810, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201038
I0105 09:53:49.112251 69543 rowset.cpp:56] rowset is loaded. rowset version:2811-2811, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201038
I0105 09:53:49.112407 69543 rowset.cpp:56] rowset is loaded. rowset version:2812-2812, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201038
I0105 09:53:49.112555 69543 rowset.cpp:56] rowset is loaded. rowset version:2813-2813, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:201038
I0105 09:53:49.122969 69543 tablet_meta_manager.cpp:113] save tablet meta , key:tabletmeta_201038_271481667 meta_size=3668
I0105 09:53:49.123032 69543 compaction.cpp:134] succeed to do cumulative compaction. tablet=201038.271481667.a04fe8080e24954e-189c5e8eb416f48d, output_version=2-2813, segments=2. elapsed time=0.013043s.
I0105 09:53:49.123040 69543 cumulative_compaction.cpp:86] after cumulative compaction, current cumulative point is 2, tablet=201038.271481667.a04fe8080e24954e-189c5e8eb416f48d
I0105 09:53:49.130970 69544 base_compaction.cpp:129] satisfy the base compaction policy. tablet=542766.1745240348.2a4ad68d91e84af8-82d8b1bb4d863a8f, cumulative_total_size=26280073, base_size=1, cumulative_base_ratio=2.62801e+07, policy_ratio=0.3
I0105 09:53:49.131008 69543 compaction.cpp:82] start base compaction. tablet=542766.1745240348.2a4ad68d91e84af8-82d8b1bb4d863a8f, output_version=0-40816, permits: 3
I0105 09:53:49.131007 69544 base_compaction.cpp:129] satisfy the base compaction policy. tablet=542598.1745240348.bf4ae8d1df40a320-ebb4b962d0c3bf86, cumulative_total_size=26268286, base_size=1, cumulative_base_ratio=2.62683e+07, policy_ratio=0.3
I0105 09:53:49.131022 69544 base_compaction.cpp:129] satisfy the base compaction policy. tablet=542410.1745240348.ef49a0d2906b3f6c-dbec2f15b22e9592, cumulative_total_size=26297284, base_size=1, cumulative_base_ratio=2.62973e+07, policy_ratio=0.3
I0105 09:53:49.131156 69543 rowset.cpp:56] rowset is loaded. rowset version:0-1, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:542766
I0105 09:53:49.131454 69543 rowset.cpp:56] rowset is loaded. rowset version:2-40815, state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:542766
F0105 09:53:49.131732 69543 olap_cond.cpp:400] Check failed: false It will not happen when the stat's min is not null and max is null

be.WARNING

W0105 09:53:39.209828 69369 data_dir.cpp:734] could not find tablet id: 27888, schema hash: 1584148754, for rowset: 0200000000c46a26184b4c258241bb864b5ef9c4adf08d8e, skip this rowset
W0105 09:53:39.209833 69369 data_dir.cpp:734] could not find tablet id: 27888, schema hash: 1584148754, for rowset: 0200000000c4730b184b4c258241bb864b5ef9c4adf08d8e, skip this rowset
...重复...
W0105 09:53:39.210197 69369 data_dir.cpp:734] could not find tablet id: 89333, schema hash: 1584148754, for rowset: 02000000055ef6e1184b4c258241bb864b5ef9c4adf08d8e, skip this rowset
W0105 09:53:39.210203 69369 data_dir.cpp:734] could not find tablet id: 89333, schema hash: 1584148754, for rowset: 02000000055f2091184b4c258241bb864b5ef9c4adf08d8e, skip this rowset
W0105 09:53:39.210208 69369 data_dir.cpp:734] could not find tablet id: 89333, schema hash: 1584148754, for rowset: 02000000055f20c6184b4c258241bb864b5ef9c4adf08d8e, skip this rowset
W0105 09:53:39.210213 69369 data_dir.cpp:734] could not find tablet id: 89333, schema hash: 1584148754, for rowset: 02000000055f8183184b4c258241bb864b5ef9c4adf08d8e, skip this rowset
W0105 09:53:39.210219 69369 data_dir.cpp:734] could not find tablet id: 89333, schema hash: 1584148754, for rowset: 02000000055f88a6184b4c258241bb864b5ef9c4adf08d8e, skip this rowset
W0105 09:53:39.210233 69369 data_dir.cpp:778] find invalid rowset: 0200000000a3b92a184b4c258241bb864b5ef9c4adf08d8e with tablet id: 14236 tablet uid: ff49813604adef0f-70d1d0c1917af4a4 schema hash: 1584148754 txn: 293764 current valid tablet uid: 024d1cfc2f31f8b1-4fb8de06faaa7d87
W0105 09:53:39.210240 69369 data_dir.cpp:778] find invalid rowset: 0200000000a3bbe5184b4c258241bb864b5ef9c4adf08d8e with tablet id: 14236 tablet uid: ff49813604adef0f-70d1d0c1917af4a4 schema hash: 1584148754 txn: 293775 current valid tablet uid: 024d1cfc2f31f8b1-4fb8de06faaa7d87
W0105 09:53:39.210247 69369 data_dir.cpp:778] find invalid rowset: 0200000000a3c28e184b4c258241bb864b5ef9c4adf08d8e with tablet id: 14236 tablet uid: ff49813604adef0f-70d1d0c1917af4a4 
...重复...
schema hash: 1584148754 txn: 294084 current valid tablet uid: 024d1cfc2f31f8b1-4fb8de06faaa7d87
W0105 09:53:39.210320 69369 data_dir.cpp:778] find invalid rowset: 0200000000a4156c184b4c258241bb864b5ef9c4adf08d8e with tablet id: 14236 tablet uid: ff49813604adef0f-70d1d0c1917af4a4 schema hash: 1584148754 txn: 294128 current valid tablet uid: 024d1cfc2f31f8b1-4fb8de06faaa7d87
W0105 09:53:39.210326 69369 data_dir.cpp:778] find invalid rowset: 0200000000a41883184b4c258241bb864b5ef9c4adf08d8e with tablet id: 14236 tablet uid: ff49813604adef0f-70d1d0c1917af4a4 schema hash: 1584148754 txn: 294143 current valid tablet uid: 024d1cfc2f31f8b1-4fb8de06faaa7d87
W0105 09:53:39.210332 69369 data_dir.cpp:778] find invalid rowset: 0200000000a41c68184b4c258241bb864b5ef9c4adf08d8e with tablet id: 14236 tablet uid: ff49813604adef0f-70d1d0c1917af4a4 schema hash: 1584148754 txn: 294151 current valid tablet uid: 024d1cfc2f31f8b1-4fb8de06faaa7d87
W0105 09:53:39.210338 69369 data_dir.cpp:778] find invalid rowset: 0200000000a425ee184b4c258241bb864b5ef9c4adf08d8e with tablet id: 14236 tablet uid: ff49813604adef0f-70d1d0c1917af4a4 schema hash: 1584148754 txn: 294286 current valid tablet uid: 024d1cfc2f31f8b1-4fb8de06faaa7d87
W0105 09:53:39.210345 69369 data_dir.cpp:734] could not find tablet id: 14668, schema hash: 1584148754, for rowset: 02000000000016ec664611cbaa56397e367f4242d87f1abb, skip this rowset
W0105 09:53:39.210355 69369 data_dir.cpp:734] could not find tablet id: 14668, schema hash: 1584148754, for rowset: 02000000000020e8664611cbaa56397e367f4242d87f1abb, skip this rowset
W0105 09:53:39.210361 69369 data_dir.cpp:734] could not find tablet id: 14668, schema hash: 1584148754, for rowset: 020000000000287f664611cbaa56397e367f4242d87f1abb, skip this rowset
W0105 09:53:39.210367 69369 data_dir.cpp:734] could not find tablet id: 14668, schema hash: 1584148754, for rowset: 0200000000002a31664611cbaa56397e367f4242d87f1abb, skip this rowset
...重复...
W0105 09:53:39.210799 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 0200000000452753c4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210806 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 0200000000452827c4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210813 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 0200000000452f09c4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210819 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 020000000045308cc4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210825 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 0200000000453556c4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210834 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 02000000004536d2c4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210841 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 0200000000453a98c4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210847 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 0200000000453c15c4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210855 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 0200000000453db6c4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210860 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 0200000000453fc1c4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210866 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 020000000045460ac4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210873 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 0200000000454ae5c4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.210880 69369 data_dir.cpp:734] could not find tablet id: 21848, schema hash: 1584148754, for rowset: 0200000000454c83c4454c14005204cdf43c32fd9135bb91, skip this rowset
W0105 09:53:39.300596 69095 cgroups_mgr.cpp:329] Could not find a valid cgroups path for resource isolation,current value is . ignore it.
W0105 09:53:39.308466 69594 utils.cpp:90] fail to get master client from cache. host=, port=0, code=7
W0105 09:53:39.312693 69594 task_worker_pool.cpp:1078] report task failed. status: -1, master host: port: 0
W0105 09:53:44.197562 69543 tablet.cpp:1391] Trace:
0105 09:53:39.365043 (+     0us) tablet.cpp:1394] execute cumulative compaction
0105 09:53:39.365044 (+     1us) cumulative_compaction.cpp:66] got cumulative compaction lock
0105 09:53:39.365046 (+     2us) compaction.cpp:55] start to do compaction
0105 09:53:39.365137 (+    91us) compaction.cpp:88] prepare finished
0105 09:53:44.197142 (+4832005us) compaction.cpp:102] merge rowsets finished
0105 09:53:44.197323 (+   181us) compaction.cpp:116] output rowset built
0105 09:53:44.197323 (+     0us) compaction.cpp:120] check correctness finished
0105 09:53:44.197548 (+   225us) compaction.cpp:124] modify rowsets finished
0105 09:53:44.197559 (+    11us) cumulative_compaction.cpp:78] compaction finished
0105 09:53:44.197562 (+     3us) cumulative_compaction.cpp:92] save cumulative compaction metrics
Metrics: {"filtered_rows":0,"input_row_num":368246,"input_rowsets_data_size":57372466,"input_segments_num":6,"merge_rowsets_latency_us":4832001,"merged_rows":0,"output_row_num":368246,"output_rowset_data_size":57260781,"output_segments_num":1}
W0105 09:53:49.109803 69543 tablet.cpp:1391] Trace:
0105 09:53:44.250804 (+     0us) tablet.cpp:1394] execute cumulative compaction
0105 09:53:44.250805 (+     1us) cumulative_compaction.cpp:66] got cumulative compaction lock
0105 09:53:44.250806 (+     1us) compaction.cpp:55] start to do compaction
0105 09:53:44.250817 (+    11us) compaction.cpp:88] prepare finished
0105 09:53:49.109379 (+4858562us) compaction.cpp:102] merge rowsets finished
0105 09:53:49.109527 (+   148us) compaction.cpp:116] output rowset built
0105 09:53:49.109527 (+     0us) compaction.cpp:120] check correctness finished
0105 09:53:49.109784 (+   257us) compaction.cpp:124] modify rowsets finished
0105 09:53:49.109797 (+    13us) cumulative_compaction.cpp:78] compaction finished
0105 09:53:49.109801 (+     4us) cumulative_compaction.cpp:92] save cumulative compaction metrics
Metrics: {"filtered_rows":0,"input_row_num":372105,"input_rowsets_data_size":57925983,"input_segments_num":6,"merge_rowsets_latency_us":4858558,"merged_rows":0,"output_row_num":372105,"output_rowset_data_size":57817860,"output_segments_num":1}
F0105 09:53:49.131732 69543 olap_cond.cpp:400] Check failed: false It will not happen when the stat's min is not null and max is null

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions