merge #2
Merged
merge #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The TabletQuorumFailedException will be thrown in commitTxn while the success replica num of tablet is less then quorom replica num. The Hadoop load does not handle this exception because the push task will retry it later. The streaming broker, insert, stream and mini load will catch this exception and abort the txn after that.
xy720
pushed a commit
that referenced
this pull request
Jul 13, 2022
* [Schema Change] support fast add/drop column (apache#49) * [feature](schema-change) support fast schema change. coauthor: yixiutt * [schema change] Using columns desc from fe to read data. coauthor: Lchangliang * [feature](schema change) schema change optimize for add/drop columns. 1.add uniqueId field for class column. 2.schema change for add/drop columns directly update schema meta Co-authored-by: yixiutt <yixiu@selectdb.com> Co-authored-by: SWJTU-ZhangLei <1091517373@qq.com> [Feature](schema change) fix write and add regression test (apache#69) Co-authored-by: yixiutt <yixiu@selectdb.com> [schema change] be ssupport that delete use newest schema add delete regression test fix regression case (apache#107) tmp [feature](schema change) light schema change exclude rollup and agg/uniq/dup key type. [feature](schema change) fe olapTable maxUniqueId write in disk. [feature](schema change) add rpc iface for sc add column. [feature](schema change) add columnsDesc to TPushReq for ligtht sc. resolve the deadlock when schema change (apache#124) fix columns from fe don't has bitmap_index flag (apache#134) add update/delete case construct MATERIALIZED schema from origin schema when insert fix not vectorized compaction coredump use segment cache choose newest schema by schema version when compaction (apache#182) [bugfix](schema change) fix ligth schema change problem. [feature](schema change) light schema change add alter job. (#1) fix be ut [bug] (schema change) unique drop key column should not light schema change [feature](schema change) add schema change regression-test. fix regression test [bugfix](schema change) fix multi alter clauses for light schema change. (#2) [bugfix](schema change) fix multi clauses calculate column unique id (#3) modify PushTask process (apache#217) [Bugfix](schema change) fix jobId replay cause bdbje exception. [bug](schema change) fix max col unique id repeatitive. (apache#232) [optimize](schema change) modify pendingMaxColUniqueId generate rule. fix compaction error * fix be ut * fix snapshot load core fix unique_id error (apache#278) [refact](fe) remove redundant code for light schema change. (#4) [refact](fe) remove redundant code for light schema change. (#4) format fe core format be core fix be ut modify fe meta version fix rebase error flush schema into rowset_meta in old table [refactor](schema change) refact fe light schema change. (#5) delete the change of schemahash and support get max version schema * modify for review * fix be ut * fix schema change test
xy720
pushed a commit
that referenced
this pull request
Mar 23, 2023
… order by keys (apache#17913) if the order by keys are not simple slot in sort node, the order by exprs have to been added to sort node's output tuple. In that case, we need add a project node above sort node to eliminate the unused order by exprs. for example: ```sql WITH t0 AS (SELECT DATE_FORMAT(date, '%Y%m%d') AS date FROM cir_1756_t1 ), t3 AS (SELECT date_format(date, '%Y%m%d') AS `date` FROM `cir_1756_t2` GROUP BY date_format(date, '%Y%m%d') **ORDER BY date_format(date, '%Y%m%d')** ) SELECT t0.date FROM t0 LEFT JOIN t3 ON t0.date = t3.date; ``` before: ``` +--------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +--------------------------------------------------------------------------------------------------------------------------------------------------+ | LogicalProject[159] ( distinct=false, projects=[date#1], excepts=[], canEliminate=true ) | | +--LogicalJoin[158] ( type=LEFT_OUTER_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(date#1 = date#3)], otherJoinConjuncts=[] ) | | |--LogicalProject[151] ( distinct=false, projects=[date_format(date#0, '%Y%m%d') AS `date`#1], excepts=[], canEliminate=true ) | | | +--LogicalOlapScan ( qualified=default_cluster:bugfix.cir_1756_t1, indexName=cir_1756_t1, selectedIndexId=412339, preAgg=ON ) | | +--LogicalSort[157] ( orderKeys=[date_format(cast(date#3 as DATETIME), '%Y%m%d') asc null first] ) | | +--LogicalAggregate[156] ( groupByExpr=[date#3], outputExpr=[date#3], hasRepeat=false ) | | +--LogicalProject[155] ( distinct=false, projects=[date_format(date#2, '%Y%m%d') AS `date`#3], excepts=[], canEliminate=true ) | | +--LogicalOlapScan ( qualified=default_cluster:bugfix.cir_1756_t2, indexName=cir_1756_t2, selectedIndexId=412352, preAgg=ON ) | +--------------------------------------------------------------------------------------------------------------------------------------------------+ ``` after: ``` +--------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +--------------------------------------------------------------------------------------------------------------------------------------------------+ | LogicalProject[171] ( distinct=false, projects=[date#2], excepts=[], canEliminate=true ) | | +--LogicalJoin[170] ( type=LEFT_OUTER_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(date#2 = date#4)], otherJoinConjuncts=[] ) | | |--LogicalProject[162] ( distinct=false, projects=[date_format(date#0, '%Y%m%d') AS `date`#2], excepts=[], canEliminate=true ) | | | +--LogicalOlapScan ( qualified=default_cluster:bugfix.cir_1756_t1, indexName=cir_1756_t1, selectedIndexId=1049812, preAgg=ON ) | | +--LogicalProject[169] ( distinct=false, projects=[date#4], excepts=[], canEliminate=false ) | | +--LogicalSort[168] ( orderKeys=[date_format(cast(date#4 as DATETIME), '%Y%m%d') asc null first] ) | | +--LogicalAggregate[167] ( groupByExpr=[date#4], outputExpr=[date#4], hasRepeat=false ) | | +--LogicalProject[166] ( distinct=false, projects=[date_format(date#3, '%Y%m%d') AS `date`#4], excepts=[], canEliminate=true ) | | +--LogicalOlapScan ( qualified=default_cluster:bugfix.cir_1756_t2, indexName=cir_1756_t2, selectedIndexId=1049825, preAgg=ON ) | +--------------------------------------------------------------------------------------------------------------------------------------------------+ ```
xy720
pushed a commit
that referenced
this pull request
Jun 1, 2023
…ugh sort (apache#20161) Support the rewrite rule for push-down filter through sort. We can directly push-down the filter through sort without any conditions check. Before this PR: ``` mysql> explain select * from (select * from t1 order by a) t2 where t2.b > 2; +-------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#2] | | b[#3] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 3:VSELECT | | | predicates: b[#3] > 2 | | | | | 2:VMERGING-EXCHANGE | | offset: 0 | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: a[#0] | | | | STREAM DATA SINK | | EXCHANGE ID: 02 | | UNPARTITIONED | | | | 1:VTOP-N | | | order by: a[#2] ASC | | | offset: 0 | | | | | 0:VOlapScanNode | | TABLE: default_cluster:test.t1(t1), PREAGGREGATION: ON | | partitions=0/1, tablets=0/0, tabletList= | | cardinality=1, avgRowSize=0.0, numNodes=1 | +-------------------------------------------------------------+ 30 rows in set (0.06 sec) ``` After this PR: ``` mysql> explain select * from (select * from t1 order by a) t2 where t2.b > 2; +-------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#2] | | b[#3] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 2:VMERGING-EXCHANGE | | offset: 0 | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: a[#0] | | | | STREAM DATA SINK | | EXCHANGE ID: 02 | | UNPARTITIONED | | | | 1:VTOP-N | | | order by: a[#2] ASC | | | offset: 0 | | | | | 0:VOlapScanNode | | TABLE: default_cluster:test.t1(t1), PREAGGREGATION: ON | | PREDICATES: b[#1] > 2 | | partitions=0/1, tablets=0/0, tabletList= | | cardinality=1, avgRowSize=0.0, numNodes=1 | +-------------------------------------------------------------+ 28 rows in set (0.40 sec) ```
xy720
pushed a commit
that referenced
this pull request
Jun 30, 2023
…scan node _conjuncts (apache#20534) =3073084==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60601897db80 at pc 0x55b2c993666e bp 0x7d1fbbfb66b0 sp 0x7d1fbbfb66a8 READ of size 8 at 0x60601897db80 thread T610 (_scanner_scan) #0 0x55b2c993666d in std::__shared_ptr<doris::vectorized::VExprContext, (__gnu_cxx::_Lock_policy)2>::get() const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291:16 #1 0x55b2dae86ec5 in doris::vectorized::VExprContext::clone(doris::RuntimeState*, std::shared_ptr<doris::vectorized::VExprContext>&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exprs/vexpr_context.cpp:98:5 #2 0x55b2e757b6d8 in doris::vectorized::VScanner::prepare(doris::RuntimeState*, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>> const&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/vscanner.cpp:47:13 #3 0x55b2e78e8155 in doris::vectorized::NewOlapScanner::init() /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/new_olap_scanner.cpp:109:5 #4 0x55b2e7551c81 in doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::VScanner>) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/scanner_scheduler.cpp:279:27 #5 0x55b2e7554d5e in doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()::operator()() const /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/scanner_scheduler.cpp:202:31 #6 0x55b2e7554c14 in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #7 0x55b2e7554bb4 in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>(doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #8 0x55b2e7554a1c in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()>::_M_invoke(std::_Any_data const&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #9 0x55b2c80f2cd2 in std::function<void ()>::operator()() const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 #10 0x55b2e755f3e4 in doris::PriorityWorkStealingThreadPool::work_thread(int) /mnt/disk2/tengjianping/doris-master/be/src/util/priority_work_stealing_thread_pool.hpp:135:17 apache#11 0x55b2e7563c72 in void std::__invoke_impl<void, void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>(std::__invoke_memfun_deref, void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 apache#12 0x55b2e7563b44 in std::__invoke_result<void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>::type std::__invoke<void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>(void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 apache#13 0x55b2e7563b14 in decltype(std::__invoke((*this)._M_pmf, std::forward<doris::PriorityWorkStealingThreadPool*&>(fp), std::forward<int&>(fp))) std::_Mem_fn_base<void (doris::PriorityWorkStealingThreadPool::*)(int), true>::operator()<doris::PriorityWorkStealingThreadPool*&, int&>(doris::PriorityWorkStealingThreadPool*&, int&) const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:131:11 apache#14 0x55b2e7563ae4 in void std::__invoke_impl<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>(std::__invoke_other, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#15 0x55b2e7563a54 in std::enable_if<is_invocable_r_v<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>, void>::type std::__invoke_r<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>(std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#16 0x55b2e75639c3 in void std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>::__call<void, 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:570:11 apache#17 0x55b2e756382d in void std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>::operator()<>() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:629:17 apache#18 0x55b2e7563744 in void std::__invoke_impl<void, std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>(std::__invoke_other, std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>&&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#19 0x55b2e7563704 in std::__invoke_result<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>::type std::__invoke<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>(std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>&&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 apache#20 0x55b2e75636dc in void std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:253:13 apache#21 0x55b2e75636b4 in std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>::operator()() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:260:11 apache#22 0x55b2e7563638 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>>::_M_run() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 apache#23 0x55b2eb41d0ef in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18 apache#24 0x7f1dfd4e1179 in start_thread pthread_create.c apache#25 0x7f1dfdd7bdf2 in clone (/lib64/libc.so.6+0xfcdf2) (BuildId: 20ee73ce1b6ac38a52440bab82ec7e28f0f5c5b9)
xy720
pushed a commit
that referenced
this pull request
Jun 30, 2023
…1350) For sql like select cast("627492340" as datetime); the string is an invalid datetime, function DateV2Value<T>::from_date_str cast it as datetime 2062-74-92 23:40:00, with an out-of-range month and day value, which cause memory violation in function DateV2Value<T>::format_datetime when trying to access s_days_in_month. ==256444==ERROR: AddressSanitizer: global-buffer-overflow on address 0x55a7c1a5cff8 at pc 0x55a7e5aa3d2a bp 0x7f3b805f0370 sp 0x7f3b805f0368 READ of size 4 at 0x55a7c1a5cff8 thread T390 (FragmentMgrThre) #0 0x55a7e5aa3d29 in doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>::format_datetime(unsigned int*, bool*) const /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdatetime_value.cpp:1821:31 #1 0x55a7e5aa3052 in doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>::from_date_str(char const*, int, int) /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdatetime_value.cpp:1968:5 #2 0x55a7d48f0c49 in bool doris::vectorized::read_datetime_v2_text_impl<unsigned long>(unsigned long&, doris::vectorized::ReadBuffer&, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/vec/io/io_helper.h:309:19 #3 0x55a7ddb21642 in bool doris::vectorized::try_read_datetime_v2_text<unsigned long>(unsigned long&, doris::vectorized::ReadBuffer&, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/vec/io/io_helper.h:409:12 #4 0x55a7ddb215ec in bool doris::vectorized::try_parse_impl<doris::vectorized::DataTypeDateTimeV2, unsigned int, void*>(doris::vectorized::DataTypeDateTimeV2::FieldType&, doris::vectorized::ReadBuffer&, DateLUTImpl const*, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:839:16 #5 0x55a7ddb21c84 in auto doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto)::operator()<std::integral_constant<bool, false>, std::integral_constant<bool, true>>(void*, auto) const /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:1340:38 #6 0x55a7ddb216f7 in void* std::__invoke_impl<doris::Status, doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::integral_constant<bool, false>, std::integral_constant<bool, true>>(std::__invoke_other, auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, true>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #7 0x55a7ddb2167f in std::__invoke_result<void*, std::integral_constant<bool, false>, std::integral_constant<bool, true>>::type std::__invoke<doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::integral_constant<bool, false>, std::integral_constant<bool, true>>(void*&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, true>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #8 0x55a7ddb20d14 in std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<doris::Status> (*)(doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&)>, std::integer_sequence<unsigned long, 0ul, 1ul>>::__visit_invoke(doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013:11 #9 0x55a7ddb20c15 in decltype(auto) std::__do_visit<std::__detail::__variant::__deduce_visit_result<doris::Status>, doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1714:14 #10 0x55a7ddb20b6a in decltype(auto) std::visit<doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(void*&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1769:9 apache#11 0x55a7ddb205ff in doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*) /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:1321:23 apache#12 0x55a7ddb1f2c7 in doris::vectorized::FunctionConvertFromString<doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:1417:20
xy720
pushed a commit
that referenced
this pull request
Aug 17, 2023
…tion (apache#21317) explain WITH cte_0 AS ( SELECT 1 AS a ) SELECT * from cte_0 t1 join cte_0 t2 on true WHERE false; before: ``` +----------------------------+ | Explain String | +----------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#1] | | a[#2] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 1:VEMPTYSET | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | a[#0] | | PARTITION: UNPARTITIONED | | | | MultiCastDataSinks | | | | 0:VUNION | | constant exprs: | | 1 | +----------------------------+ ``` after: ``` +----------------------------+ | Explain String | +----------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#0] | | a[#1] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 0:VEMPTYSET | +----------------------------+ ```
xy720
pushed a commit
that referenced
this pull request
Aug 17, 2023
fix: apache#21136 mem tracker group uses class static variables instead of global variables https://stackoverflow.com/questions/2204608/does-c-call-destructors-for-global-and-class-static-variables TODO: A mem tracker manager is required, don't use global variables, it will sad ==3623982==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f0000056b8 at pc 0x56478bbe3ae0 bp 0x7f20953d2270 sp 0x7f20953d2268 READ of size 8 at 0x60f0000056b8 thread T41 (memory_tracker_) *** Query id: 0-0 *** *** Aborted at 1689749969 (unix time) try "date -d @1689749969" if you are using GNU date *** *** Current BE git commitID: b3e9cad *** *** SIGSEGV address not mapped to object (@0x0) received by PID 3623982 (TID 3624277 OR 0x7f19e06dd640) from PID 0; stack trace: *** #0 0x56478bbe3adf in std::__shared_ptr::operator bool() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1295:16 #1 0x56478bbe306e in doris::MemTracker::refresh_profile_counter() /doris/be/src/runtime/memory/mem_tracker.h:149:13 #2 0x56478bbec669 in doris::MemTrackerLimiter::refresh_all_tracker_profile() /doris/be/src/runtime/memory/mem_tracker_limiter.cpp:119:22 #3 0x564788f53fa0 in doris::Daemon::memory_tracker_profile_refresh_thread() /doris/be/src/common/daemon.cpp:295:9 #4 0x564788f5d04b in doris::Daemon::start()::$_4::operator()() const /doris/be/src/common/daemon.cpp:473:30 #5 0x564788f5cff6 in void std::__invoke_impl(std::__invoke_other, doris::Daemon::start()::$_4&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #6 0x564788f5cf78 in std::enable_if, void>::type std::__invoke_r(doris::Daemon::start()::$_4&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #7 0x564788f5cdae in std::_Function_handler::_M_invoke(std::_Any_data const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #8 0x56478903f576 in std::function::operator()() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 #9 0x56478c4a35af in doris::Thread::supervise_thread(void*) /doris/be/src/util/thread.cpp:465:5 #10 0x7f217c8a244f in start_thread nptl/pthread_create.c:473:8 apache#11 0x7f217cb27d52 in __clone misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 0x60f0000056b8 is located 56 bytes inside of 168-byte region [0x60f000005680,0x60f000005728) freed by thread T0 here: #0 0x564788e7280d in operator delete(void*) (/mnt/hdd01/dorisTestEnv/NEREIDS_ASAN/be/lib/doris_be+0x1758280d) (BuildId: 219493cc924323ee) #1 0x56478acec1d5 in std::default_delete::operator()(doris::MemTrackerLimiter*) const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 #2 0x56478ace9faf in std::unique_ptr >::~unique_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 #3 0x56478ace1471 in doris::ShardedLRUCache::~ShardedLRUCache() /doris/be/src/olap/lru_cache.cpp:581:1 #4 0x56478ace14c8 in doris::ShardedLRUCache::~ShardedLRUCache() /doris/be/src/olap/lru_cache.cpp:572:37 #5 0x56478acd0984 in std::default_delete::operator()(doris::Cache*) const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 #6 0x56478acceddf in std::unique_ptr >::~unique_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 #7 0x56478ad96dc6 in doris::StoragePageCache::~StoragePageCache() /doris/be/src/olap/page_cache.h:78:7 #8 0x7f217ca54146 in __run_exit_handlers stdlib/exit.c:108:8 previously allocated by thread T0 here: #0 0x564788e71fad in operator new(unsigned long) (/mnt/hdd01/dorisTestEnv/NEREIDS_ASAN/be/lib/doris_be+0x17581fad) (BuildId: 219493cc924323ee) #1 0x56478ace9c90 in std::_MakeUniq::__single_object std::make_unique, std::allocator > const&>(doris::MemTrackerLimiter::Type&&, std::__cxx11::basic_string, std::allocator > const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:30 #2 0x56478acde930 in doris::ShardedLRUCache::ShardedLRUCache(std::__cxx11::basic_string, std::allocator > const&, unsigned long, doris::LRUCacheType, unsigned int, unsigned int) /doris/be/src/olap/lru_cache.cpp:526:20 #3 0x56478ace22e1 in doris::new_lru_cache(std::__cxx11::basic_string, std::allocator > const&, unsigned long, doris::LRUCacheType, unsigned int) /doris/be/src/olap/lru_cache.cpp:670:16 #4 0x56478ad91da2 in doris::StoragePageCache::StoragePageCache(unsigned long, int, long, unsigned int) /doris/be/src/olap/page_cache.cpp:47:17 #5 0x56478ad9156e in doris::StoragePageCache::create_global_cache(unsigned long, int, long, unsigned int) /doris/be/src/olap/page_cache.cpp:31:29 #6 0x56478b98b3d3 in doris::ExecEnv::_init_mem_env() /doris/be/src/runtime/exec_env_init.cpp:251:5 #7 0x56478b98946c in doris::ExecEnv::_init(std::vector > const&) /doris/be/src/runtime/exec_env_init.cpp:182:5 #8 0x56478b987139 in doris::ExecEnv::init(doris::ExecEnv*, std::vector > const&) /doris/be/src/runtime/exec_env_init.cpp:98:17 #9 0x564788e79b50 in main /doris/be/src/service/doris_main.cpp:429:5 #10 0x7f217ca38564 in __libc_start_main csu/../csu/libc-start.c:332:16 Thread T41 (memory_tracker_) created by T0 here: #0 0x564788e1fcaa in pthread_create (/mnt/hdd01/dorisTestEnv/NEREIDS_ASAN/be/lib/doris_be+0x1752fcaa) (BuildId: 219493cc924323ee) #1 0x56478c4a2366 in doris::Thread::start_thread(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, std::function const&, unsigned long, scoped_refptr*) /doris/be/src/util/thread.cpp:419:15 #2 0x564788f59b91 in doris::Status doris::Thread::create(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, doris::Daemon::start()::$_4 const&, scoped_refptr*) /doris/be/src/util/thread.h:50:16 #3 0x564788f58165 in doris::Daemon::start() /doris/be/src/common/daemon.cpp:471:10 #4 0x564788e79a96 in main /doris/be/src/service/doris_main.cpp:420:12 #5 0x7f217ca38564 in __libc_start_main csu/../csu/libc-start.c:332:16
xy720
pushed a commit
that referenced
this pull request
Sep 13, 2023
when output meet order by not meet distribution. we use a trick way to
do enforce by set current output to any. but when we do enforce later,
we still use the old output. So when we do choose best plan, we could
not find the older output's plan, since we have replace it by any.
For example:
```
lowest Plan(cost, properties, plan, childrenRequires)
18.0 ANY
id:138#4 cost=0 [0/0/0/] estRows=4 children=[@0 ] (plan=PhysicalWindow[139]@4 ( windowFrameGroup=(Funcs=[row_number() WindowSpec(PARTITION BY b#1, a#0 ROWS BETWEEN UNBOUNDED_PRECEDING AND CURRENT_ROW) AS `r1`#2], PartitionKeys=[b#1, a#0], OrderKeys=[], WindowFrame=WindowFrame(ROWS, UNBOUNDED_PRECEDING, CURRENT_ROW)), requiredProperties=[DistributionSpecHash ( orderedShuffledColumns=[1, 0], shuffleType=REQUIRE, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1], [0]], exprIdToEquivalenceSet={0=1, 1=0} ) Order: ([b#1 asc, a#0 asc])], stats=null ))
[DistributionSpecHash ( orderedShuffledColumns=[0], shuffleType=NATURAL, tableId=3547296, selectedIndexId=3547297, partitionIds=[3547295], equivalenceExprIds=[[0]], exprIdToEquivalenceSet={0=0} ) Order: ([b#1 asc, a#0 asc])]
32.01171875 DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=REQUIRE, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} ) Order: ([b#1 asc])
id:161#4 cost=14 [4/4/4/] estRows=4 children=[@4 ] (plan=PhysicalQuickSort[162]@4 ( orderKeys=[b#1 asc], phase=LOCAL_SORT, stats=null ))
[DistributionSpecHash ( orderedShuffledColumns=[0], shuffleType=NATURAL, tableId=3547296, selectedIndexId=3547297, partitionIds=[3547295], equivalenceExprIds=[[0]], exprIdToEquivalenceSet={0=0} ) Order: ([b#1 asc, a#0 asc])]
32.01171875 DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} ) Order: ([b#1 asc])
id:161#4 cost=14 [4/4/4/] estRows=4 children=[@4 ] (plan=PhysicalQuickSort[162]@4 ( orderKeys=[b#1 asc], phase=LOCAL_SORT, stats=null ))
[DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} ) Order: ([])]
18.01171875 DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} ) Order: ([])
id:157#4 cost=0 [0/0/0/] estRows=4 children=[@4 ] (plan=PhysicalDistribute[158]@4 ( distributionSpec=DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} ), stats=null ))
[DistributionSpecHash ( orderedShuffledColumns=[0], shuffleType=NATURAL, tableId=3547296, selectedIndexId=3547297, partitionIds=[3547295], equivalenceExprIds=[[0]], exprIdToEquivalenceSet={0=0} ) Order: ([b#1 asc, a#0 asc])]
```
the last one require a natural shuffle type property from this group.
but this property already been removed when we do
enforceDistributionButMeetSort. So, such exception will be thrown
```
Caused by: org.apache.doris.nereids.exceptions.AnalysisException: Failed to choose best plan
at org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:340) ~[classes/:?]
at org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:323) ~[classes/:?]
... 18 more
Caused by: org.apache.doris.nereids.exceptions.AnalysisException: lowestCostPlans with physicalProperties(DistributionSpecHash ( orderedShuffledColumns=[0], shuffleType=NATURAL, tableId=3547296, selectedIndexId=3547297, partitionIds=[35
47295], equivalenceExprIds=[[0]], exprIdToEquivalenceSet={0=0} ) Order: ([b#1 asc, a#0 asc])) doesn't exist in root group
at org.apache.doris.nereids.NereidsPlanner.lambda$chooseBestPlan$1(NereidsPlanner.java:318) ~[classes/:?]
at java.util.Optional.orElseThrow(Optional.java:408) ~[?:?]
at org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:317) ~[classes/:?]
at org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:323) ~[classes/:?]
... 18 more
```
xy720
pushed a commit
that referenced
this pull request
Oct 11, 2023
… convert function (apache#17032) sql select convert('a' using utf8) cause coredump of BE: Thread 338 "FragmentMgrThre" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f0b2c21a700 (LWP 2865259)] 0x00007fff04f8a015 in __gconv_close () from /lib64/libc.so.6 (gdb) bt #0 0x00007fff04f8a015 in __gconv_close () from /lib64/libc.so.6 #1 0x00007fff04f899e3 in iconv_close () from /lib64/libc.so.6 #2 0x000055556f92f5a7 in doris::vectorized::FunctionConvertTo::close (this=0x606003f28f50, context=0x6020039fe710, scope=doris_udf::FunctionContext::THREAD_LOCAL) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/functions/function_string.h:2442 #3 0x000055556db0598d in doris::vectorized::DefaultFunction::close (this=0x607000748f00, context=0x6020039fe710, scope=doris_udf::FunctionContext::THREAD_LOCAL) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/functions/function.h:524 #4 0x000055556da00b66 in doris::vectorized::VExpr::close_function_context (this=0x619004ca1d80, context=0x606003fad300, scope=doris_udf::FunctionContext::FRAGMENT_LOCAL, function=...) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/exprs/vexpr.cpp:373 #5 0x000055556da16455 in doris::vectorized::VectorizedFnCall::close (this=0x619004ca1d80, state=0x61c00026c880, context=0x606003fad300, scope=doris_udf::FunctionContext::FRAGMENT_LOCAL) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/exprs/vectorized_fn_call.cpp:92 #6 0x000055556da26afd in doris::vectorized::VExprContext::close (this=0x606003fad300, state=0x61c00026c880) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/exprs/vexpr_context.cpp:72 #7 0x000055556d9fcff1 in doris::vectorized::VExpr::close (ctxs=..., state=0x61c00026c880) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/exprs/vexpr.cpp:242 #8 0x000055556aa32c9a in doris::vectorized::VSortExecExprs::close (this=0x618000345f00, state=0x61c00026c880) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/common/sort/vsort_exec_exprs.cpp:77 #9 0x000055556aa26dae in doris::vectorized::VSortNode::close (this=0x618000345c80, state=0x61c00026c880) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/exec/vsort_node.cpp:129 #10 0x000055556794b5ef in doris::PlanFragmentExecutor::close (this=0x6170005a35f0) at /mnt/disk2/tengjianping/doris-1.2/be/src/runtime/plan_fragment_executor.cpp:676 apache#11 0x00005555678ce1fb in doris::FragmentExecState::execute (this=0x6170005a3580) at /mnt/disk2/tengjianping/doris-1.2/be/src/runtime/fragment_mgr.cpp:258 apache#12 0x00005555678d6633 in doris::FragmentMgr::_exec_actual(std::shared_ptr<doris::FragmentExecState>, std::function<void (doris::PlanFragmentExecutor*)>) (this=0x627000003900, exec_state=..., cb=...) at /mnt/disk2/tengjianping/doris-1.2/be/src/runtime/fragment_mgr.cpp:497 apache#13 0x00005555678d8aa3 in operator() (__closure=0x60800014cda0) at /mnt/disk2/tengjianping/doris-1.2/be/src/runtime/fragment_mgr.cpp:721
xy720
pushed a commit
that referenced
this pull request
Oct 11, 2023
… memory apache#17189 #0 doris::MemTracker::consumption (this=0x0) at /root/doris/be/src/runtime/memory/mem_tracker.h:65 #1 doris::PushHandler::_convert_v2 (this=this@entry=0x7ffec1e17f90, cur_tablet=..., cur_rowset=cur_rowset@entry=0x7ffec1e177a0, tablet_schema=...) at /root/doris/be/src/olap/push_handler.cpp:247 #2 0x000055555a271bc6 in doris::PushHandler::_do_streaming_ingestion (this=this@entry=0x7ffec1e17f90, tablet=..., request=..., push_type=push_type@entry=doris::PUSH_NORMAL_V2, tablet_info_vec=tablet_info_vec@entry=0x7ffec1e185d0) at /var/local/ldb-toolchain/include/c++/11/ext/atomicity.h:109 #3 0x000055555a27222c in doris::PushHandler::process_streaming_ingestion
xy720
pushed a commit
that referenced
this pull request
Oct 24, 2023
Template arguments for decimalv3 type is wrong when creating MinMaxNumFunc, which result in wrong decimal type like vectorized::Decimal<doris::vectorized::Decimal128I> :
class MinmaxFunctionTraits {
public:
using BasePtr = MinMaxFuncBase*;
template <PrimitiveType type, size_t N>
static BasePtr get_function() {
return new MinMaxNumFunc<std::conditional_t<
type == TYPE_DECIMAL32 || type == TYPE_DECIMAL64 || type == TYPE_DECIMAL128I,
vectorized::Decimal<typename PrimitiveTypeTraits<type>::CppType>,
typename PrimitiveTypeTraits<type>::CppType>>();
}
};
(gdb) bt
#0 doris::vectorized::Decimal<__int128>::operator<=>(doris::vectorized::Decimal<__int128> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381
#1 doris::vectorized::Decimal<doris::vectorized::Decimal128I>::operator<=>(doris::vectorized::Decimal<doris::vectorized::Decimal128I> const&) const (x=..., this=<optimized out>)
at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381
#2 doris::MinMaxNumFunc<doris::vectorized::Decimal<doris::vectorized::Decimal128I> >::insert (this=0x7ef19de903c0, data=0x7c612928d860) at /mnt/disk2/user/doris-1.1/be/src/exprs/minmax_predicate.h:61
#3 0x0000558465c2844a in doris::RuntimePredicateWrapper::insert (this=0x7f47a2e3d720, value=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:492
#4 doris::RuntimePredicateWrapper::insert_batch (this=0x7f47a2e3d720, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:504
#5 0x0000558465c1f6c3 in doris::IRuntimeFilter::insert_batch (this=<optimized out>, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:1060
#6 0x000055846883dafd in doris::VRuntimeFilterSlots::insert (this=0x7d7e767f4e70, datas=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter_slots.h:188
#7 0x0000558468840535 in doris::vectorized::ProcessRuntimeFilterBuild::operator()<doris::vectorized::MethodOneNumber<doris::vectorized::UInt128, PartitionedHashMapTable<HashMapTable<doris::vectorized::UInt128, HashMapCell<doris::vectorized::UInt128, doris::vectorized::RowRefList, HashCRC32<doris::vectorized::UInt128>, HashTableNoState>, HashCRC32<doris::vectorized::UInt128>, PartitionedHashTableGrower<8ul>, Allocator<true, true, false> > > >, doris::vectorized::HashJoinNode> (
this=this@entry=0x7e7c6a5bee40, state=0x7ef19df00400, hash_table_ctx=..., parent=0x7ef19dee3c00) at /mnt/disk2/user/doris-1.1/be/src/vec/exec/join/vhash_join_node.h:92
xy720
pushed a commit
that referenced
this pull request
Dec 19, 2023
…_in_blocks` (apache#28536) Otherwise using the stale `_row_in_blocks` will result in heap-buffer-overflow ``` ==2695213==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62900122e210 at pc 0x56524744aecf bp 0x7f62c595ef7 0 sp 0x7f62c595ef68 READ of size 8 at 0x62900122e210 thread T1627 (MemTableFlushTh) #0 0x56524744aece in doris::vectorized::ColumnVector<long>::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_vector.cpp:378:33 #1 0x5652472a7538 in doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_nullable.cpp:310:25 #2 0x56524782a62a in doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/core/block.cpp:961:14 #3 0x565233f187ae in doris::MemTable::_put_into_output(doris::vectorized::Block&) /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:248:27 #4 0x565233f1db66 in doris::MemTable::to_block() /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:496:13 #5 0x565233efae60 in doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:121:62 #6 0x565233efc8d6 in doris::FlushToken::_flush_memtable(doris::MemTable*, int, long) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:150:16 #7 0x565233f0c5eb in doris::MemtableFlushTask::run() /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:58:23 ```
xy720
pushed a commit
that referenced
this pull request
Jun 11, 2024
…l_backend() (apache#35634) ## Proposed changes ```cpp ==1825462==ERROR: AddressSanitizer: heap-use-after-free on address 0x5030012ea880 at pc 0x555cc645e1aa bp 0x7fe4a9407e20 sp 0x7fe4a94075e0 READ of size 22 at 0x5030012ea880 thread T2250 (REPORT_DISK_STA) #0 0x555cc645e1a9 in __asan_memcpy (/mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be+0x22cb01a9) (BuildId: 76fba423c52f4da5) #1 0x555cc64b6990 in std::char_traits<char>::copy(char*, char const*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/char_traits.h:445:33 #2 0x555cc64b68a1 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_S_copy(char*, char const*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:420:4 #3 0x555cc64c526e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:291:6 #4 0x555cc64c5070 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:1596:8 #5 0x555cc64b1d9c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:802:15 #6 0x555ccb05a722 in doris::TBackend::TBackend(doris::TBackend const&) /mnt/disk1/xiaolei/incubator-doris/gensrc/build/gen_cpp/Types_types.cpp:5755:8 #7 0x555cc9c9234e in doris::BackendOptions::get_local_backend() /mnt/disk1/xiaolei/incubator-doris/be/src/service/backend_options.cpp:79:12 #8 0x555cc6586e5a in doris::report_disk_callback(doris::StorageEngine&, doris::TMasterInfo const&) /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:1021:27 #9 0x555cc64f2a0a in doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/agent_server.cpp:190:133 #10 0x555cc64f2994 in void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#11 0x555cc64f2944 in std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#12 0x555cc64f280c in std::_Function_handler<void (), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#13 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#14 0x555cc65b01f9 in doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:689:13 apache#15 0x555cc65afcd4 in void std::__invoke_impl<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(std::__invoke_other, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#16 0x555cc65afc74 in std::enable_if<is_invocable_r_v<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>, void>::type std::__invoke_r<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#17 0x555cc65afa6c in std::_Function_handler<void (), doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#18 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#19 0x555cca138dab in doris::Thread::supervise_thread(void*) /mnt/disk1/xiaolei/incubator-doris/be/src/util/thread.cpp:498:5 apache#20 0x555cc645de0a in asan_thread_start(void*) crtstuff.c apache#21 0x7fef061601c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) apache#22 0x7fef06b4fe72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) 0x5030012ea880 is located 0 bytes inside of 31-byte region [0x5030012ea880,0x5030012ea89f) freed by thread T2249 (REPORT_TASK-182) here: #0 0x555cc649d43d in operator delete(void*) (/mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be+0x22cef43d) (BuildId: 76fba423c52f4da5) #1 0x555cc64b6bbc in std::__new_allocator<char>::deallocate(char*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/new_allocator.h:168:2 #2 0x555cc64b6b5d in std::allocator<char>::deallocate(char*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/allocator.h:210:25 #3 0x555cc64b6b5d in std::allocator_traits<std::allocator<char>>::deallocate(std::allocator<char>&, char*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/alloc_traits.h:516:13 #4 0x555cc64b6b5d in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_destroy(unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:289:9 #5 0x555cc64b6a56 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_dispose() /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:283:4 #6 0x555cc64c51ee in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:285:8 #7 0x555cc64c5070 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:1596:8 #8 0x555cc64b1d9c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:802:15 #9 0x555ccb058510 in doris::TBackend::__set_host(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/incubator-doris/gensrc/build/gen_cpp/Types_types.cpp:5584:14 #10 0x555cc9c92239 in doris::BackendOptions::get_local_backend() /mnt/disk1/xiaolei/incubator-doris/be/src/service/backend_options.cpp:75:14 apache#11 0x555cc6585b9d in doris::report_task_callback(doris::TMasterInfo const&) /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:1004:27 apache#12 0x555cc64f24b9 in doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/agent_server.cpp:187:112 apache#13 0x555cc64f2474 in void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#14 0x555cc64f2424 in std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#15 0x555cc64f22ec in std::_Function_handler<void (), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#16 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#17 0x555cc65b01f9 in doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:689:13 apache#18 0x555cc65afcd4 in void std::__invoke_impl<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(std::__invoke_other, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#19 0x555cc65afc74 in std::enable_if<is_invocable_r_v<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>, void>::type std::__invoke_r<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#20 0x555cc65afa6c in std::_Function_handler<void (), doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#21 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#22 0x555cca138dab in doris::Thread::supervise_thread(void*) /mnt/disk1/xiaolei/incubator-doris/be/src/util/thread.cpp:498:5 apache#23 0x555cc645de0a in asan_thread_start(void*) crtstuff.c previously allocated by thread T2250 (REPORT_DISK_STA) here: #0 0x555cc649cbdd in operator new(unsigned long) (/mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be+0x22ceebdd) (BuildId: 76fba423c52f4da5) #1 0x555cc64b684e in std::__new_allocator<char>::allocate(unsigned long, void const*) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/new_allocator.h:147:27 #2 0x555cc64b67b0 in std::allocator<char>::allocate(unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/allocator.h:198:32 #3 0x555cc64b67b0 in std::allocator_traits<std::allocator<char>>::allocate(std::allocator<char>&, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/alloc_traits.h:482:20 #4 0x555cc64b67b0 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_S_allocate(std::allocator<char>&, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:126:16 #5 0x555cc64b63b1 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_create(unsigned long&, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:155:14 #6 0x555cc64c51de in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:284:24 #7 0x555cc64c5070 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:1596:8 #8 0x555cc64b1d9c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:802:15 #9 0x555ccb058510 in doris::TBackend::__set_host(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/incubator-doris/gensrc/build/gen_cpp/Types_types.cpp:5584:14 #10 0x555cc9c92239 in doris::BackendOptions::get_local_backend() /mnt/disk1/xiaolei/incubator-doris/be/src/service/backend_options.cpp:75:14 apache#11 0x555cc6586e5a in doris::report_disk_callback(doris::StorageEngine&, doris::TMasterInfo const&) /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:1021:27 apache#12 0x555cc64f2a0a in doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/agent_server.cpp:190:133 apache#13 0x555cc64f2994 in void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#14 0x555cc64f2944 in std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#15 0x555cc64f280c in std::_Function_handler<void (), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#16 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#17 0x555cc65b01f9 in doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:689:13 apache#18 0x555cc65afcd4 in void std::__invoke_impl<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(std::__invoke_other, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#19 0x555cc65afc74 in std::enable_if<is_invocable_r_v<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>, void>::type std::__invoke_r<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#20 0x555cc65afa6c in std::_Function_handler<void (), doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#21 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#22 0x555cca138dab in doris::Thread::supervise_thread(void*) /mnt/disk1/xiaolei/incubator-doris/be/src/util/thread.cpp:498:5 apache#23 0x555cc645de0a in asan_thread_start(void*) crtstuff.c ```
xy720
pushed a commit
that referenced
this pull request
Jun 11, 2024
…apache#35458) ## Proposed changes Move memtable core when use multi table load: ``` 0x51f000c73860 is located 3040 bytes inside of 3456-byte region [0x51f000c72c80,0x51f000c73a00) freed by thread T4867 (FragmentMgrThre) here: #0 0x558f6ad7f43d in operator delete(void*) (/mnt/hdd01/STRESS_ENV/be/lib/doris_be+0x22eec43d) (BuildId: b46f73d1f76dfcd6) #1 0x558f6e6cea2c in std::__new_allocator<doris::PTabletID>::deallocate(doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/new_allocator.h:168:2 #2 0x558f6e6ce9e7 in std::allocator<doris::PTabletID>::deallocate(doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/allocator.h:210:25 #3 0x558f6e6ce9e7 in std::allocator_traits<std::allocator<doris::PTabletID>>::deallocate(std::allocator<doris::PTabletID>&, doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/alloc_traits.h:516:13 #4 0x558f6e6ce9e7 in std::_Vector_base<doris::PTabletID, std::allocator<doris::PTabletID>>::_M_deallocate(doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_vector.h:387:4 #5 0x558f6e6d0780 in void std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>::_M_range_insert<__gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>>(__gnu_cxx::__normal_iterator<doris::PTabletID*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, std::forward_iterator_tag) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/vector.tcc:832:3 #6 0x558f6e6c54c5 in __gnu_cxx::__normal_iterator<doris::PTabletID*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>> std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>::insert<__gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, void>(__gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_vector.h:1483:4 #7 0x558f9b4b214f in doris::LoadStreamMap::save_tablets_to_commit(long, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>> const&) /mnt/disk2/xujianxu/doris/be/src/vec/sink/load_stream_map_pool.cpp:90:13 #8 0x558f9b7258dd in doris::vectorized::VTabletWriterV2::_calc_tablets_to_commit() /mnt/disk2/xujianxu/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:650:27 #9 0x558f9b7229f1 in doris::vectorized::VTabletWriterV2::close(doris::Status) /mnt/disk2/xujianxu/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:547:9 ``` Multiple sinks with different table loads use the load id, causing confusion in the use of shared data structures between sinks.
xy720
pushed a commit
that referenced
this pull request
Jun 14, 2024
…he#36086) ## Proposed changes ``` /root/doris/be/src/vec/common/pod_array.h:448:13: runtime error: applying non-zero offset 16 to null pointer #0 0x562f904159f3 in void doris::vectorized::PODArray<doris::vectorized::AggregateFunctionSequenceMatchData<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long, doris::vectorized::AggregateFunctionSequenceMatch<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long> >::PatternAction, 64ul, AllocatorWithStackMemory<Allocator<false, false, false>, 64ul, 8ul>, 0ul, 0ul>::emplace_back<doris::vectorized::AggregateFunctionSequenceMatchData<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long, doris::vectorized::AggregateFunctionSequenceMatch<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long> >::PatternActionType>(doris::vectorized::AggregateFunctionSequenceMatchData<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long, doris::vectorized::AggregateFunctionSequenceMatch<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long> >::PatternActionType&&) /root/doris/be/src/vec/common/pod_array.h:448:13 #1 0x562f90412d6a in doris::vectorized::AggregateFunctionSequenceMatchData<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long, doris::vectorized::AggregateFunctionSequenceMatch<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long> >::parse_pattern() /root/doris/be/src/vec/aggregate_functions/aggregate_function_sequence_match.h:208:17 #2 0x562f9040a824 in doris::vectorized::AggregateFunctionSequenceMatchData<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long, doris::vectorized::AggregateFunctionSequenceMatch<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long> >::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long) /root/doris/be/src/vec/aggregate_functions/aggregate_function_sequence_match.h:95:13 ``` this->c_end may be nullptr
xy720
pushed a commit
that referenced
this pull request
Jun 14, 2024
## Proposed changes If group commit time interval is larger than the load timeout, and there is no new client load to reuse the internal group commit load, the group commit can not cancel in time because it stuck in wait: ``` #0 0x00007f33937a47aa in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005651105dbd05 in __gthread_cond_timedwait(pthread_cond_t*, pthread_mutex_t*, timespec const*) () #2 0x000056511063f385 in std::__condvar::wait_until(std::mutex&, timespec&) () #3 0x000056511063dc2e in std::cv_status std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) () #4 0x000056511063cedf in std::cv_status std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) () #5 0x0000565110824f48 in std::cv_status std::condition_variable::wait_for<long, std::ratio<1l, 1000l> >(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&) () #6 0x0000565113b5612a in doris::LoadBlockQueue::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*, bool*) () #7 0x000056513f900941 in doris::pipeline::GroupCommitOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) () #8 0x000056513c69c0b6 in doris::pipeline::ScanOperatorX<doris::pipeline::GroupCommitLocalState>::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) () #9 0x000056514009d5f1 in doris::pipeline::PipelineTask::execute(bool*) () #10 0x00005651400fb24a in doris::pipeline::TaskScheduler::_do_work(unsigned long) () ```
xy720
pushed a commit
that referenced
this pull request
Jun 14, 2024
…t is not nullable (apache#36263) ## Proposed changes For a null safe equal join, FE may generate a plan that both sides of the conjuct are not nullable, which is a valid plan, BE will coredump in this situation. Issue Number: close #xxx Coredump stacktrace: ``` #0 0x00007fff042caacf in raise () from /lib64/libc.so.6 #1 0x00007fff0429dea5 in abort () from /lib64/libc.so.6 #2 0x00005555a7ec7cad in ?? () #3 0x00005555a7eba34a in google::LogMessage::Fail() () #4 0x00005555a7ebd436 in google::LogMessage::SendToLog() () #5 0x00005555a7eb9e80 in google::LogMessage::Flush() () #6 0x00005555a7ebdc79 in google::LogMessageFatal::~LogMessageFatal() () #7 0x000055557824c9ac in doris::vectorized::IColumn::get_raw_data (this=0x60400387df90) at /mnt/disk2/tengjianping/doris-39/be/src/vec/columns/column.h:590 #8 0x00005555a4a85f32 in doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>::pack_fixeds<unsigned long> (this=0x6110086dcb50, row_numbers=21, key_columns=std::vector of length 1, capacity 1 = {...}, nullmap_columns=std::vector of length 0, capacity 0, result=std::vector of length 21, capacity 21 = {...}) at /mnt/disk2/tengjianping/doris-39/be/src/vec/common/hash_table/hash_map_context.h:405 #9 0x00005555a4a83a77 in doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>::init_serialized_keys (this=0x6110086dcb50, key_columns=std::vector of length 1, capacity 1 = {...}, num_rows=21, null_map=0x0, is_join=true, is_build=true, bucket_size=32) at /mnt/disk2/tengjianping/doris-39/be/src/vec/common/hash_table/hash_map_context.h:470 #10 0x00005555a48d9228 in doris::pipeline::ProcessHashTableBuild<doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false> >::run<1, false, false, false> (this=0x75d6195d92a0, hash_table_ctx=..., null_map=0x0, has_null_key=0x6120057db3c0) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.h:208 apache#11 0x00005555a456dd23 in doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeS tate*, doris::vectorized::Block&)::$_1::operator()<doris::vectorized::MethodKeysFixed<doris::JoinHashTable< unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<do--Type <RET> for more, q to qu --Type <RET> for more, q to quit, c to continue without paging--c ris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false> >(doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>) const (this=0x75d61989aa10, arg=..., join_op=..., has_null_value=..., short_circuit_for_null_in_build_side=..., with_other_conjuncts=...) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.cpp:296 apache#12 0x00005555a456d93c in std::__invoke_impl<doris::Status, doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false> >(std::__invoke_other, doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) (__f=..., __args=..., __args=..., __args=..., __args=..., __args=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 apache#13 0x00005555a456d88c in std::__invoke<doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false> >(doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) (__fn=..., __args=..., __args=..., __args=..., __args=..., __args=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96 apache#14 0x00005555a43eef75 in _ZNSt8__detail9__variant17__gen_vtable_implINS0_12_Multi_arrayIPFNS0_21__deduce_visit_resultIN5doris6StatusEEEONS4_10vectorized8OverloadIJZNS4_8pipeline27HashJoinBuildSinkLocalState19process_build_blockEPNS4_12RuntimeStateERNS7_5BlockEE3$_0ZNSA_19process_build_blockESC_SE_E3$_1EEERSt7variantIJSt9monostateNS7_16MethodSerializedINS4_13JoinHashTableINS4_9StringRefE11DefaultHashISN_vEEEEENS7_15MethodOneNumberIhNSM_Ih9HashCRC32IhEEEEENSS_ItNSM_ItST_ItEEEEENSS_IjNSM_IjST_IjEEEEENSS_ImNSM_ImST_ImEEEEENSS_IN4wide7integerILm128EjEENSM_IS18_ST_IS18_EEEEENSS_INS17_ILm256EjEENSM_IS1C_ST_IS1C_EEEEENS7_15MethodKeysFixedIS14_Lb1EEENS1G_IS14_Lb0EEENS1G_IS1A_Lb1EEENS1G_IS1A_Lb0EEENS1G_IS1E_Lb1EEENS1G_IS1E_Lb0EEENS1G_INSM_INS7_7UInt136EST_IS1N_EEELb1EEENS1G_IS1P_Lb0EEEEERSJ_IJSt17integral_constantINS4_7TJoinOp4typeELS1W_0EES1U_IS1W_LS1W_2EES1U_IS1W_LS1W_8EES1U_IS1W_LS1W_1EES1U_IS1W_LS1W_4EES1U_IS1W_LS1W_3EES1U_IS1W_LS1W_5EES1U_IS1W_LS1W_7EES1U_IS1W_LS1W_9EES1U_IS1W_LS1W_10EES1U_IS1W_LS1W_11EEEEOSJ_IJS1U_IbLb0EES1U_IbLb1EEEES2D_S2D_EJEEESt16integer_sequenceImJLm9ELm3ELm0ELm0ELm0EEEE14__visit_invokeESI_S1T_S29_S2D_S2D_S2D_ (__visitor=..., __vars=..., __vars=..., __vars=..., __vars=..., __vars=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013 apache#15 0x00005555a43cf068 in std::__do_visit<std::__detail::__variant::__deduce_visit_result<doris::Status>, doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> > >(doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&) (__visitor=..., __variants=..., __variants=..., __variants=..., __variants=..., __variants=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1714 apache#16 0x00005555a43c8dea in std::visit<doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> > >(doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&) (__visitor=..., __variants=..., __variants=..., __variants=..., __variants=..., __variants=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1769 apache#17 0x00005555a43c848d in doris::pipeline::HashJoinBuildSinkLocalState::process_build_block (this=0x616000669080, state=0x61e003eca080, block=...) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.cpp:278 apache#18 0x00005555a43cd0a2 in doris::pipeline::HashJoinBuildSinkOperatorX::sink (this=0x613001702bc0, state=0x61e003eca080, in_block=0x60d0000f7d40, eos=true) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.cpp:557 apache#19 0x00005555a7bdd4e1 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const (this=0x75d619925bd0) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/pipeline_task.cpp:356 apache#20 0x00005555a7bda84c in doris::pipeline::PipelineTask::execute (this=0x61700073d800, eos=0x75d6198f4230) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/pipeline_task.cpp:359 apache#21 0x00005555a7c34c6a in doris::pipeline::TaskScheduler::_do_work (this=0x60b00226b9e0, index=10) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/task_scheduler.cpp:135 apache#22 0x00005555a7c374bb in doris::pipeline::TaskScheduler::start()::$_0::operator()() const (this=0x6060055527f8) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/task_scheduler.cpp:64 ```
xy720
pushed a commit
that referenced
this pull request
Jun 18, 2024
…6275) If we execute a delete predicate in an inverted index table and then query something from it, it will cause a core dump in shrink_char_type_column_suffix_zero. This occurs because the delete predicate column ID in _char_type_idx is incorrectly shifted to the result column inserted by the inverted index result. coredump stack like: ``` COULD NOT CREATE A LOGGINGFILE 20240604-145331!F20240604 14:53:31.991016 24178 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8)F20240604 14:53:31.991645 24160 column.h:134] Cannot get_shrinked_column() column Const(UInt8) #0 0x00007fa0bf784387 in ?? () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libc.so.6 #1 0x0000557dc8e617a0 in google::LogMessage::Flush() () #2 0x0000557dc8e65599 in google::LogMessageFatal::~LogMessageFatal() () #3 0x0000557dbe4ec939 in doris::vectorized::IColumn::get_shrinked_column (this=0x7fa094ac16a0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column.h:134 #4 0x0000557dc236d8a2 in doris::vectorized::Block::shrink_char_type_column_suffix_zero (this=0x7f9a372a4b70, char_type_idx=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp:1126 #5 0x0000557dbee9ea95 in doris::segment_v2::SegmentIterator::_next_batch_internal (this=<optimized out>, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2408 #6 0x0000557dbee9cc5a in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const (this=<optimized out>) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2090 #7 doris::segment_v2::SegmentIterator::next_batch (this=0x1c5d, block=0x6) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2089 #8 0x0000557dbeebbb44 in doris::BetaRowsetReader::next_block (this=0x7f9a3a041600, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/beta_rowset_reader.cpp:342 #9 0x0000557dc8590ddd in doris::vectorized::VCollectIterator::Level0Iterator::_refresh (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.h:256 #10 doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:514 apache#11 0x0000557dc8591055 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref (this=0x1c5d) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:493 apache#12 0x0000557dc85934f2 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref (this=0x7fa090e36c00) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:692 apache#13 0x0000557dc858ddc9 in doris::vectorized::VCollectIterator::build_heap (this=0x7f9d80976dc0, rs_readers=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:186 apache#14 0x0000557dc857e808 in doris::vectorized::BlockReader::_init_collect_iter (this=this@entry=0x7f9d80976800, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:156 apache#15 0x0000557dc857f64d in doris::vectorized::BlockReader::init (this=<optimized out>, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:228 apache#16 0x0000557dc370dc2c in doris::vectorized::NewOlapScanner::open (this=0x7f9a2aa68210, state=<optimized out>) --Type <RET> for more, q to quit, c to continue without paging--c at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:227 apache#17 0x0000557dc3718d89 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=std::shared_ptr<doris::vectorized::ScannerContext> (use count 4, weak count 1) = {...}, scan_task=std::shared_ptr<doris::vectorized::ScanTask> (use count 2, weak count 0) = {...}) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:242 apache#18 0x0000557dc3719b77 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const (this=0x0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:158 apache#19 std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__f=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 apache#20 std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__fn=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111 apache#21 std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1>::_M_invoke(std::_Any_data const&) (__functor=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 apache#22 0x0000557dbf3c7788 in doris::ThreadPool::dispatch_thread (this=0x7f9d0bbdc400) at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:543 apache#23 0x0000557dbf3bcb41 in std::function<void ()>::operator()() const (this=0x1471) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 apache#24 doris::Thread::supervise_thread (arg=0x7f9d0bbdf4c0) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:498 apache#25 0x00007fa0bee1dea5 in sem_open () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libpthread.so.0 ```
xy720
pushed a commit
that referenced
this pull request
Jun 21, 2024
… rewrite logic (apache#36161) NormalizeAggregate rewrite logic has a bug, for sql like this: SELECT CASE 1 WHEN CAST( NULL AS SIGNED ) THEN NULL WHEN COUNT( DISTINCT CAST( NULL AS SIGNED ) ) THEN NULL ELSE null END ; This is the plan after NormalizeAggregate, the LogicalAggregate only output `count(DISTINCT cast(NULL as SIGNED))`#3, do not output cast(NULL as SIGNED)#2, but the upper project use cast(NULL as SIGNED)#2, so Doris report error "cast(NULL as SIGNED) not in aggregate's output". LogicalResultSink[29] ( outputExprs=[__case_when_0#1] ) +--LogicalProject[26] ( distinct=false, projects=[CASE WHEN (1 = cast(NULL as SIGNED)#2) THEN NULL WHEN (1 = count(DISTINCT cast(NULL as SIGNED))#3) THEN NULL ELSE NULL END AS `CASE WHEN (1 = cast(NULL as SIGNED)) THEN NULL WHEN (1 = count(DISTINCT cast(NULL as SIGNED))) THEN NULL ELSE NULL END`#1], excepts=[] ) +--LogicalAggregate[25] ( groupByExpr=[], outputExpr=[count(DISTINCT cast(NULL as SIGNED)#2) AS `count(DISTINCT cast(NULL as SIGNED))`#3], hasRepeat=false ) +--LogicalProject[24] ( distinct=false, projects=[cast(NULL as SIGNED) AS `cast(NULL as SIGNED)`#2], excepts=[] ) +--LogicalOneRowRelation ( projects=[0 AS `0`#0] ) The problem is that the cast(NULL as SIGNED)#2 should not outputted by LogicalAggregate, cast(NULL as SIGNED) should be computed in LogicalProject. This pr change the upper project projections rewrite logic: aggregateOutputs is rewritten and become the upper-level LogicalProject projections. During the rewriting process, the expressions inside the agg function can be rewritten with expressions in aggregate function arguments and group by expressions, but the ones outside the agg function can only be rewritten with group by expressions.
xy720
pushed a commit
that referenced
this pull request
Jul 1, 2024
## Proposed changes
The legacy coordinator act not only scheduler but also distribute planner. The code is so complex to understand, and hard to extend, and exist many limitations.
This pr extract and refine the computation of degree of parallel(dop) to a new DistributePlanner and resolve the limitations.
## How to use this function
This function only use for nereids + pipelinex, and current only support query statement, and non cloud mode.
Open this session variables to use this function:
```sql
set enable_nereids_distribute_planner=true; -- default is false
set enable_nereids_planner=true; -- default is true
```
## Core process and concepts
```
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ │
│ ┌──────────────┐ ┌───────────────┐ ┌───────────────────┐ ┌─────────────────────────┐ │
│ Translate │ │ Typed │ │ Assign │ │ Wrap │ │ │
│ ──────────► │ PlanFragment │ ──────► │ UnassignedJob │ ───────► │ StaticAssignedJob │ ─────► │ PipelineDistributedPlan │ │
│ │ │ │ │ │ │ │ │ │
│ └──────────────┘ └───────────────┘ └───────────────────┘ └─────────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
│ │
│ │
└──────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────┘
│ │
│ │
┌──────────────┐ ┌─────────────────┐ ┌───────────────────┐
│ │ Distribute │ │ AdHoc │ │
│ PhysicalPlan │ ───────────► │ DistributedPlan │ ──────► │ PipelineScheduler │
│ │ │ │ │ │
└──────────────┘ └─────────────────┘ └───────────────────┘
```
DistributePlanner is a new planner to compute dop and generate instances, it consume PlanFragment and do this tasks
1. Use PlanFragment to generate `UnassignedJob`, it's a **Typed Fragment**, decided how to calculate dop and how to select the datasource, but this fragment not yet assigned some backends and datasources. These are some unassignedJobs: UnassignedScanSingleOlapTableJob, UnassignedScanBucketOlapTableJob, UnassignedShuffleJob, UnassignedQueryConstantJob. Keep UnassignedJob different can decoupling unrelated logic, and easy to extend: just and a new type of UnassignedJob.
2. Use UnassignedJob to select datasource, compute dop, and generate `AssignedJob`, means a instance, which already assigned datasource and backend. There are StaticAssignedJob and LocalShuffleAssignedJob, we will add DynamicAssignedJob when support StageScheduler and adaptive query execution
3. Wrap PlanFragment, UnassignedJob and AssignedJob to `PipelineDistributedPlan`, the coordinator will consume the DistributedPlan and translate to TPlan and schedule instances
## Resolve limitations
**1. left table shuffle to right table**
if right table has distribution which distribute by `storage hash`, and left table has distribution which distribute by `compute hash`, we can shuffle left to right by `storage hash` to do shuffle bucket join, and keep right side not move.
```sql
select *
from
(
select id2
from test_shuffle_left
group by id2
) a
inner join [shuffle]
test_shuffle_left b
on a.id2=b.id;
| PhysicalResultSink[288] ( outputExprs=[id2#1, id#2, id2#3] ) ...
| +--PhysicalHashJoin[285]@4 ( type=INNER_JOIN, stats=3, hashCondition=[(id2#1 = id#2)], otherCondition=[], markCondition=[], hint=[shuffle] ) ...
| |--PhysicalDistribute[281]@2 ( stats=1.5, distributionSpec=DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=STORAGE_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=...
| | +--PhysicalHashAggregate[278]@2 ( aggPhase=GLOBAL, aggMode=BUFFER_TO_RESULT, maybeUseStreaming=false, groupByExpr=[id2#1], outputExpr=[id2#1], partitionExpr=Optional[[id2#1]], requir...
| | +--PhysicalDistribute[275]@7 ( stats=1.5, distributionSpec=DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, parti...
| | +--PhysicalHashAggregate[272]@7 ( aggPhase=LOCAL, aggMode=INPUT_TO_BUFFER, maybeUseStreaming=true, groupByExpr=[id2#1], outputExpr=[id2#1], partitionExpr=Optional[[id2#1]], req...
| | +--PhysicalProject[269]@1 ( stats=3, projects=[id2#1] ) ...
| | +--PhysicalOlapScan[test_shuffle_left]@0 ( stats=3 ) ...
| +--PhysicalOlapScan[test_shuffle_left]@3 ( stats=3 )
```
**2. support colocate union numbers function**
support use one instance to union/join numbers, note this plan no any PhysicalDistribute plan:
```sql
explain physical plan
select * from numbers('number'='3')a
union all
select * from numbers('number'='4')b
PhysicalResultSink[98] ( outputExprs=[number#2] )
+--PhysicalUnion@ ( qualifier=ALL, outputs=[number#2], regularChildrenOutputs=[[number#0], [number#1]], constantExprsList=[], stats=7 )
|--PhysicalTVFRelation ( qualified=NumbersTableValuedFunction, output=[number#0], function=numbers('number' = '3') )
+--PhysicalTVFRelation ( qualified=NumbersTableValuedFunction, output=[number#1], function=numbers('number' = '4') )
```
**3. support bucket prune with right outer bucket shuffle join**
left table prune some buckets, say [bucket 1, bucket 3]
we should process the right bucket shuffle join like this
```
[
(left bucket 1) right outer join (exchange right table which should process by bucket 1),
(empty bucket) right outer join (exchange right table which should process by bucket 2),
(left bucket 3) right outer join (exchange right table which should process by bucket 3)
]
```
the left bucket 2 is pruned, so right table can not shuffle to left, because the left instance not exists, so bucket 2 will return empty rows and wrong.
new DistributePlanner can fill up this instance.
the case:
```sql
explain physical plan
SELECT * FROM
(select * from test_outer_join1 where c0 =1)a
RIGHT OUTER JOIN
(select * from test_outer_join2)b
ON a.c0 = b.c0
```
### New feature
add an explain statement to show distributed plans
```sql
explain distributed plan select ...
```
for example, you can use this function to check how many instances generated, how many bytes the instance will scan, which backend will process the instance:
```sql
MySQL root@127.0.0.1:test> explain distributed plan select * from test_shuffle_left2 a join [shuffle] test_shuffle_left2 b on a.id2=b.id;
Explain String(Nereids Planner)
-------------------------------------------------------------------------------------------------------
PipelineDistributedPlan(
id: 0,
parallel: 2,
fragmentJob: UnassignedScanSingleOlapTableJob,
fragment: {
OUTPUT EXPRS:
id[#8]
id2[#9]
id[#10]
id2[apache#11]
PARTITION: HASH_PARTITIONED: id2[#3]
HAS_COLO_PLAN_NODE: false
VRESULT SINK
MYSQL_PROTOCAL
3:VHASH JOIN(152)
| join op: INNER JOIN(PARTITIONED)[]
| equal join conjunct: (id2[#3] = id[#0])
| cardinality=3
| vec output tuple id: 3
| output tuple id: 3
| vIntermediate tuple ids: 2
| hash output slot ids: 0 1 2 3
| isMarkJoin: false
| final projections: id[#4], id2[#5], id[#6], id2[#7]
| final project output tuple id: 3
| distribute expr lists: id2[#3]
| distribute expr lists: id[#0]
| tuple ids: 1 0
|
|----0:VOlapScanNode(149)
| TABLE: test.test_shuffle_left2(test_shuffle_left2), PREAGGREGATION: ON
| partitions=1/1 (test_shuffle_left2)
| tablets=10/10, tabletList=22038,22040,22042 ...
| cardinality=3, avgRowSize=0.0, numNodes=1
| pushAggOp=NONE
| tuple ids: 0
|
2:VEXCHANGE
offset: 0
distribute expr lists: id[#2]
tuple ids: 1
},
instanceJobs: [
LocalShuffleAssignedJob(
index: 0,
worker: BackendWorker(id: 10095, address: 192.168.126.1:9050),
shareScanIndex: 0,
scanSource: [
{
scanNode: OlapScanNode{id=0, tid=0, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1},
scanRanges: ScanRanges(bytes: 400, ranges: [
tablet 22038, bytes: 0,
tablet 22042, bytes: 0,
tablet 22046, bytes: 0,
tablet 22050, bytes: 400,
tablet 22054, bytes: 0
])
}
]
),
LocalShuffleAssignedJob(
index: 1,
worker: BackendWorker(id: 10096, address: 192.168.126.2:9051),
shareScanIndex: 1,
scanSource: [
{
scanNode: OlapScanNode{id=0, tid=0, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1},
scanRanges: ScanRanges(bytes: 796, ranges: [
tablet 22040, bytes: 397,
tablet 22044, bytes: 0,
tablet 22048, bytes: 399,
tablet 22052, bytes: 0,
tablet 22056, bytes: 0
])
}
]
)
]
)
PipelineDistributedPlan(
id: 1,
parallel: 2,
fragmentJob: UnassignedScanSingleOlapTableJob,
fragment: {
PARTITION: HASH_PARTITIONED: id[#2]
HAS_COLO_PLAN_NODE: false
STREAM DATA SINK
EXCHANGE ID: 02
HASH_PARTITIONED: id2[#3]
1:VOlapScanNode(145)
TABLE: test.test_shuffle_left2(test_shuffle_left2), PREAGGREGATION: ON
partitions=1/1 (test_shuffle_left2)
tablets=10/10, tabletList=22038,22040,22042 ...
cardinality=3, avgRowSize=0.0, numNodes=1
pushAggOp=NONE
tuple ids: 1
},
instanceJobs: [
LocalShuffleAssignedJob(
index: 0,
worker: BackendWorker(id: 10095, address: 192.168.126.1:9050),
shareScanIndex: 0,
scanSource: [
{
scanNode: OlapScanNode{id=1, tid=1, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1},
scanRanges: ScanRanges(bytes: 400, ranges: [
tablet 22038, bytes: 0,
tablet 22042, bytes: 0,
tablet 22046, bytes: 0,
tablet 22050, bytes: 400,
tablet 22054, bytes: 0
])
}
]
),
LocalShuffleAssignedJob(
index: 1,
worker: BackendWorker(id: 10096, address: 192.168.126.2:9051),
shareScanIndex: 1,
scanSource: [
{
scanNode: OlapScanNode{id=1, tid=1, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1},
scanRanges: ScanRanges(bytes: 796, ranges: [
tablet 22040, bytes: 397,
tablet 22044, bytes: 0,
tablet 22048, bytes: 399,
tablet 22052, bytes: 0,
tablet 22056, bytes: 0
])
}
]
)
]
)
Hint log:
Used: [shuffle]_2
UnUsed:
SyntaxError:
```
## TODO
1. extract PipelineScheduler from Coordinator
2. move this framework into cascades and compute cost by dop
3. support StageScheduler, adaptive query execution and DynamicAssignedJob
xy720
pushed a commit
that referenced
this pull request
Jul 1, 2024
… rewrite logic (apache#36623) cherry-pick apache#36161 to branch-2.0 NormalizeAggregate rewrite logic has a bug, for sql like this: SELECT CASE 1 WHEN CAST( NULL AS SIGNED ) THEN NULL WHEN COUNT( DISTINCT CAST( NULL AS SIGNED ) ) THEN NULL ELSE null END ; This is the plan after NormalizeAggregate, the LogicalAggregate only output `count(DISTINCT cast(NULL as SIGNED))`#3, do not output cast(NULL as SIGNED)#2, but the upper project use cast(NULL as SIGNED)#2, so Doris report error "cast(NULL as SIGNED) not in aggregate's output". LogicalResultSink[29] ( outputExprs=[__case_when_0#1] ) +--LogicalProject[26] ( distinct=false, projects=[CASE WHEN (1 = cast(NULL as SIGNED)#2) THEN NULL WHEN (1 = count(DISTINCT cast(NULL as SIGNED))#3) THEN NULL ELSE NULL END AS `CASE WHEN (1 = cast(NULL as SIGNED)) THEN NULL WHEN (1 = count(DISTINCT cast(NULL as SIGNED))) THEN NULL ELSE NULL END`#1], excepts=[] ) +--LogicalAggregate[25] ( groupByExpr=[], outputExpr=[count(DISTINCT cast(NULL as SIGNED)#2) AS `count(DISTINCT cast(NULL as SIGNED))`#3], hasRepeat=false ) +--LogicalProject[24] ( distinct=false, projects=[cast(NULL as SIGNED) AS `cast(NULL as SIGNED)`#2], excepts=[] ) +--LogicalOneRowRelation ( projects=[0 AS `0`#0] ) The problem is that the cast(NULL as SIGNED)#2 should not outputted by LogicalAggregate, cast(NULL as SIGNED) should be computed in LogicalProject. This pr change the upper project projections rewrite logic: aggregateOutputs is rewritten and become the upper-level LogicalProject projections. During the rewriting process, the expressions inside the agg function can be rewritten with expressions in aggregate function arguments and group by expressions, but the ones outside the agg function can only be rewritten with group by expressions. --------- Co-authored-by: moailing <moailing@selectdb.com>
xy720
pushed a commit
that referenced
this pull request
Aug 22, 2024
query like ``` select a, `__DORIS_DELETE_SIGN__` from `test_tbl` WHERE k = '1111111' ``` will not contain project list on top of OlapScanNode, so for short circuit queries, we should use output exprs on top of plan fragment. ``` +----------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | k1[#0] | | k2[#1] | | k3[#2] | | k4[#3] | | k5[#4] | | k6[#5] | | k7[#6] | | k8[#7] | | k9[#8] | | k10[#9] | | k11[#10] | | k12[apache#11] | | new_column0[apache#12] | | new_column1[apache#13] | | __DORIS_DELETE_SIGN__[apache#14] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 1:VEXCHANGE | | offset: 0 | | distribute expr lists: k1[#0], k2[#1], k3[#2] | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: k1[#0], k2[#1], k3[#2] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | UNPARTITIONED | | | | 0:VOlapScanNode(149) | | TABLE: regression_test_serving_p0.tbl_point_query0(tbl_point_query0), PREAGGREGATION: ON | | PREDICATES: (k1[#0] = 1231) AND (k2[#1] = 119291.110000000) AND (k3[#2] = 'ddd') AND (__DORIS_DELETE_SIGN__[apache#14] = 0) | | partitions=1/1 (tbl_point_query0) | | tablets=1/1, tabletList=13203 | | cardinality=0, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | SHORT-CIRCUIT | | | | | | Statistics | | planed with unknown column statistics | +----------------------------------------------------------------------------------------------------------------------------+ ```
xy720
pushed a commit
that referenced
this pull request
Aug 22, 2024
## Proposed changes Currently, Arrow use separate Jemalloc, and use non-standard methods `mallocx`, `sdallocx`, `rallocx` in `memory_pool_jemalloc.cc` to optimize memory allocation. But this may be incompatible with older versions of the Linux kernel. when we use Arrow on Arm Kirin v10 or Centos 7.4, it will get stuck on the Jemalloc Lock, with the stack below, it will appear when calling `arrow::RecordBatch::MakeEmpty`. the kernel version of Arm Kylin v10 is 4.19.90, and the kernel version of Centos 7.4 is 4.14. ``` #0 0x0000ffffae3ceff8 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x0000ffffae3c9b50 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x0000ffffae61c834 in pthread_mutex_lock () from /lib64/libc.so.6 #3 0x0000aaaac99bc1e0 in je_arrow_private_je_malloc_mutex_lock_slow () #4 0x0000aaaac99af3a4 in ?? () #5 0x0000aaaac99b576c in je_arrow_mallocx () #6 0x0000aaaac99a8aec in ?? () #7 0x0000aaaac99a9858 in arrow::AllocateResizableBuffer(long, arrow::MemoryPool*) () #8 0x0000aaaac399f8b8 in arrow::BufferBuilder::Resize(long, bool) () #9 0x0000aaaac983715c in arrow::BaseBinaryBuilder<arrow::BinaryType>::Resize(long) () #10 0x0000aaaac39a47e0 in arrow::BaseBinaryBuilder<arrow::BinaryType>::Append(unsigned char const*, int) () ``` After disable separate Jemalloc when compiling Arrow, the above error disappeared, and Arrow will use the default memory allocator, which is Doris Jemalloc.
xy720
pushed a commit
that referenced
this pull request
Aug 22, 2024
## Proposed changes
- Null pointer of type 'doris::StringRef' in orc reader. The root cause
is error will throw when `num_values == 0` in
`_decode_string_non_dict_encoded_column`.
```
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:9: runtime error: reference binding to null pointer of type 'doris::StringRef'
#0 0x562516fa9770 in std::vector<doris::StringRef, std::allocator<doris::StringRef> >::operator[](unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:2
#1 0x562516fa9770 in doris::Status doris::vectorized::OrcReader::_decode_string_non_dict_encoded_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::EncodedStringVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1164:39
#2 0x562516f9c08b in doris::Status doris::vectorized::OrcReader::_decode_string_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1116:16
#3 0x562516f91d73 in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1357:16
#4 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5
#5 0x562516f9339a in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1402:9
#6 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5
...
```
- Shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka
'__int128') in parquet reader. The root cause is error will throw when
`len == 0`.
```
/root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27: runtime error: shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128')
#0 0x56251760fbc7 in doris::vectorized::parquet::StringToDecimal<doris::vectorized::Decimal128V3, (doris::vectorized::DecimalScaleParams::ScaleType)1>::physical_convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27
#1 0x562517290dc4 in doris::vectorized::parquet::PhysicalToLogicalConverter::convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::TypeDescriptor, std::shared_ptr<doris::vectorized::IDataType const> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, bool) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:209:9
#2 0x562517284a6d in doris::vectorized::ScalarColumnReader::read_column_data(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const>&, doris::vectorized::ColumnSelectVector&, unsigned long, unsigned long*, bool*, bool) /root/doris/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:569:24
#3 0x56251725ae7e in doris::vectorized::RowGroupReader::_read_column_data(doris::vectorized::Block*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned long, unsigned long*, bool*, doris::vectorized::ColumnSelectVector&) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:421:13
#4 0x56251724d6d2 in doris::vectorized::RowGroupReader::next_batch(doris::vectorized::Block*, unsigned long, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:321:9
#5 0x56251708eb97 in doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:530:36
#6 0x56253036772d in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:311:13
#7 0x562530366549 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:253:17
#8 0x5625176e79c8 in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:117:17
#9 0x5625176e6fc1 in doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:84:12
#10 0x562517698047 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:250:5
apache#11 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:172:25
apache#12 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:171:35
...
```
xy720
pushed a commit
that referenced
this pull request
Aug 22, 2024
… StreamLoadContext when operate_txn_2pc (apache#38033) /root/doris/be/src/runtime/stream_load/stream_load_executor.cpp:384:14: runtime error: load of value 3200171710, which is not a valid value for type 'TLoadType::type' #0 0x55be4e2cf808 in doris::StreamLoadExecutor::collect_load_stat(doris::StreamLoadContext*, doris::TTxnCommitAttachment*) /root/doris/be/src/runtime/stream_load/stream_load_executor.cpp:384:14 #1 0x55be4e2d2666 in doris::StreamLoadExecutor::rollback_txn(doris::StreamLoadContext*) /root/doris/be/src/runtime/stream_load/stream_load_executor.cpp:364:9 #2 0x55be80413748 in doris::CloudStreamLoadExecutor::operate_txn_2pc(doris::StreamLoadContext*) /root/doris/be/src/cloud/cloud_stream_load_executor.cpp:80:33 #3 0x55be50cad893 in doris::StreamLoad2PCAction::handle(doris::HttpRequest*) /root/doris/be/src/http/action/stream_load_2pc.cpp:85:49
xy720
pushed a commit
that referenced
this pull request
Aug 22, 2024
…out range of _buf (apache#37936) if we select nested type such as map/array/struct after large string , when string type in mysql_row_buf reserve make buffer size is not large enough , which will lead nested type open_dynamic_mode make _pos pointer out range of mysql_row_buf, then nested type call push_string, and reserve() will make heap_buffer_overflow ``` ==200769==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d0051c12ba at pc 0x55a77788692d bp 0x7fb52f474a30 sp 0x7fb52f4741f8 READ of size 36541 at 0x62d0051c12ba thread T2309 (Pipe_normal [wo) #0 0x55a77788692c in __asan_memcpy (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c1c92c) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f622fa in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:140:5 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a21f16eb in doris::Status doris::vectorized::DataTypeMapSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:410:21 #4 0x55a7a21e4c1e in doris::vectorized::DataTypeMapSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:478:12 #5 0x55a7a22070e6 in doris::Status doris::vectorized::DataTypeNullableSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:300:9 #6 0x55a7a21fbc5e in doris::vectorized::DataTypeNullableSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:317:12 #7 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 #8 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 #9 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 #10 0x55a7c99a4b27 in doris::pipeline::PipelineTask::execute(bool*) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:364:22 apache#11 0x55a7c9a23a2b in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:138:9 apache#12 0x55a7c9a269ca in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:64:9 apache#13 0x55a7c9a2694e in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#14 0x55a7c9a268ae in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#15 0x55a7c9a26635 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#16 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#17 0x55a77e95ec94 in doris::FunctionRunnable::run() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:48:27 apache#18 0x55a77e941015 in doris::ThreadPool::dispatch_thread() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:543:24 apache#19 0x55a77e97eb23 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 apache#20 0x55a77e97e928 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 apache#21 0x55a77e97e860 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11 apache#22 0x55a77e97e655 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17 apache#23 0x55a77e97e54e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#24 0x55a77e97e48e in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#25 0x55a77e97dd85 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#26 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#27 0x55a77e8fb841 in doris::Thread::supervise_thread(void*) /mnt/disk1/wangqiannan/amory/doris/be/src/util/thread.cpp:498:5 apache#28 0x7fc1c3a111c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) apache#29 0x7fc1c4400e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) 0x62d0051c12ba is located 0 bytes after 36538-byte region [0x62d0051b8400,0x62d0051c12ba) allocated by thread T2309 (Pipe_normal [wo) here: #0 0x55a7778c20bd in operator new[](unsigned long) (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c580bd) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f621c1 in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:137:21 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a1fd0d75 in doris::Status doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:260:16 #4 0x55a7a1fccc1e in doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:215:16 #5 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 #6 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 #7 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
xy720
pushed a commit
that referenced
this pull request
Aug 22, 2024
…on in cloud mode (apache#37293)" (apache#38828) We have to figure out why it causes a SEGV when running cloud_p0 later ``` #0 doris::cloud::TabletCompactionJobPB::_internal_input_versions(int) const /root/doris/cloud/../gensrc/build/gen_cpp/cloud.pb.h:48193:33 #1 doris::cloud::MetaServiceImpl::start_tablet_job(google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*) /root/doris/cloud/src/meta-service/meta_service_job.cpp:436:9 #2 void doris::cloud::MetaServiceProxy::call_impl<doris::cloud::StartTabletJobRequest, doris::cloud::StartTabletJobResponse>(void (doris::cloud::MetaService::*)(google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*), google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*) /root/doris/cloud/src/meta-service/meta_service.h:684:13 #3 doris::cloud::MetaServiceProxy::start_tablet_job(google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*) /root/doris/cloud/src/meta-service/meta_service.h:478:9 #4 doris::cloud::MetaService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*) /root/doris/gensrc/build/gen_cpp/cloud.pb.cc:0:7 ``` This PR also add some FE log
xy720
pushed a commit
that referenced
this pull request
Jul 23, 2025
### What problem does this PR solve? ### Problem after enable java support, be can not start correctly, it will hang on stack: ``` (gdb) bt #0 0x00007f5fb1e97ce6 in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f5fb1e9a798 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f5fb2c98bf3 in os::PlatformEvent::park() () from /usr/lib/jvm/java-17//lib/server/libjvm.so #3 0x00007f5fb2c693a5 in ObjectMonitor::wait(long, bool, JavaThread*) () from /usr/lib/jvm/java-17//lib/server/libjvm.so #4 0x00007f5fb2e8b316 in ObjectSynchronizer::wait(Handle, long, JavaThread*) () from /usr/lib/jvm/java-17//lib/server/libjvm.so #5 0x00007f5fb2934a97 in JVM_MonitorWait () from /usr/lib/jvm/java-17//lib/server/libjvm.so #6 0x00007f5f9de245ba in ?? () #7 0x00007f5f49446158 in ?? () #8 0x00007f5faeb9fa00 in ?? () #9 0x00007ffc37f91178 in ?? () #10 0x00007f5f9de304bd in ?? () apache#11 0x00007ffc37f910a0 in ?? () apache#12 0x0000000000000000 in ?? () ``` jstack of be: ``` "main" #1 prio=5 os_prio=0 cpu=931.38ms elapsed=66.08s tid=0x00007fab8e12c400 nid=0x3e68aa in Object.wait() [0x00007ffdb3d4c000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@17.0.15-ga/Native Method) - waiting on <0x00000000bd183b68> (a java.lang.ProcessImpl) at java.lang.Object.wait(java.base@17.0.15-ga/Object.java:338) at java.lang.ProcessImpl.waitFor(java.base@17.0.15-ga/ProcessImpl.java:434) - locked <0x00000000bd183b68> (a java.lang.ProcessImpl) at org.apache.hadoop.util.Shell.runCommand(Shell.java:1061) at org.apache.hadoop.util.Shell.run(Shell.java:957) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1282) at org.apache.hadoop.util.Shell.isSetsidSupported(Shell.java:853) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:838) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1713) at org.apache.hadoop.security.SecurityUtil.setConfigurationInternal(SecurityUtil.java:103) at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:92) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:312) - locked <0x00000000bd5fd708> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:300) - locked <0x00000000bd5fd708> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:575) at org.apache.hadoop.fs.viewfs.ViewFileSystem.<init>(ViewFileSystem.java:279) at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@17.0.15-ga/Native Method) at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(java.base@17.0.15-ga/NativeConstructorAccessorImpl.java:77) at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@17.0.15-ga/DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstanceWithCaller(java.base@17.0.15-ga/Constructor.java:500) at java.lang.reflect.Constructor.newInstance(java.base@17.0.15-ga/Constructor.java:481) at java.util.ServiceLoader$ProviderImpl.newInstance(java.base@17.0.15-ga/ServiceLoader.java:789) at java.util.ServiceLoader$ProviderImpl.get(java.base@17.0.15-ga/ServiceLoader.java:729) at java.util.ServiceLoader$3.next(java.base@17.0.15-ga/ServiceLoader.java:1403) at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3534) - locked <0x00000000826529f0> (a java.lang.Class for org.apache.hadoop.fs.FileSystem) ``` This problem comes from: apache#45287 after this fix, we could enable java support: apache#52412 ### Another Fix Method Add JAVA_OPTS `-Djdk.lang.processReaperUseDefaultStackSize=true` inside be.conf, also can fix this problem: https://bugs.openjdk.org/browse/JDK-8153057 From gemini: 
xy720
pushed a commit
that referenced
this pull request
Jul 23, 2025
… leaks in MemTable (apache#52902) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Direct leak of 193504 byte(s) in 6047 object(s) allocated from: #0 0x559f1d2a45fd in operator new(unsigned long) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x2a0015fd) #1 0x559f1f54c41d in doris::MemTable::insert(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/memtable.cpp:243:38 #2 0x559f1f597c08 in doris::MemTableWriter::write(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/memtable_writer.cpp:107:27 #3 0x559f467a48c0 in doris::DeltaWriterV2::write(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/delta_writer_v2.cpp:164:30 #4 0x559f4711e2f5 in doris::vectorized::VTabletWriterV2::_write_memtable(std::shared_ptr<doris::vectorized::Block>, long, doris::vectorized::Rows const&) /root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:552:24 #5 0x559f4711b4bf in doris::vectorized::VTabletWriterV2::write(doris::RuntimeState*, doris::vectorized::Block&) /root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:486:9 #6 0x559f4702bb4b in doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*, doris::RuntimeProfile*) /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:169:23 #7 0x559f4702e7e8 in doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0::operator()() const /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:103:5
xy720
pushed a commit
that referenced
this pull request
Jul 23, 2025
…rflow error (apache#53018) ### What problem does this PR solve? Problem Summary: ``` ==2403213==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x506001097220 at pc 0x55837edd7fe9 bp 0x7f7efe2dc830 sp 0x7f7efe2dbff8 READ of size 63 at 0x506001097220 thread T1262 (brpc_light) #0 0x55837edd7fe8 in strlen (/mnt/disk8/zhangsida/doris/output/be/lib/doris_be+0x2c8d7fe8) (BuildId: 5acbe48773972a88) #1 0x5583bd3ff969 in fmt::v7::detail::buffer_appender<char> fmt::v7::detail::write<char, fmt::v7::detail::buffer_appender<char>>(fmt::v7::detail::buffer_appender<char>, char const*) (/mnt/disk8/zhangsida/doris/output/be/lib/doris_be+0x6aeff969) (BuildId: 5acbe48773972a88) #2 0x5583bd41640e in char const* fmt::v7::detail::parse_replacement_field<char, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char>>&>(char const*, char const*, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char>>&) (/mnt/disk8/zhangsida/doris/output/be/lib/doris_be+0x6af1640e) (BuildId: 5acbe48773972a88) #3 0x5583bd416c7f in void fmt::v7::detail::vformat_to<char>(fmt::v7::detail::buffer<char>&, fmt::v7::basic_string_view<char>, fmt::v7::basic_format_args<fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<fmt::v7::type_identity<char>::type>, fmt::v7::type_identity<char>::type>>, fmt::v7::detail::locale_ref) (/mnt/disk8/zhangsida/doris/output/be/lib/doris_be+0x6af16c7f) (BuildId: 5acbe48773972a88) #4 0x5583bd3f74aa in fmt::v7::detail::vformat[abi:cxx11](fmt::v7::basic_string_view<char>, fmt::v7::format_args) (/mnt/disk8/zhangsida/doris/output/be/lib/doris_be+0x6aef74aa) (BuildId: 5acbe48773972a88) #5 0x5583a6b4d3bb in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> fmt::v7::format<char [45], char const*&, char>(char const (&) [45], char const*&) /mnt/disk8/zhangsida/doris/thirdparty/installed/include/fmt/core.h:2076:10 #6 0x5583a6b4d3bb in doris::vectorized::BitmapFromBase64::vector(doris::vectorized::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::NoTrackingDefaultMemoryAllocator>, 16ul, 15ul> const&, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::NoTrackingDefaultMemoryAllocator>, 16ul, 15ul> const&, std::vector<doris::BitmapValue, std::allocator<doris::BitmapValue>>&, doris::vectorized::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::NoTrackingDefaultMemoryAllocator>, 16ul, 15ul>&, unsigned long) /mnt/disk8/zhangsida/doris/be/src/vec/functions/function_bitmap.cpp:310:29 #7 0x5583a6b4b93b in doris::vectorized::FunctionBitmapAlwaysNull<doris::vectorized::BitmapFromBase64>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function_bitmap.cpp:381:13 #8 0x5583a02062ef in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.h:459:26 #9 0x5583a49d77e3 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.cpp #10 0x5583a49d0bf8 in doris::vectorized::PreparedFunctionImpl::default_implementation_for_constant_arguments(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool, bool*) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.cpp:168:5 apache#11 0x5583a49d3324 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.cpp:237:5 ```
xy720
pushed a commit
that referenced
this pull request
Jul 23, 2025
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
1.fix LRU queue crash use after free
2.fix extra LRU queue info when 'need_to_move' flag unset
3.use concurrent queueu to record queueu change info for thread safety
```
ERROR: AddressSanitizer: heap-use-after-free on address 0x603005548c40 at pc 0x55f28e8c4785 bp 0x7f603582e1f0 sp 0x7f603582e1e8
READ of size 8 at 0x603005548c40 thread T201
#0 0x55f28e8c4784 in std::_Head_base<0ul, doris::io::CacheLRULog*, false>::_Head_base<doris::io::CacheLRULog*>(doris::io::CacheLRULog*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:190:17
#1 0x55f28e8c4784 in std::_Tuple_impl<0ul, doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>::_Tuple_impl(std::_Tuple_impl<0ul, doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:292:2
#2 0x55f28e8c4784 in std::tuple<doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>::tuple(std::tuple<doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:1079:17
#3 0x55f28e8c4784 in std::_uniq_ptr_impl<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>::uniq_ptr_impl(std::_uniq_ptr_impl<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:162:9
#4 0x55f28e8c4784 in std::_uniq_ptr_data<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>, true, true>::uniq_ptr_data(std::_uniq_ptr_data<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>, true, true>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:211:7
#5 0x55f28e8c4784 in std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>::unique_ptr(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:327:7
#6 0x55f28e8c4784 in doris::io::LRUQueueRecorder::replay_queue_event(doris::io::FileCacheType) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:40:20
#7 0x55f28e82d620 in doris::io::BlockFileCache::run_background_lru_log_replay() /root/doris/be/src/io/cache/block_file_cache.cpp:2242:24
#8 0x55f2cdc2720f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc+-v3/src/c11/../../../../../libstdc-v3/src/c+11/thread.cc:82:18
#9 0x7f61f1842608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
#10 0x7f61f1aef132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
0x603005548c40 is located 16 bytes inside of 24-byte region [0x603005548c30,0x603005548c48)
freed by thread T201 here:
#0 0x55f28e51680d in operator delete(void*) (/home/work/unlimit_teamcity/TeamCity/Agents/20250708205944agent_172.16.0.48_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x3975a80d) (BuildId: 8b6ba6101e736655)
#1 0x55f28e8c3ce0 in std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::pop_front() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1198:15
#2 0x55f28e8c3ce0 in doris::io::LRUQueueRecorder::replay_queue_event(doris::io::FileCacheType) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:41:19
#3 0x55f28e82d620 in doris::io::BlockFileCache::run_background_lru_log_replay() /root/doris/be/src/io/cache/block_file_cache.cpp:2242:24
#4 0x55f2cdc2720f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc+-v3/src/c11/../../../../../libstdc-v3/src/c+11/thread.cc:82:18
previously allocated by thread T607 (CumuCompactionT) here:
#0 0x55f28e515fad in operator new(unsigned long) (/home/work/unlimit_teamcity/TeamCity/Agents/20250708205944agent_172.16.0.48_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x39759fad) (BuildId: 8b6ba6101e736655)
#1 0x55f28e8c660d in __gnu_cxx::new_allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::allocate(unsigned long, void const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:121:27
#2 0x55f28e8c660d in std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::allocate(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:173:32
#3 0x55f28e8c660d in std::allocator_traits<std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>>::allocate(std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>&, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:460:20
#4 0x55f28e8c660d in std::__cxx11::_List_base<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_get_node() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:442:16
#5 0x55f28e8c660d in std::List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>* std::_cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_create_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:634:21
#6 0x55f28e8c660d in void std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_insert<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>(std::_List_iterator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>, std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1911:18
#7 0x55f28e8c3522 in std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::push_back(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1217:15
#8 0x55f28e8c3522 in doris::io::LRUQueueRecorder::record_queue_event(doris::io::FileCacheType, doris::io::CacheLRULogType, doris::io::UInt128Wrapper, unsigned long, unsigned long) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:29:15
#9 0x55f28e82f09b in doris::io::BlockFileCache::use_cell(doris::io::BlockFileCache::FileBlockCell const&, std::__cxx11::list<std::shared_ptr<doris::io::FileBlock>, std::allocator<std::shared_ptr<doris::io::FileBlock>>>*, bool, std::lock_guard<std::mutex>&) /root/doris/be/src/io/cache/block_file_cache.cpp:380:20
#10 0x55f28e833d1b in doris::io::BlockFileCache::get_impl[abi:cxx11](doris::io::UInt128Wrapper const&, doris::io::CacheContext const&, doris::io::FileBlock::Range const&, std::lock_guard<std::mutex>&) /root/doris/be/src/io/cache/block_file_cache.cpp:572:13
apache#11 0x55f28e83b4ef in doris::io::BlockFileCache::get_or_set(doris::io::UInt128Wrapper const&, unsigned long, unsigned long, doris::io::CacheContext&) /root/doris/be/src/io/cache/block_file_cache.cpp:762:27
apache#12 0x55f28e7ffcee in doris::io::CachedRemoteFileReader::read_at_impl(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /root/doris/be/src/io/cache/cached_remote_file_reader.cpp:191:21
apache#13 0x55f28e7f8017 in doris::io::FileReader::read_at(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /root/doris/be/src/io/fs/file_reader.cpp:34:17
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---------
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
xy720
pushed a commit
that referenced
this pull request
Jul 23, 2025
==15185==ERROR: AddressSanitizer: heap-use-after-free on address 0x7da44ced96d0 at pc 0x55795dac5f46 bp 0x7bc26b9a3e50 sp 0x7bc26b9a3e48 READ of size 8 at 0x7da44ced96d0 thread T1357 #0 0x55795dac5f45 in std::_Function_base::_M_empty() const /usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:249:37 #1 0x55795dac5f45 in std::function<doris::Status ()>::operator()() const /usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:591:6 #2 0x55795dac5f45 in doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status ()>, std::allocator<std::function<doris::Status ()>>> const&, int)::$_0::operator()() const /root/doris/be/src/cloud/cloud_meta_mgr.cpp:106:23 #3 0x55795dac5f45 in void std::_invoke_impl<void, doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status ()>, std::allocator<std::function<doris::Status ()>>> const&, int)::$_0&>(std::_invoke_other, doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status ()>, std::allocator<std::function<doris::Status ()>>> const&, int)::$_0&) /usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 #4 0x55795dac5f45 in std::enable_if<is_invocable_r_v<void, doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status ()>, std::allocator<std::function<doris::Status ()>>> const&, int)::$0&>, void>::type std::_invoke_r<void, doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status ()>, std::allocator<std::function<doris::Status ()>>> const&, int)::$_0&>(doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status ()>, std::allocator<std::function<doris::Status ()>>> const&, int)::$_0&) /usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113:2 #5 0x55795dac5f45 in std::_Function_handler<void (), doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status ()>, std::allocator<std::function<doris::Status ()>>> const&, int)::$_0>::_M_invoke(std::_Any_data const&) /usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 #6 0x55795da78d1d in std::function<void ()>::operator()() const /usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 #7 0x55795da78d1d in doris::cloud::run_bthread_work(void*) /root/doris/be/src/cloud/cloud_meta_mgr.cpp:74:5 #8 0x55795f3da334 in bthread::TaskGroup::task_runner(long) (/home/work/unlimit_teamcity/TeamCity/Agents/20250718121603agent_172.16.0.202_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x50a4e334) #9 0x55795f3ccc80 in bthread_make_fcontext (/home/work/unlimit_teamcity/TeamCity/Agents/20250718121603agent_172.16.0.202_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x50a40c80) 0x7da44ced96d0 is located 2640 bytes inside of 2688-byte region [0x7da44ced8c80,0x7da44ced9700) freed by thread T1431 here: #0 0x5579380e1f12 in operator delete(void*, unsigned long) (/home/work/unlimit_teamcity/TeamCity/Agents/20250718121603agent_172.16.0.202_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x29755f12) previously allocated by thread T795 (brpc_light) here: #0 0x5579380e12ad in operator new(unsigned long) (/home/work/unlimit_teamcity/TeamCity/Agents/20250718121603agent_172.16.0.202_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x297552ad) #1 0x55795c7b57ab in std::__new_allocator<std::function<doris::Status ()>>::allocate(unsigned long, void const*) /usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/new_allocator.h:151:27 #2 0x55795c7b57ab in std::allocator<std::function<doris::Status ()>>::allocate(unsigned long) /usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/allocator.h:203:32 #3 0x55795c7b57ab in std::allocator_traits<std::allocator<std::function<doris::Status ()>>>::allocate(std::allocator<std::function<doris::Status ()>>&, unsigned long) /usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/alloc_traits.h:614:20 #4 0x55795c7b57ab in std::_Vector_base<std::function<doris::Status ()>, std::allocator<std::function<doris::Status ()>>>::_M_allocate(unsigned long) /usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_vector.h:387:20 #5 0x55795c7b57ab in std::vector<std::function<doris::Status ()>, std::allocator<std::function<doris::Status ()>>>::reserve(unsigned long) /usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/vector.tcc:79:22 #6 0x55795c7787f6 in doris::pipeline::OlapScanLocalState::_sync_cloud_tablets(doris::RuntimeState*) /root/doris/be/src/pipeline/exec/olap_scan_operator.cpp:466:20 #7 0x55795c777e7d in doris::pipeline::OlapScanLocalState::init(doris::RuntimeState*, doris::pipeline::LocalStateInfo&) /root/doris/be/src/pipeline/exec/olap_scan_operator.cpp:51:5 #8 0x55795b459bf9 in doris::pipeline::OperatorX<doris::pipeline::OlapScanLocalState>::setup_local_state(doris::RuntimeState*, doris::pipeline::LocalStateInfo&) /root/doris/be/src/pipeline/exec/operator.cpp:486:5 #9 0x55795d9c9a36 in doris::pipeline::PipelineTask::prepare(std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams>> const&, int, doris::TDataSink const&) /root/doris/be/src/pipeline/pipeline_task.cpp:118:9
xy720
pushed a commit
that referenced
this pull request
Jul 27, 2025
…#53713) ### What problem does this PR solve? Problem Summary: memcpy(dst, &result, sizeof(result)); when use memcpy, it's size if sizeof(result), so use int64 maybe overflow of dst ``` ==3524968==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f18404e1d73 at pc 0x559e2c162b01 bp 0x7f18439e5dc0 sp 0x7f18439e5db8 WRITE of size 8 at 0x7f18404e1d73 thread T1265 (brpc_light) #0 0x559e2c162b00 in bool doris::vectorized::parse_ipv4<char const, doris::vectorized::parse_ipv6(char const*, char const*, unsigned char*)::'lambda'()>(char const*&, doris::vectorized::parse_ipv6(char const*, char const*, unsigned char*)::'lambda'(), unsigned char*, long) /mnt/disk8/zhangsida/doris/be/src/vec/common/format_ip.h:165:5 #1 0x559e2c161eb3 in bool doris::vectorized::parse_ipv6<char const, doris::vectorized::parse_ipv6(char const*, char const*, unsigned char*)::'lambda'()>(char const*&, doris::vectorized::parse_ipv6(char const*, char const*, unsigned char*)::'lambda'(), unsigned char*, int) /mnt/disk8/zhangsida/doris/be/src/vec/common/format_ip.h:416:18 #2 0x559e2c160c44 in doris::vectorized::parse_ipv6(char const*, char const*, unsigned char*) /mnt/disk8/zhangsida/doris/be/src/vec/common/format_ip.h:467:9 #3 0x559e2c160c44 in doris::vectorized::parse_ipv6_whole(char const*, char const*, unsigned char*) /mnt/disk8/zhangsida/doris/be/src/vec/common/format_ip.h:475:12 #4 0x559e2c160c44 in doris::IPv6Value::from_string(unsigned __int128&, char const*, unsigned long) /mnt/disk8/zhangsida/doris/be/src/vec/runtime/ipv6_value.h:71:16 #5 0x559e4fdb05f3 in doris::vectorized::FunctionToIP<(doris::vectorized::IPConvertExceptionMode)0, (doris::PrimitiveType)37>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function_ip.h:1180:21 #6 0x559e4c233b1e in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.h:447:26 #7 0x559e4eebcef3 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.cpp #8 0x559e4eeb68c4 in doris::vectorized::PreparedFunctionImpl::default_implementation_for_constant_arguments(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool, bool*) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.cpp:168:5 #9 0x559e4eeb8fc4 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.cpp:237:5 ```
xy720
pushed a commit
that referenced
this pull request
Jul 30, 2025
### What problem does this PR solve? set NDEBUG_SANITIZER on asan mode ``` AddressSanitizer:DEADLYSIGNAL 18:05:08 ================================================================= 18:05:08 ==17515==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x563cea4a925f bp 0x7ffedd655da0 sp 0x7ffedd655d90 T0) 18:05:08 ==17515==The signal is caused by a READ memory access. 18:05:08 ==17515==Hint: this fault was caused by a dereference of a high value address (see register values below). Disassemble the provided pc to learn which register was used. 18:05:17 #0 0x563cea4a925f in absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>>::parent() const /var/local/thirdparty/installed/include/absl/container/internal/btree.h:714:39 18:05:17 #1 0x563cea4a9216 in absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>>::is_root() const /var/local/thirdparty/installed/include/absl/container/internal/btree.h:718:33 18:05:17 #2 0x563cea4a9fae in absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>>::get_root_generation() const /var/local/thirdparty/installed/include/absl/container/internal/btree.h:729:19 18:05:17 #3 0x563cea4a9b86 in absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>>::generation() const /var/local/thirdparty/installed/include/absl/container/internal/btree.h:735:41 18:05:17 #4 0x563cea4a90bb in void absl::lts_20250512::container_internal::btree_iterator_generation_info_enabled::assert_valid_generation<absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>>>(absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>> const*) const /var/local/thirdparty/installed/include/absl/container/internal/btree.h:1077:34 18:05:17 #5 0x563cea4b9771 in absl::lts_20250512::container_internal::btree_iterator<absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>> const, std::pair<S2CellId const, S2ShapeIndexCell*> const&, std::pair<S2CellId const, S2ShapeIndexCell*> const*>::increment() /var/local/thirdparty/installed/include/absl/container/internal/btree.h:1296:5 18:05:17 #6 0x563cea4b94fa in absl::lts_20250512::container_internal::btree_iterator<absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>> const, std::pair<S2CellId const, S2ShapeIndexCell*> const&, std::pair<S2CellId const, S2ShapeIndexCell*> const*>::operator++() /var/local/thirdparty/installed/include/absl/container/internal/btree.h:1215:5 18:05:17 #7 0x563cea4a7dbd in MutableS2ShapeIndex::Iterator::Next() /var/local/thirdparty/installed/include/s2/mutable_s2shape_index.h:720:3 18:05:17 #8 0x563d0901dada (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4b466ada) 18:05:17 #9 0x563d0901eeac in s2shapeutil::FindSelfIntersection(S2ShapeIndex const&, S2Error*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4b467eac) 18:05:17 #10 0x563d08fa4fbf in S2Loop::IsValid() const (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4b3edfbf) 18:05:17 apache#11 0x563cea498899 in doris::to_s2loop(doris::GeoCoordinateList const&, std::unique_ptr<S2Loop, std::default_delete<S2Loop>>*) /root/doris/be/src/geo/geo_types.cpp:340:19 18:05:17 apache#12 0x563cea4891cf in doris::to_s2polygon(doris::GeoCoordinateListList const&, std::unique_ptr<S2Polygon, std::default_delete<S2Polygon>>*) /root/doris/be/src/geo/geo_types.cpp:374:20 18:05:17 apache#13 0x563cea488ee4 in doris::GeoPolygon::from_coords(doris::GeoCoordinateListList const&) /root/doris/be/src/geo/geo_types.cpp:786:12 18:05:17 apache#14 0x563cea46d9be in wkt_parse(WktParseContext*) /root/doris/be/src/geo/wkt_yacc.y:130:38 18:05:17 apache#15 0x563cea45fefc in doris::WktParse::parse_wkt(char const*, unsigned long, doris::GeoShape**) /root/doris/be/src/geo/wkt_parse.cpp:37:16 18:05:17 apache#16 0x563cea4814c2 in doris::GeoShape::from_wkt(char const*, unsigned long, doris::GeoParseStatus*) /root/doris/be/src/geo/geo_types.cpp:407:15 18:05:17 apache#17 0x563cda7df4ed in doris::GeoTypesTest_point_intersects_Test::TestBody() /root/doris/be/test/geo/geo_types_test.cpp:108:13 18:05:17 apache#18 0x563d07bf5023 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a03e023) 18:05:17 apache#19 0x563d07be4245 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a02d245) 18:05:17 apache#20 0x563d07bbed66 in testing::Test::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a007d66) 18:05:17 apache#21 0x563d07bbfa4c in testing::TestInfo::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a008a4c) 18:05:17 apache#22 0x563d07bc01df in testing::TestSuite::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a0091df) 18:05:17 apache#23 0x563d07bd199b in testing::internal::UnitTestImpl::RunAllTests() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a01a99b) 18:05:17 apache#24 0x563d07bf6e53 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a03fe53) 18:05:17 apache#25 0x563d07be6465 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a02f465) 18:05:17 apache#26 0x563d07bd152a in testing::UnitTest::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a01a52a) 18:05:17 apache#27 0x563cdd9d3322 in RUN_ALL_TESTS() /var/local/thirdparty/installed/include/gtest/gtest.h:2490:46 18:05:17 apache#28 0x563cdd9c689e in main /root/doris/be/test/testutil/run_all_tests.cpp:111:19 18:05:17 apache#29 0x7f73ae7a4554 in __libc_start_main (/lib64/libc.so.6+0x22554) (BuildId: 1a8fb61bb4614a483833d5334202ab50edda2a25) 18:05:17 apache#30 0x563cd9b0e029 in _start (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x1bf57029) ```
xy720
pushed a commit
that referenced
this pull request
Jul 30, 2025
) ### What problem does this PR solve? In branch-2.0, we already refractor these codes in pr: apache#18590 Deadlock stack: 1、While load, we alloc MemTracker and need lock TrackerGroup.group_lock ``` NodeChannel::NodeChannel _node_channel_tracker = std::make_shared<MemTracker> MemTracker::bind_parent std::lock_guard<std::mutex> l(mem_tracker_pool[_parent_group_num].group_lock); _tracker_group_it = mem_tracker_pool[_parent_group_num].trackers.insert( mem_tracker_pool[_parent_group_num].trackers.end(), this); ``` 2、but while we try to call std::list::insert, we need alloc std::_List_node，here new_hook (in file tcmalloc_hook.h) is triggered, then we lock the same TrackerGroup.group_lock, make it deadlock ``` new_hook doris::ThreadMemTrackerMgr::consume doris::ThreadMemTrackerMgr::flush_untracked_mem<true, true> doris::ThreadMemTrackerMgr::exceeded doris::MemTrackerLimiter::print_log_usage doris::MemTracker::make_group_snapshot std::lock_guard<std::mutex> l(mem_tracker_pool[group_num].group_lock); ``` Full stack info: ``` (gdb) bt #0 0x00007f219772454d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f219771fe9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007f219771fd68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000563ca7a8a824 in __gthread_mutex_lock (__mutex=0x563cb18cbc58) at /var/local/ldb-toolchain/include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:749 #4 std::mutex::lock (this=0x563cb18cbc58) at /var/local/ldb-toolchain/include/c++/11/bits/std_mutex.h:100 #5 std::lock_guard<std::mutex>::lock_guard (__m=..., this=<synthetic pointer>) at /var/local/ldb-toolchain/include/c++/11/bits/std_mutex.h:229 #6 doris::MemTracker::make_group_snapshot (snapshots=0x7f1f7fe06ea0, group_num=<optimized out>, parent_label=...) at /data/TCHouse-D-1.2/be/src/runtime/memory/mem_tracker.cpp:115 #7 0x0000563ca7a7eeb4 in doris::MemTrackerLimiter::print_log_usage (this=0x5640ab7956c0, msg=...) at /data/TCHouse-D-1.2/be/src/runtime/memory/mem_tracker_limiter.cpp:198 #8 0x0000563ca7a8d1e4 in doris::ThreadMemTrackerMgr::exceeded (this=this@entry=0x563ce6a2c820, size=1048584) at /data/TCHouse-D-1.2/be/src/runtime/memory/thread_mem_tracker_mgr.cpp:59 #9 0x0000563ca78cfeb4 in doris::ThreadMemTrackerMgr::flush_untracked_mem<true, true> (this=0x563ce6a2c820) at /data/TCHouse-D-1.2/be/src/runtime/memory/thread_mem_tracker_mgr.h:223 #10 doris::ThreadMemTrackerMgr::consume (size=<optimized out>, this=0x563ce6a2c820) at /data/TCHouse-D-1.2/be/src/runtime/memory/thread_mem_tracker_mgr.h:188 apache#11 doris::ThreadMemTrackerMgr::consume (size=<optimized out>, this=0x563ce6a2c820) at /data/TCHouse-D-1.2/be/src/runtime/memory/thread_mem_tracker_mgr.h:178 apache#12 new_hook (ptr=<optimized out>, size=24) at /data/TCHouse-D-1.2/be/src/runtime/memory/tcmalloc_hook.h:39 apache#13 0x0000563caf3dfa78 in MallocHook::InvokeNewHookSlow (p=p@entry=0x56422d713b40, s=s@entry=24) at src/malloc_hook.cc:498 apache#14 0x0000563caf55f2c1 in MallocHook::InvokeNewHook (s=24, p=0x56422d713b40) at src/malloc_hook-inl.h:127 apache#15 tcmalloc::do_allocate_full<tcmalloc::cpp_throw_oom> (size=size@entry=24) at src/tcmalloc.cc:1805 apache#16 tcmalloc::allocate_full_cpp_throw_oom (size=size@entry=24) at src/tcmalloc.cc:1815 apache#17 0x0000563caf55f429 in tcmalloc::dispatch_allocate_full<tcmalloc::cpp_throw_oom> (size=24) at src/tcmalloc.cc:1822 apache#18 0x0000563ca7a8ab9a in __gnu_cxx::new_allocator<std::_List_node<doris::MemTracker*> >::allocate (__n=1, this=0x563cb18cbc40) at /var/local/ldb-toolchain/include/c++/11/ext/new_allocator.h:103 apache#19 std::allocator_traits<std::allocator<std::_List_node<doris::MemTracker*> > >::allocate (__n=1, __a=...) at /var/local/ldb-toolchain/include/c++/11/bits/alloc_traits.h:460 apache#20 std::__cxx11::_List_base<doris::MemTracker*, std::allocator<doris::MemTracker*> >::_M_get_node (this=0x563cb18cbc40) at /var/local/ldb-toolchain/include/c++/11/bits/stl_list.h:442 apache#21 std::__cxx11::list<doris::MemTracker*, std::allocator<doris::MemTracker*> >::_M_create_node<doris::MemTracker*> (this=0x563cb18cbc40) at /var/local/ldb-toolchain/include/c++/11/bits/stl_list.h:634 apache#22 std::__cxx11::list<doris::MemTracker*, std::allocator<doris::MemTracker*> >::emplace<doris::MemTracker*> (__position=..., this=0x563cb18cbc40) at /var/local/ldb-toolchain/include/c++/11/bits/list.tcc:92 apache#23 std::__cxx11::list<doris::MemTracker*, std::allocator<doris::MemTracker*> >::insert (__x=<optimized out>, __position=..., this=0x563cb18cbc40) at /var/local/ldb-toolchain/include/c++/11/bits/stl_list.h:1309 apache#24 doris::MemTracker::bind_parent (this=0x563eff46ad10, parent=<optimized out>) at /data/TCHouse-D-1.2/be/src/runtime/memory/mem_tracker.cpp:79 apache#25 0x0000563ca7a8b608 in doris::MemTracker::MemTracker (this=this@entry=0x563eff46ad10, label=..., parent=parent@entry=0x0) at /data/TCHouse-D-1.2/be/src/runtime/memory/mem_tracker.cpp:66 apache#26 0x0000563ca7967467 in __gnu_cxx::new_allocator<doris::MemTracker>::construct<doris::MemTracker, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__p=0x563eff46ad10, this=<optimized out>) at /var/local/ldb-toolchain/include/c++/11/ext/new_allocator.h:154 apache#27 std::allocator_traits<std::allocator<doris::MemTracker> >::construct<doris::MemTracker, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__p=0x563eff46ad10, __a=...) at /var/local/ldb-toolchain/include/c++/11/bits/alloc_traits.h:512 apache#28 std::_Sp_counted_ptr_inplace<doris::MemTracker, std::allocator<doris::MemTracker>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__a=..., this=0x563eff46ad00) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr_base.h:519 apache#29 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<doris::MemTracker, std::allocator<doris::MemTracker>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__a=..., __p=<optimized out>, this=<optimized out>) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr_base.h:650 apache#30 std::__shared_ptr<doris::MemTracker, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<doris::MemTracker>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__tag=..., this=<optimized out>) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr_base.h:1337 apache#31 std::shared_ptr<doris::MemTracker>::shared_ptr<std::allocator<doris::MemTracker>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__tag=..., this=<optimized out>) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr.h:409 apache#32 std::allocate_shared<doris::MemTracker, std::allocator<doris::MemTracker>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__a=...) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr.h:861 apache#33 std::make_shared<doris::MemTracker, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > () at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr.h:877 apache#34 doris::stream_load::NodeChannel::NodeChannel (this=this@entry=0x563d0ca7e110, parent=<optimized out>, index_channel=index_channel@entry=0x563d4b43f200, node_id=<optimized out>) at /data/TCHouse-D-1.2/be/src/exec/tablet_sink.cpp:49 apache#35 0x0000563cac74a82d in doris::stream_load::VNodeChannel::VNodeChannel (this=this@entry=0x563d0ca7e110, parent=<optimized out>, index_channel=index_channel@entry=0x563d4b43f200, node_id=<optimized out>) at /data/TCHouse-D-1.2/be/src/vec/sink/vtablet_sink.cpp:37 apache#36 0x0000563ca7967eaa in __gnu_cxx::new_allocator<doris::stream_load::VNodeChannel>::construct<doris::stream_load::VNodeChannel, doris::stream_load::OlapTableSink*&, doris::stream_load::IndexChannel*, long&> (__p=0x563d0ca7e110, this=<optimized out>) at /var/local/ldb-toolchain/include/c++/11/ext/new_allocator.h:154 apache#37 std::allocator_traits<std::allocator<doris::stream_load::VNodeChannel> >::construct<doris::stream_load::VNodeChannel, doris::stream_load::OlapTableSink*&, doris::stream_load::IndexChannel*, long&> (__p=0x563d0ca7e110, __a=...) at /var/local/ldb-toolchain/include/c++/11/bits/alloc_traits.h:512 apache#38 std::_Sp_counted_ptr_inplace<doris::stream_load::VNodeChannel, std::allocator<doris::stream_load::VNodeChannel>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<doris::stream_load::OlapTableSink*&, doris::stream_load::IndexChannel*, long&> (__a=..., this=0x563d0ca7e100) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr_base.h:519 apache#39 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<doris::stream_load::VNodeChannel, std::allocator<doris::stream_load::VNodeChannel>, doris::stream_load::OlapTableSink*&, doris::stream_load::IndexChannel*, long&> (__a=..., __p=<optimized out>, this=<optimized out>) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr_base.h:650 apache#40 std::__shared_ptr<doris::stream_load::VNodeChannel, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<doris::stream_load::VNodeChannel>, doris::stream_load::OlapTableSink*&, doris::stream_load::IndexChannel*, long&> (__tag=..., this=<optimized out>) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr_base.h:1337 apache#41 std::shared_ptr<doris::stream_load::VNodeChannel>::shared_ptr<std::allocator<doris::stream_load::VNodeChannel>, doris::stream_load::OlapTableSink*&, doris::stream_load::IndexChannel*, long&> (__tag=..., this=<optimized out>) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr.h:409 apache#42 std::allocate_shared<doris::stream_load::VNodeChannel, std::allocator<doris::stream_load::VNodeChannel>, doris::stream_load::OlapTableSink*&, doris::stream_load::IndexChannel*, long&> (__a=...) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr.h:861 apache#43 std::make_shared<doris::stream_load::VNodeChannel, doris::stream_load::OlapTableSink*&, doris::stream_load::IndexChannel*, long&> () at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr.h:877 apache#44 doris::stream_load::IndexChannel::init (this=this@entry=0x563d4b43f200, state=state@entry=0x563e027a3500, tablets=...) at /data/TCHouse-D-1.2/be/src/exec/tablet_sink.cpp:705 apache#45 0x0000563ca79698d8 in doris::stream_load::OlapTableSink::prepare (this=this@entry=0x5644b9efe880, state=state@entry=0x563e027a3500) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr_base.h:1290 apache#46 0x0000563cac74db65 in doris::stream_load::VOlapTableSink::prepare (this=0x5644b9efe880, state=0x563e027a3500) at /data/TCHouse-D-1.2/be/src/vec/sink/vtablet_sink.cpp:450 apache#47 0x0000563ca7946c21 in doris::PlanFragmentExecutor::prepare (this=this@entry=0x563f55efd280, request=..., fragments_ctx=<optimized out>) at /var/local/ldb-toolchain/include/c++/11/bits/unique_ptr.h:173 apache#48 0x0000563ca791f17c in doris::FragmentExecState::prepare (this=this@entry=0x563f55efd200, params=...) --Type <RET> for more, q to quit, c to continue without paging-- at /data/TCHouse-D-1.2/be/src/runtime/fragment_mgr.cpp:238 apache#49 0x0000563ca7926629 in doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function<void (doris::PlanFragmentExecutor*)>) (this=this@entry=0x563cb6e93400, params=..., cb=...) at /data/TCHouse-D-1.2/be/src/runtime/fragment_mgr.cpp:720 apache#50 0x0000563ca7928dab in doris::FragmentMgr::exec_plan_fragment (this=0x563cb6e93400, params=...) at /data/TCHouse-D-1.2/be/src/runtime/fragment_mgr.cpp:564 apache#51 0x0000563ca7aaf507 in doris::PInternalServiceImpl::_exec_plan_fragment_impl (this=this@entry=0x563cb577b880, ser_request=..., version=<optimized out>, compact=<optimized out>) at /data/TCHouse-D-1.2/be/src/service/internal_service.cpp:480 apache#52 0x0000563ca7aaf703 in doris::PInternalServiceImpl::_exec_plan_fragment_in_pthread (this=0x563cb577b880, controller=<optimized out>, request=0x563d6c5bb9b0, response=0x564203b92ce0, done=0x563d885a60c0) at /data/TCHouse-D-1.2/be/src/service/internal_service.cpp:254 apache#53 0x0000563ca78d6dbd in std::function<void ()>::operator()() const (this=0x7f1f7fe090c8) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560 apache#54 doris::PriorityThreadPool::work_thread (this=0x563cb577ba38, thread_id=<optimized out>) at /data/TCHouse-D-1.2/be/src/util/priority_thread_pool.hpp:145 apache#55 0x0000563caf526b00 in std::execute_native_thread_routine (__p=0x563cbfc81d10) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 apache#56 0x00007f219771dea5 in start_thread () from /lib64/libpthread.so.0 apache#57 0x00007f2197a309fd in clone () from /lib64/libc.so.6 ``` ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
xy720
pushed a commit
that referenced
this pull request
Aug 17, 2025
…O0 (apache#54672) ### What problem does this PR solve? Because on the pipeline we use O1 for compilation, but some UBSan errors only appear locally when compiling with O0, for example: ``` 0x7ba502eb3a01: note: pointer points here 02 00 00 00 30 52 8c 02 95 7b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ #0 0x55cbe9f9acd2 in doris::Slice::compare(doris::Slice const&) const /mnt/disk8/yanxuecheng/tmp-doris/be/src/util/slice.h #1 0x55cbede3405c in doris::FieldTypeTraits<(doris::FieldType)13>::cmp(void const*, void const*) /mnt/disk8/yanxuecheng/tmp-doris/be/src/olap/types.h:1347:25 #2 0x55cbe9e98dae in doris::ScalarTypeInfo::cmp(void const*, void const*) const /mnt/disk8/yanxuecheng/tmp-doris/be/src/olap/types.h:104:74 #3 0x55cbeee73cf0 in int doris::Field::compare_cell<doris::RowCursorCell, doris::RowCursorCell>(doris::RowCursorCell const&, doris::RowCursorCell const&) const /mnt/disk8/yanxuecheng/tmp-doris/be/src/olap/field.h:120:41 #4 0x55cbeee6a030 in doris::compare_row_key(doris::RowCursor const&, doris::RowCursor const&) /mnt/disk8/yanxuecheng/tmp-doris/be/src/olap/tablet_reader.h:75:47 #5 0x55cbeee43237 in doris::TabletReader::_capture_rs_readers(doris::TabletReader::ReaderParams const&) /mnt/disk8/yanxuecheng/tmp-doris/be/src/olap/tablet_reader.cpp:188:17 #6 0x55cc1d5f7985 in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/olap/block_reader.cpp:106:16 #7 0x55cc1d5fd004 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/olap/block_reader.cpp:226:19 #8 0x55cc15e09f53 in doris::vectorized::OlapScanner::open(doris::RuntimeState*) /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/exec/scan/olap_scanner.cpp:260:32 #9 0x55cc15d7d89d in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182:5 #10 0x55cc15d88ce6 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96:17 apache#11 0x55cc15d884b1 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95:27 apache#12 0x55cc15d8829c in bool std::__invoke_impl<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&) /mnt/disk6/common/ldb_toolchain_taipan/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 apache#13 0x55cc15d8819c in std::enable_if<is_invocable_r_v<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>, bool>::type std::__invoke_r<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&) /mnt/disk6/common/ldb_toolchain_taipan/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9 apache#14 0x55cc15d87c94 in std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /mnt/disk6/common/ldb_toolchain_taipan/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 apache#15 0x55cbe9632efd in std::function<bool ()>::operator()() const /mnt/disk6/common/ldb_toolchain_taipan/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#16 0x55cc15d87314 in doris::vectorized::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l>>) /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/exec/scan/scanner_scheduler.cpp:407:25 ```
xy720
pushed a commit
that referenced
this pull request
Aug 18, 2025
…4737) Related PR: apache#54598 before 54598, logf is using impl from toolchain, after 54598, logf is using impl from doris glibc-compatibility Analysis from gpt5: ``` the 0x1ff << 23 part is evaluated in type int. 0x1ff = 511 decimal. Left-shifting it 23 bits gives 0x3ff800000 in math, which is way bigger than INT_MAX (on 32-bit signed int, any shift producing a bit in the sign position or beyond is undefined behavior in C). Even if ix and tmp are larger types, that constant expression is first computed as an int before any promotions, triggering the UB warning from UBSan. ``` Fix the problem below ```text #0 0x5632a618a17 in logf / root/doris/be/src/glibc-compatibility/musl/logf. c: 56:25 #1 0x5632a59fec26 in std:: log(float) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/cmath: 334:12 #2 0x5632a59fcee3 in lucene:: index: :MultiLevelSkipListWriter::MultiLevelSkipListWriter(int, int, int) /root/doris/contrib/clucene/src/core/CLucene/index/SkipListWriter.cpp:70:56 #3 0x5632a59fe497 in lucene:: index: :DefaultSkipListWriter: :DefaultSkipListWriter(int, int, int, lucene::store::IndexOutput*, lucene::store: :IndexOutput*) /root/doris/contrib/clucene/src/core/CLucene/index/SkipListWriter. cpp:175:11 + echo 'cp -r /mnt/ssd01/pipline/0penSourceDoris/clusterEnv/PO//Cluster0/fe/bin /home/work/pipline/backup_center/54276_0b92341b39685a8528a1d769885cefe95197e7fb_p0/fe/' + cp -r /mnt/ssd01/pipline/0penSourceDoris/clusterEnv/PO//Cluster0/fe/conf /home/work/pipline/backup_center/54276_0b92341b39685a8528a1d769885cefe95197e7fb_p0/fe/ #4 0x5632a5affdb in lucene:: index:: SDocumentsWriter<char>: :writeSegment(std: :vector<std::__cxx11::basic_string<char, std: :char_traits<char>, std: :allocator<char> >, std: :allocator<std::_cxx11::basic_string<char, std:: char_traits<chars #5 0x5632a5af009e in lucene:: index::DocumentsWriter<char>:: flush(bool) /root/doris/contrib/clucene/src/core/Lucene/index/DocumentWriter. cpp: 1340:9 #6 0x5632a5a41842 in lucene:: index:: IndexWriter:: doFlush(bool) /root/doris/contrib/clucene/src/core/CLucene/index/IndexWriter. cpp:2262:58 #7 0x5632a5a1f59b in lucene:: index:: IndexWriter:: flush(bool, bool) / root/doris/contrib/clucene/src/core/CLucene/index/IndexWriter.cpp:2154:9 #8 0x5632a5a1dd09 in lucene:: index:: IndexWriter:: closeInternal(bool) /root/doris/contrib/clucene/src/core/CLucene/index/IndexWriter. cpp: 546:9 #9 0x5632a5a1d16e in lucene:: index:: IndexWriter:: close(bool) / root/doris/contrib/clucene/src/core/CLucene/index/IndexWriter.cpp: 519:9 #10 0x563282c28c76 in _ZN5doris10segment_v213finally_closeISt10unique_ptrIN6lucene5index11IndexWriterESt14default_deleteIS5_EEQ8HasCloseIT_EEEVRS9_RNS0_12ErrorContextE /root/doris/be/src/olap/rowset/segment_v2/inverted_index_common.h:56 apache#11 0x563282c3b9c2 in doris::segment_v2: :InvertedIndexColumnWriter<(doris::FieldType)17>::finish) /root/doris/be/src/olap/rowset/segment_v2/inverted index_writer. cpp: 687:9 apache#12 0x563282b9fble in doris:: segment_v2::ArrayColumnWriter::write_inverted_index() /root/doris/be/src/olap/rowset/segment_v2/column_writer.cpp: 940:41 apache#13 0x56328248ec in doris:: segment_v2::VerticalSegmentWriter::_write_inverted_index() /root/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp: 1296:9 + echo cp -r /mnt/ssd01/pipline/0penSourceDoris/clusterEnv/P0//Cluster0/fe/log /home/work/pipline/backup_center/54276_0b92341b39685a8528a1d769885cefe95197e7fb_p0/fe/' apache#14 0x5632824d120 in doris::segment_v2::VerticalSegmentWriter::finalize_columns_indexunsigned long*) / root/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp: 1204:5 + cp -r /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0//Cluster0/fe/log /home/work/pipline/backup_center/54276_0b92341b39685a8528a1d769885cefe95197e7fb_p0/fe/ apache#15 0x56328251046 in doris: :segment_v2: :VerticalSegmentWriter:: finalize(unsigned long*, unsigned long*) / root/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp: 1253:5 apache#16 0x563282519a5a in doris: SegmentFlusher: :_flush_segment_writer(std: :unique_ptr<doris::segment_v2::VerticalSegmentWriter, std: :default_delete<doris::segment_v2: :VerticalSegmentWriter> >&, std: :shared_ptr<doris:: TabletSchema>, long*) apache#17 0x56328251642f in doris::SegmentFlusher::flush_single_block(doris::vectorized::Block const*, int, long*) /root/doris/be/src/olap/rowset/segment_creator.cpp:77:9 apache#18 0x563282521151 in doris::SegmentCreator::flush_single_block(doris::vectorized::Block const*, int, long*) /root/doris/be/src/olap/rowset/segment_creator.cpp: 389:5 apache#19 0x56329fe59dc4 in doris::BetaRowsetWriterV2:: flush_memtable(doris::vectorized::Block*, int, long*) /root/doris/be/src/olap/rowset/beta_rowset_writer_v2.cpp: 109:9 apache#20 0x56328215778 in doris:: FlushToken: :_do_flush_memtable(doris::MemTable*, int, long*) / root/doris/be/src/olap/memtable_flush_executor.cpp:210:9 apache#21 apache#22 0x56328217962 in doris:: FlushToken: :_flush_memtable(std: :shared_ptr<doris::MemTable>, int, long) / root/doris/be/src/olap/memtable_flush_executor. cpp:265:16 0x5632821f1f7d in doris::MemtableFlushTask:: run() /root/doris/be/src/olap/memtable_flush_executor.cpp:63:20 apache#23 0x56328437d33 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:614:24 apache#24 0x56328435аабс in std:: function<void ()›::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#25 0x56328435aac in doris::Thread:: supervise_thread(void*) / root/doris/be/src/util/thread. cpp: 461:5 apache#26 0x56327fc4fd26 in asan_thread_start(void*) (/mnt/ssd01/pipline/0penSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x28da4d26) apache#27 0x7fdd62967608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c: 477:8 apache#28 0x7fdd6287a132 in _clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S: 95 ```
xy720
pushed a commit
that referenced
this pull request
Aug 29, 2025
### What problem does this PR solve? For `alias(slotReference as slotReference)`, can replace with `slotReference` directly. For example, for sql `select first_value(b) over() from t`, Before this pr, it will have: LogicalProject[26] ( distinct=false, projects=[first_value(b) over()#2 AS first_value(b) over()#2] ) | LogicalWindow With this pr, it will have: LogicalProject[26] ( distinct=false, projects=[first_value(b) over()#2] ) | LogicalWindow
xy720
pushed a commit
that referenced
this pull request
Sep 4, 2025
TaskQueue should be released after all theads stop.
```
=================================================================
==6632==ERROR: AddressSanitizer: heap-use-after-free on address 0x61700518d440 at pc 0x5616fa5c2a87 bp 0x7f7635b709b0 sp 0x7f7635b709a8
READ of size 8 at 0x61700518d440 thread T5188 (Pipe_test_follo)
#0 0x5616fa5c2a86 in std::__atomic_base<unsigned long>::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x5616fa5c2a86 in std::__atomic_base<unsigned long>::operator unsigned long() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:341:16
#2 0x5616fa5c2a86 in doris::pipeline::PriorityTaskQueue::_try_take_unprotected(bool) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:58:9
#3 0x5616fa5c2e16 in doris::pipeline::PriorityTaskQueue::take(unsigned int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:110:16
#4 0x5616fa5c4b06 in doris::pipeline::MultiCoreTaskQueue::take(int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:170:51
#5 0x5616fa5d1919 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:267:35
#6 0x5616d1e169f8 in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.cpp:544:24
#7 0x5616d1df3e38 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#8 0x5616d1df3e38 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:498:5
#9 0x7f83a4c35ac2 in start_thread nptl/pthread_create.c:442:8
#10 0x7f83a4cc784f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
0x61700518d440 is located 704 bytes inside of 728-byte region [0x61700518d180,0x61700518d458)
freed by thread T2166 here:
#0 0x5616cea38d9d in operator delete(void*) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16dc0d9d) (BuildId: 09b7b6856fee4f7f)
#1 0x5616fa5cae77 in std::default_delete<doris::pipeline::PriorityTaskQueue>::operator()(doris::pipeline::PriorityTaskQueue*) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
#2 0x5616fa5cae77 in std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>::~unique_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
#3 0x5616fa5cae77 in void std::destroy_at<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
#4 0x5616fa5cae77 in void std::_Destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
#5 0x5616fa5cae77 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
#6 0x5616fa5cae77 in void std::_Destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
#7 0x5616fa5cae77 in void std::_Destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
#8 0x5616fa5cae77 in std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
#9 0x5616fa5c466d in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6
#10 0x5616fa5c466d in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11
apache#11 0x5616fa5c466d in std::__shared_ptr<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31
apache#12 0x5616fa5c466d in void std::atomic_store<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>>(std::shared_ptr<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>>*, std::shared_ptr<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>>) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_atomic.h:149:7
apache#13 0x5616fa5c466d in doris::pipeline::MultiCoreTaskQueue::close() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:151:5
apache#14 0x5616fa5ce9f4 in doris::pipeline::TaskScheduler::stop() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:437:26
apache#15 0x5616d19b7d36 in doris::WorkloadGroup::try_stop_schedulers() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:654:22
apache#16 0x5616d19c6042 in doris::WorkloadGroupMgr::delete_workload_group_by_ids(std::set<unsigned long, std::less<unsigned long>, std::allocator<unsigned long>>) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group_manager.cpp:107:13
apache#17 0x5616d1a7f6a1 in doris::WorkloadGroupListener::handle_topic_info(std::vector<doris::TopicInfo, std::allocator<doris::TopicInfo>> const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/workload_group_listener.cpp:82:38
apache#18 0x5616d1a7dd7e in doris::TopicSubscriber::handle_topic_info(doris::TPublishTopicRequest const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/topic_subscriber.cpp:45:35
apache#19 0x5616d2002932 in doris::BackendServiceProcessor::process_publish_topic_info(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:6577:13
apache#20 0x5616d1fd6f11 in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, int, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:5416:3
apache#21 0x5616d2039186 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) /home/zcp/repo_center/doris_branch-2.1/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12
apache#22 0x5616faee19f7 in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432699f7) (BuildId: 09b7b6856fee4f7f)
apache#23 0x5616faee2cf6 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326acf6) (BuildId: 09b7b6856fee4f7f)
apache#24 0x5616faee830d in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4327030d) (BuildId: 09b7b6856fee4f7f)
apache#25 0x5616faee81b3 in void std::__invoke_impl<void, void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(std::__invoke_other, void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432701b3) (BuildId: 09b7b6856fee4f7f)
apache#26 0x5616faee8106 in std::__invoke_result<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>::type std::__invoke<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270106) (BuildId: 09b7b6856fee4f7f)
apache#27 0x5616faee8076 in void std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270076) (BuildId: 09b7b6856fee4f7f)
apache#28 0x5616faee7ffb in std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::operator()() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326fffb) (BuildId: 09b7b6856fee4f7f)
apache#29 0x5616faee7f9f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326ff9f) (BuildId: 09b7b6856fee4f7f)
apache#30 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
previously allocated by thread T3392 here:
#0 0x5616cea3853d in operator new(unsigned long) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16dc053d) (BuildId: 09b7b6856fee4f7f)
#1 0x5616fa5c3e29 in std::_MakeUniq<doris::pipeline::PriorityTaskQueue>::__single_object std::make_unique<doris::pipeline::PriorityTaskQueue>() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:30
#2 0x5616fa5c3e29 in doris::pipeline::MultiCoreTaskQueue::MultiCoreTaskQueue(int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:139:39
#3 0x5616d19b276d in decltype(::new((void*)(0)) doris::pipeline::MultiCoreTaskQueue(std::declval<int&>())) std::construct_at<doris::pipeline::MultiCoreTaskQueue, int&>(doris::pipeline::MultiCoreTaskQueue*, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39
#4 0x5616d19b276d in void std::allocator_traits<std::allocator<doris::pipeline::MultiCoreTaskQueue>>::construct<doris::pipeline::MultiCoreTaskQueue, int&>(std::allocator<doris::pipeline::MultiCoreTaskQueue>&, doris::pipeline::MultiCoreTaskQueue*, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:514:4
#5 0x5616d19b276d in std::_Sp_counted_ptr_inplace<doris::pipeline::MultiCoreTaskQueue, std::allocator<doris::pipeline::MultiCoreTaskQueue>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<int&>(std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:519:4
#6 0x5616d19b276d in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<doris::pipeline::MultiCoreTaskQueue, std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(doris::pipeline::MultiCoreTaskQueue*&, std::_Sp_alloc_shared_tag<std::allocator<doris::pipeline::MultiCoreTaskQueue>>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:651:6
#7 0x5616d19b276d in std::__shared_ptr<doris::pipeline::MultiCoreTaskQueue, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(std::_Sp_alloc_shared_tag<std::allocator<doris::pipeline::MultiCoreTaskQueue>>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1337:14
#8 0x5616d19b276d in std::shared_ptr<doris::pipeline::MultiCoreTaskQueue>::shared_ptr<std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(std::_Sp_alloc_shared_tag<std::allocator<doris::pipeline::MultiCoreTaskQueue>>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:409:4
#9 0x5616d19b276d in std::shared_ptr<doris::pipeline::MultiCoreTaskQueue> std::allocate_shared<doris::pipeline::MultiCoreTaskQueue, std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(std::allocator<doris::pipeline::MultiCoreTaskQueue> const&, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:860:14
#10 0x5616d19b276d in std::shared_ptr<doris::pipeline::MultiCoreTaskQueue> std::make_shared<doris::pipeline::MultiCoreTaskQueue, int&>(int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:876:14
apache#11 0x5616d19b276d in doris::WorkloadGroup::upsert_thread_pool_no_lock(doris::WorkloadGroupInfo*, std::shared_ptr<doris::CgroupCpuCtl>, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:462:27
apache#12 0x5616d19b5991 in doris::WorkloadGroup::upsert_task_scheduler(doris::WorkloadGroupInfo*, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:572:9
apache#13 0x5616d1a7ecd8 in doris::WorkloadGroupListener::handle_topic_info(std::vector<doris::TopicInfo, std::allocator<doris::TopicInfo>> const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/workload_group_listener.cpp:60:13
apache#14 0x5616d1a7dd7e in doris::TopicSubscriber::handle_topic_info(doris::TPublishTopicRequest const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/topic_subscriber.cpp:45:35
apache#15 0x5616d2002932 in doris::BackendServiceProcessor::process_publish_topic_info(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:6577:13
apache#16 0x5616d1fd6f11 in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, int, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:5416:3
apache#17 0x5616d2039186 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) /home/zcp/repo_center/doris_branch-2.1/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12
apache#18 0x5616faee19f7 in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432699f7) (BuildId: 09b7b6856fee4f7f)
apache#19 0x5616faee2cf6 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326acf6) (BuildId: 09b7b6856fee4f7f)
apache#20 0x5616faee830d in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4327030d) (BuildId: 09b7b6856fee4f7f)
apache#21 0x5616faee81b3 in void std::__invoke_impl<void, void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(std::__invoke_other, void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432701b3) (BuildId: 09b7b6856fee4f7f)
apache#22 0x5616faee8106 in std::__invoke_result<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>::type std::__invoke<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270106) (BuildId: 09b7b6856fee4f7f)
apache#23 0x5616faee8076 in void std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270076) (BuildId: 09b7b6856fee4f7f)
apache#24 0x5616faee7ffb in std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::operator()() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326fffb) (BuildId: 09b7b6856fee4f7f)
apache#25 0x5616faee7f9f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326ff9f) (BuildId: 09b7b6856fee4f7f)
apache#26 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
Thread T5188 (Pipe_test_follo) created by T3392 here:
#0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
#1 0x5616d1df2cc9 in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()> const&, unsigned long, scoped_refptr<doris::Thread>*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:449:15
#2 0x5616d1e1d979 in doris::Status doris::Thread::create<void (doris::ThreadPool::*)(), doris::ThreadPool*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, void (doris::ThreadPool::* const&)(), doris::ThreadPool* const&, scoped_refptr<doris::Thread>*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.h:56:16
#3 0x5616d1e1423d in doris::ThreadPool::create_thread() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.cpp:612:12
#4 0x5616d1e13e43 in doris::ThreadPool::init() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.cpp:266:25
#5 0x5616cec31535 in doris::Status doris::ThreadPoolBuilder::build<doris::ThreadPool>(std::unique_ptr<doris::ThreadPool, std::default_delete<doris::ThreadPool>>*) const /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.h:121:13
#6 0x5616fa5cef22 in doris::pipeline::TaskScheduler::start() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:208:5
#7 0x5616d19b2a9b in doris::WorkloadGroup::upsert_thread_pool_no_lock(doris::WorkloadGroupInfo*, std::shared_ptr<doris::CgroupCpuCtl>, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:467:47
#8 0x5616d19b5991 in doris::WorkloadGroup::upsert_task_scheduler(doris::WorkloadGroupInfo*, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:572:9
#9 0x5616d1a7ecd8 in doris::WorkloadGroupListener::handle_topic_info(std::vector<doris::TopicInfo, std::allocator<doris::TopicInfo>> const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/workload_group_listener.cpp:60:13
#10 0x5616d1a7dd7e in doris::TopicSubscriber::handle_topic_info(doris::TPublishTopicRequest const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/topic_subscriber.cpp:45:35
apache#11 0x5616d2002932 in doris::BackendServiceProcessor::process_publish_topic_info(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:6577:13
apache#12 0x5616d1fd6f11 in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, int, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:5416:3
apache#13 0x5616d2039186 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) /home/zcp/repo_center/doris_branch-2.1/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12
apache#14 0x5616faee19f7 in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432699f7) (BuildId: 09b7b6856fee4f7f)
apache#15 0x5616faee2cf6 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326acf6) (BuildId: 09b7b6856fee4f7f)
apache#16 0x5616faee830d in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4327030d) (BuildId: 09b7b6856fee4f7f)
apache#17 0x5616faee81b3 in void std::__invoke_impl<void, void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(std::__invoke_other, void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432701b3) (BuildId: 09b7b6856fee4f7f)
apache#18 0x5616faee8106 in std::__invoke_result<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>::type std::__invoke<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270106) (BuildId: 09b7b6856fee4f7f)
apache#19 0x5616faee8076 in void std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270076) (BuildId: 09b7b6856fee4f7f)
apache#20 0x5616faee7ffb in std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::operator()() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326fffb) (BuildId: 09b7b6856fee4f7f)
apache#21 0x5616faee7f9f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326ff9f) (BuildId: 09b7b6856fee4f7f)
apache#22 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
Thread T3392 created by T933 here:
#0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
#1 0x5616fd8057b5 in __gthread_create /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663:35
#2 0x5616fd8057b5 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:147:37
#3 0x5616faee667c in apache::thrift::concurrency::Thread::start() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326e67c) (BuildId: 09b7b6856fee4f7f)
#4 0x5616faee2a26 in apache::thrift::server::TThreadedServer::onClientConnected(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326aa26) (BuildId: 09b7b6856fee4f7f)
#5 0x5616faedebc4 in apache::thrift::server::TServerFramework::newlyConnectedClient(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43266bc4) (BuildId: 09b7b6856fee4f7f)
#6 0x5616faede25f in apache::thrift::server::TServerFramework::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326625f) (BuildId: 09b7b6856fee4f7f)
#7 0x5616faee2791 in apache::thrift::server::TThreadedServer::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326a791) (BuildId: 09b7b6856fee4f7f)
#8 0x5616d1e31419 in doris::ThriftServer::ThriftServerEventProcessor::supervise() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_server.cpp:206:34
#9 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
Thread T933 created by T0 here:
#0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
#1 0x5616fd8057b5 in __gthread_create /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663:35
#2 0x5616fd8057b5 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:147:37
#3 0x5616d1e37788 in doris::ThriftServer::start() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_server.cpp:421:5
#4 0x5616cea3f174 in main /home/zcp/repo_center/doris_branch-2.1/doris/be/src/service/doris_main.cpp:536:25
#5 0x7f83a4bcad8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
Thread T2166 created by T933 here:
#0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
#1 0x5616fd8057b5 in __gthread_create /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663:35
#2 0x5616fd8057b5 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:147:37
#3 0x5616faee667c in apache::thrift::concurrency::Thread::start() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326e67c) (BuildId: 09b7b6856fee4f7f)
#4 0x5616faee2a26 in apache::thrift::server::TThreadedServer::onClientConnected(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326aa26) (BuildId: 09b7b6856fee4f7f)
#5 0x5616faedebc4 in apache::thrift::server::TServerFramework::newlyConnectedClient(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43266bc4) (BuildId: 09b7b6856fee4f7f)
#6 0x5616faede25f in apache::thrift::server::TServerFramework::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326625f) (BuildId: 09b7b6856fee4f7f)
#7 0x5616faee2791 in apache::thrift::server::TThreadedServer::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326a791) (BuildId: 09b7b6856fee4f7f)
#8 0x5616d1e31419 in doris::ThriftServer::ThriftServerEventProcessor::supervise() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_server.cpp:206:34
#9 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
SUMMARY: AddressSanitizer: heap-use-after-free /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9 in std::__atomic_base<unsigned long>::load(std::memory_order) const
Shadow bytes around the buggy address:
0x61700518d180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x61700518d200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x61700518d280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x61700518d300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x61700518d380: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x61700518d400: fd fd fd fd fd fd fd fd[fd]fd fd fa fa fa fa fa
0x61700518d480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x61700518d500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x61700518d580: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x61700518d600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x61700518d680: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==6632==ABORTING
```
xy720
pushed a commit
that referenced
this pull request
Sep 4, 2025
* aws/aws-sdk-cpp#2314 * In the aws-sdk-cpp code, there is a suggestion: "Please call ShutdownAPI from the same thread from which InitAPI() has been called" otherwise it will cause memory leak. ``` ================================================================= ==3271527==ERROR: LeakSanitizer: detected memory leaks Direct leak of 216 byte(s) in 1 object(s) allocated from: #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0) #1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0) #2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0) #3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0) #4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28 #5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15 #6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21 ...... ``` ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
xy720
pushed a commit
that referenced
this pull request
Sep 4, 2025
…pache#55500) ### What problem does this PR solve? Related PR: apache#45937 Problem Summary: Fix the error case on ingestion load and the core in parquet reader. ==8898==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62f0020603fc at pc 0x55f634e64ded bp 0x7fba0d03c410 sp 0x7fba0d03bbd8 READ of size 4 at 0x62f0020603fc thread T768 (PUSH-9699) #0 0x55f634e64dec in __asan_memcpy (/mnt/hdd01/ci/doris-deploy-branch-3.1-local/be/lib/doris_be+0x39a24dec) (BuildId: 9b04e7f7d3075dac) #1 0x55f634eca93f in std::char_traits::copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/char_traits.h:409:33 #2 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:351:4 #3 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy_chars(char*, char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:398:9 #4 0x55f634eca93f in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:225:6 #5 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct_aux(char const*, char const*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11 #6 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4 #7 0x55f654a4f74d in std::__cxx11::basic_string, std::allocator>::basic_string(char const*, unsigned long, std::allocator const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:513:9 #8 0x55f654a4f74d in doris::vectorized::parse_thrift_footer(std::shared_ptr, doris::vectorized::FileMetaData**, unsigned long*, doris::io::IOContext*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/parquet_thrift_util.h:55:17
xy720
pushed a commit
that referenced
this pull request
Sep 7, 2025
…pache#55346) ### What problem does this PR solve? Related PR: apache#53729 Problem Summary: runtime error: member access within null pointer of type 'FileReaderStats' in `TracingFileReader` ``` /home/zcp/repo_center/doris_branch-3.1/doris/be/src/io/fs/tracing_file_reader.h:34:9: runtime error: member access within null pointer of type 'FileReaderStats' #0 0x55c028481ace in doris::io::TracingFileReader::read_at_impl(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/io/fs/tracing_file_reader.h:34:9 #1 0x55c00922a1f7 in doris::io::FileReader::read_at(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/io/fs/file_reader.cpp:34:17 #2 0x55c028ae8a18 in doris::vectorized::parse_thrift_footer(std::shared_ptr<doris::io::FileReader>, doris::vectorized::FileMetaData**, unsigned long*, doris::io::IOContext*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/parquet_thrift_util.h:44:5 #3 0x55c028ae8a18 in doris::vectorized::ParquetReader::_open_file() /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:234:21 #4 0x55c028aeb69c in doris::vectorized::ParquetReader::init_reader(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::variant<doris::ColumnValueRange<(doris::PrimitiveType)3>, doris::ColumnValueRange<(doris::PrimitiveType)4>, doris::ColumnValueRange<(doris::PrimitiveType)5>, doris::ColumnValueRange<(doris::PrimitiveType)6>, doris::ColumnValueRange<(doris::PrimitiveType)7>, doris::ColumnValueRange<(doris::PrimitiveType)36>, doris::ColumnValueRange<(doris::PrimitiveType)37>, doris::ColumnValueRange<(doris::PrimitiveType)15>, doris::ColumnValueRange<(doris::PrimitiveType)10>, doris::ColumnValueRange<(doris::PrimitiveType)23>, doris::ColumnValueRange<(doris::PrimitiveType)11>, doris::ColumnValueRange<(doris::PrimitiveType)25>, doris::ColumnValueRange<(doris::PrimitiveType)12>, doris::ColumnValueRange<(doris::PrimitiveType)26>, doris::ColumnValueRange<(doris::PrimitiveType)20>, doris::ColumnValueRange<(doris::PrimitiveType)2>, doris::ColumnValueRange<(doris::PrimitiveType)19>, doris::ColumnValueRange<(doris::PrimitiveType)28>, doris::ColumnValueRange<(doris::PrimitiveType)29>, doris::ColumnValueRange<(doris::PrimitiveType)30>, doris::ColumnValueRange<(doris::PrimitiveType)35>>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const, std::variant<doris::ColumnValueRange<(doris::PrimitiveType)3>, doris::ColumnValueRange<(doris::PrimitiveType)4>, doris::ColumnValueRange<(doris::PrimitiveType)5>, doris::ColumnValueRange<(doris::PrimitiveType)6>, doris::ColumnValueRange<(doris::PrimitiveType)7>, doris::ColumnValueRange<(doris::PrimitiveType)36>, doris::ColumnValueRange<(doris::PrimitiveType)37>, doris::ColumnValueRange<(doris::PrimitiveType)15>, doris::ColumnValueRange<(doris::PrimitiveType)10>, doris::ColumnValueRange<(doris::PrimitiveType)23>, doris::ColumnValueRange<(doris::PrimitiveType)11>, doris::ColumnValueRange<(doris::PrimitiveType)25>, doris::ColumnValueRange<(doris::PrimitiveType)12>, doris::ColumnValueRange<(doris::PrimitiveType)26>, doris::ColumnValueRange<(doris::PrimitiveType)20>, doris::ColumnValueRange<(doris::PrimitiveType)2>, doris::ColumnValueRange<(doris::PrimitiveType)19>, doris::ColumnValueRange<(doris::PrimitiveType)28>, doris::ColumnValueRange<(doris::PrimitiveType)29>, doris::ColumnValueRange<(doris::PrimitiveType)30>, doris::ColumnValueRange<(doris::PrimitiveType)35>>>>> const*, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>> const&, doris::TupleDescriptor const*, doris::RowDescriptor const*, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, int, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const, int>>> const*, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>> const*, std::unordered_map<int, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>>>>> const*, std::shared_ptr<doris::vectorized::TableSchemaChangeHelper::Node>, bool) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:318:5 #5 0x55c00cceda85 in doris::PushBrokerReader::_get_next_reader() /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/push_handler.cpp:671:39 #6 0x55c00cce72df in doris::PushBrokerReader::next(doris::vectorized::Block*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/push_handler.cpp:434:9 #7 0x55c00cce27b3 in doris::PushHandler::_convert_v2(std::shared_ptr<doris::Tablet>, std::shared_ptr<doris::Rowset>*, std::shared_ptr<doris::TabletSchema>, doris::PushType) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/push_handler.cpp:293:31 #8 0x55c00ccdd92d in doris::PushHandler::_do_streaming_ingestion(std::shared_ptr<doris::Tablet>, doris::TPushReq const&, doris::PushType, std::vector<doris::TTabletInfo, std::allocator<doris::TTabletInfo>>*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/push_handler.cpp:199:11 #9 0x55c00ccd96f3 in doris::PushHandler::process_streaming_ingestion(std::shared_ptr<doris::Tablet>, doris::TPushReq const&, doris::PushType, std::vector<doris::TTabletInfo, std::allocator<doris::TTabletInfo>>*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/push_handler.cpp:94:11 #10 0x55c00cccff9c in doris::EngineBatchLoadTask::_push(doris::TPushReq const&, std::vector<doris::TTabletInfo, std::allocator<doris::TTabletInfo>>*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/task/engine_batch_load_task.cpp:293:28 apache#11 0x55c00cccbbc3 in doris::EngineBatchLoadTask::_process() /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/task/engine_batch_load_task.cpp:249:18 apache#12 0x55c00ccc6851 in doris::EngineBatchLoadTask::execute() /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/task/engine_batch_load_task.cpp:83:22 apache#13 0x55c0090293eb in doris::push_callback(doris::StorageEngine&, doris::TAgentTaskRequest const&) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/agent/task_worker_pool.cpp:1880:31 apache#14 0x55c008ff6554 in doris::PriorTaskWorkerPool::normal_loop() /home/zcp/repo_center/doris_branch-3.1/doris/be/src/agent/task_worker_pool.cpp:687:9 apache#15 0x55c00db29237 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/util/thread.cpp:498:5 apache#16 0x7fa0651f2ac2 in start_thread nptl/pthread_create.c:442:8 apache#17 0x7fa06528484f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 ```
xy720
pushed a commit
that referenced
this pull request
Sep 18, 2025
Related PR: apache#45937 branch-3.1: apache#55500 Problem Summary: Fix the error case on ingestion load and the core in parquet reader. ``` ==8898==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62f0020603fc at pc 0x55f634e64ded bp 0x7fba0d03c410 sp 0x7fba0d03bbd8 READ of size 4 at 0x62f0020603fc thread T768 (PUSH-9699) #0 0x55f634e64dec in __asan_memcpy (/mnt/hdd01/ci/doris-deploy-branch-3.1-local/be/lib/doris_be+0x39a24dec) (BuildId: 9b04e7f7d3075dac) #1 0x55f634eca93f in std::char_traits::copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/char_traits.h:409:33 #2 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:351:4 #3 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy_chars(char*, char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:398:9 #4 0x55f634eca93f in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:225:6 #5 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct_aux(char const*, char const*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11 #6 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4 #7 0x55f654a4f74d in std::__cxx11::basic_string, std::allocator>::basic_string(char const*, unsigned long, std::allocator const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:513:9 #8 0x55f654a4f74d in doris::vectorized::parse_thrift_footer(std::shared_ptr, doris::vectorized::FileMetaData**, unsigned long*, doris::io::IOContext*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/parquet_thrift_util.h:55:17 ```
xy720
pushed a commit
that referenced
this pull request
Oct 16, 2025
…armup` due to capture by reference (apache#56395) ### What problem does this PR solve? introduced in apache#54611 ``` (gdb) bt #0 __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ./nptl/pthread_sigmask.c:43 #1 0x00007f7aa6b5d71e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so #2 0x00007f7aa6b5e206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so #3 <signal handler called> #4 0x000055e20050d443 in std::_Hashtable<doris::RowsetId, std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> >, std::__detail::_Select1st, std::equal_to<doris::RowsetId>, std::hash<doris::RowsetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node (this=0x7f79e69acde0, __bkt=4186920012728959759, __k=..., __code=11200260987994981938) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:2205 #5 std::_Hashtable<doris::RowsetId, std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> >, std::__detail::_Select1st, std::equal_to<doris::RowsetId>, std::hash<doris::RowsetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_locate (this=this@entry=0x7f79e69acde0, __k=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:2283 #6 0x000055e200505a8d in std::_Hashtable<doris::RowsetId, std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> >, std::__detail::_Select1st, std::equal_to<doris::RowsetId>, std::hash<doris::RowsetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find (this=0x7f79e69acde0, __k=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:1929 #7 std::unordered_map<doris::RowsetId, doris::CloudTablet::RowsetWarmUpInfo, std::hash<doris::RowsetId>, std::equal_to<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> > >::contains (this=0x7f79e69acde0, __x=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unordered_map.h:999 #8 doris::CloudTablet::complete_rowset_segment_warmup (this=<optimized out>, rowset_id=..., status=..., segment_num=1, inverted_idx_num=0) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_tablet.cpp:1692 #9 0x000055e2005338ba in doris::CloudWarmUpManager::handle_jobs()::$_0::operator()(doris::Status) const (this=0x7f79e8bebad0, st=...) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_warm_up_manager.cpp:243 #10 std::__invoke_impl<void, doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status>(std::__invoke_other, doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status&&) (__f=..., __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63 apache#11 std::__invoke_r<void, doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status>(doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status&&) (__fn=..., __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113 apache#12 std::_Function_handler<void (doris::Status), doris::CloudWarmUpManager::handle_jobs()::$_0>::_M_invoke(std::_Any_data const&, doris::Status&&) (__functor=..., __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 apache#13 0x000055e200533310 in std::function<void (doris::Status)>::operator()(doris::Status) const (this=0x7f795a3ff1d0, __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593 apache#14 doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0::operator()(doris::Status) const ( this=0x7f79f0710b60, st=...) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_warm_up_manager.cpp:149 apache#15 std::__invoke_impl<void, doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status>(std::__invoke_other, doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status&&) ( __f=..., __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63 apache#16 std::__invoke_r<void, doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status>(doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status&&) (__fn=..., __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113 apache#17 std::_Function_handler<void (doris::Status), doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0>::_M_invoke(std::_Any_data const&, doris::Status&&) (__functor=..., __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 apache#18 0x000055e200469cb1 in std::function<void (doris::Status)>::operator()(doris::Status) const (this=0x7f796941cee0, __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593 apache#19 doris::io::FileCacheBlockDownloader::download_segment_file (this=<optimized out>, meta=...) at /home/zcp/repo_center/doris_master/doris/be/src/io/cache/block_file_cache_downloader.cpp:297 apache#20 0x000055e1fbfcbe85 in doris::ThreadPool::dispatch_thread (this=0x7f7995a1da00) at /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:614 apache#21 0x000055e1fbfc0fac in std::function<void ()>::operator()() const (this=0x7f796941cee0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593 apache#22 doris::Thread::supervise_thread (arg=0x7f79f13de010) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:460 apache#23 0x00007f7aa5bf8ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 apache#24 0x00007f7aa5c8a850 in __closefrom_fallback (from=1685031872, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45 apache#25 0x0000000000000000 in ?? () ``` Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
xy720
pushed a commit
that referenced
this pull request
Oct 16, 2025
```
/home/zcp/repo_center/doris_master/doris/be/src/util/counts.h:127:85: runtime error: signed integer overflow: 147483648 - -2147483648 cannot be represented in type 'int'
#0 0x558c5a9d58cd in doris::Counts::terminate(double) /home/zcp/repo_center/doris_master/doris/be/src/util/counts.h:127:85
#1 0x558c5ab0ce99 in doris::vectorized::PercentileState<(doris::PrimitiveType)5>::insert_result_into(doris::vectorized::IColumn&) const /home/zcp/repo_center/doris_master/doris/be/src/vec/aggregate_functions/aggregate_function_percentile.h:430:49
#2 0x558c5ab0812f in doris::vectorized::AggregateFunctionPercentileArray<(doris::PrimitiveType)5>::insert_result_into(char const*, doris::vectorized::IColumn&) const /home/zcp/repo_center/doris_master/doris/be/src/vec/aggregate_functions/aggregate_function_percentile.h:557:59
#3 0x558c5ab086a8 in doris::vectorized::IAggregateFunctionHelper>::insert_result_into_vec(std::vector> const&, unsigned long, doris::vectorized::IColumn&, unsigned long) const /home/zcp/repo_center/doris_master/doris/be/src/vec/aggregate_functions/aggregate_function.h:393:22
#4 0x558c68af8445 in _ZZN5doris8pipeline13AggLocalState31_get_with_serialized_key_resultEPNS_12RuntimeStateEPNS_10vectorized5BlockEPbENK3$_1clINS4_15MethodKeysFixedI9PHHashMapIN4wide7integerILm256EjEEPc9HashCRC32ISE_EEEEEEvRT_ /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/aggregation_source_operator.cpp:287:67
#5 0x558c68af8445 in void std::__invoke_impl, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>&>(std::__invoke_other, doris::vectorized::Overload&&, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14
#6 0x558c68af8445 in std::__invoke_result, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>&>::type std::__invoke, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>&>(doris::vectorized::Overload&&, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:98:14
#7 0x558c68af8445 in std::__detail::__variant::__gen_vtable_impl (*)(doris::vectorized::Overload&&, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&)>, std::integer_sequence>::__visit_invoke(doris::vectorized::Overload&&, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/variant:1055:11
#8 0x558c68a85790 in decltype(auto) std::__do_visit, doris::vectorized::Overload, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&>(doris::vectorized::Overload&&, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/variant:1858:15
#9 0x558c68a85790 in std::invoke_result, std::__conditional>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&>>::type>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&>()))>::type>::type&&>>::type std::visit, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&>(doris::vectorized::Overload&&, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/variant:1954:13
#10 0x558c68a85790 in doris::pipeline::AggLocalState::_get_with_serialized_key_result(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/aggregation_source_operator.cpp:251:5
apache#11 0x558c68b3e3e7 in doris::pipeline::AggLocalState::init(doris::RuntimeState*, doris::pipeline::LocalStateInfo&)::$_2::operator()(doris::RuntimeState*, doris::vectorized::Block*, bool*) const /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/aggregation_source_operator.cpp:75:24
apache#12 0x558c68b3e3e7 in doris::Status std::__invoke_impl(std::__invoke_other, doris::pipeline::AggLocalState::init(doris::RuntimeState*, doris::pipeline::LocalStateInfo&)::$_2&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14
apache#13 0x558c68b3e3e7 in std::enable_if, doris::Status>::type std::__invoke_r(doris::pipeline::AggLocalState::init(doris::RuntimeState*, doris::pipeline::LocalStateInfo&)::$_2&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9
apache#14 0x558c68b3e3e7 in std::_Function_handler::_M_invoke(std::_Any_data const&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9
apache#15 0x558c68a8b819 in std::function::operator()(doris::RuntimeState*, doris::vectorized::Block*, bool*) const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
apache#16 0x558c68a8b819 in doris::pipeline::AggSourceOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/aggregation_source_operator.cpp:448:5
apache#17 0x558c68f652cb in doris::pipeline::PartitionedAggSourceOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp:167:36
apache#18 0x558c676b1ddd in doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/operator.cpp:391:18
apache#19 0x558c69b0493d in doris::pipeline::PipelineTask::execute(bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.cpp:521:13
apache#20 0x558c69b52916 in doris::pipeline::TaskScheduler::_do_work(int) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/task_scheduler.cpp:147:9
apache#21 0x558c54fa66f2 in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:614:24
apache#22 0x558c54f83766 in std::function::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
apache#23 0x558c54f83766 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:460:5
apache#24 0x558c507cbd26 in asan_thread_start(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x1f57fd26)
apache#25 0x7fb97f744ac2 in start_thread nptl/pthread_create.c:442:8
apache#26 0x7fb97f7d684f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/zcp/repo_center/doris_master/doris/be/src/util/counts.h:127:85
```
xy720
pushed a commit
that referenced
this pull request
Oct 16, 2025
### What problem does this PR solve? introduced by apache#56405 2 changes here: 1. print local runtime filter mgr info inside each task 2. make a pair for tasks with its corresponding runtime state. ==198869==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7ba3927ac338 at pc 0x559d81e109b9 bp 0x7b7a5b8ad410 sp 0x7b7a5b8ad408 12:38:06 READ of size 8 at 0x7ba3927ac338 thread T1845 (EvHttpServer [w) 12:38:06 #0 0x559d81e109b8 in std::__uniq_ptr_impl<doris::RuntimeState, std::default_delete<doris::RuntimeState> >::_M_ptr() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:193:51 12:38:06 #1 0x559d81e109b8 in std::unique_ptr<doris::RuntimeState, std::default_delete<doris::RuntimeState> >::get() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:473:21 12:38:06 #2 0x559d81e109b8 in std::unique_ptr<doris::RuntimeState, std::default_delete<doris::RuntimeState> >::operator->() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:466:9 12:38:06 #3 0x559d81e109b8 in doris::pipeline::PipelineFragmentContext::debug_string[abi:cxx11]() /root/doris/be/src/pipeline/pipeline_fragment_context.cpp:1941:28 12:38:06 #4 0x559d6d65aa2f in doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0::operator()(phmap::flat_hash_map<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, phmap::Hash<std::pair<doris::TUniqueId, int> >, phmap::EqualTo<std::pair<doris::TUniqueId, int> >, std::allocator<std::pair<std::pair<doris::TUniqueId, int> const, std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&) const /root/doris/be/src/runtime/fragment_mgr.cpp:807:36 12:38:06 + echo 'cp -r /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0//Cluster0/fe/bin /home/work/pipline/backup_center/56368_04d746711764283b1cbafe40b1f849e10a5bcb6e_p0/fe/' 12:38:06 #5 0x559d6d65aa2f in doris::Status std::__invoke_impl<doris::Status, doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0&, phmap::flat_hash_map<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, phmap::Hash<std::pair<doris::TUniqueId, int> >, phmap::EqualTo<std::pair<doris::TUniqueId, int> >, std::allocator<std::pair<std::pair<doris::TUniqueId, int> const, std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&>(std::__invoke_other, doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0&, phmap::flat_hash_map<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, phmap::Hash<std::pair<doris::TUniqueId, int> >, phmap::EqualTo<std::pair<doris::TUniqueId, int> >, std::allocator<std::pair<std::pair<doris::TUniqueId, int> const, std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 12:38:06 + cp -r /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0//Cluster0/fe/conf /home/work/pipline/backup_center/56368_04d746711764283b1cbafe40b1f849e10a5bcb6e_p0/fe/ 12:38:06 #6 0x559d6d65aa2f in std::enable_if<is_invocable_r_v<doris::Status, doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0&, phmap::flat_hash_map<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, phmap::Hash<std::pair<doris::TUniqueId, int> >, phmap::EqualTo<std::pair<doris::TUniqueId, int> >, std::allocator<std::pair<std::pair<doris::TUniqueId, int> const, std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&>, doris::Status>::type std::__invoke_r<doris::Status, doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0&, phmap::flat_hash_map<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, phmap::Hash<std::pair<doris::TUniqueId, int> >, phmap::EqualTo<std::pair<doris::TUniqueId, int> >, std::allocator<std::pair<std::pair<doris::TUniqueId, int> const, std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&>(doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0&, phmap::flat_hash_map<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, phmap::Hash<std::pair<doris::TUniqueId, int> >, phmap::EqualTo<std::pair<doris::TUniqueId, int> >, std::allocator<std::pair<std::pair<doris::TUniqueId, int> const, std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9 12:38:06 #7 0x559d6d65aa2f in std::_Function_handler<doris::Status (phmap::flat_hash_map<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, phmap::Hash<std::pair<doris::TUniqueId, int> >, phmap::EqualTo<std::pair<doris::TUniqueId, int> >, std::allocator<std::pair<std::pair<doris::TUniqueId, int> const, std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&), doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0>::_M_invoke(std::_Any_data const&, phmap::flat_hash_map<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, phmap::Hash<std::pair<doris::TUniqueId, int> >, phmap::EqualTo<std::pair<doris::TUniqueId, int> >, std::allocator<std::pair<std::pair<doris::TUniqueId, int> const, std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 12:38:06 #8 0x559d6d665d37 in std::function<doris::Status (phmap::flat_hash_map<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, phmap::Hash<std::pair<doris::TUniqueId, int> >, phmap::EqualTo<std::pair<doris::TUniqueId, int> >, std::allocator<std::pair<std::pair<doris::TUniqueId, int> const, std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&)>::operator()(phmap::flat_hash_map<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, phmap::Hash<std::pair<doris::TUniqueId, int> >, phmap::EqualTo<std::pair<doris::TUniqueId, int> >, std::allocator<std::pair<std::pair<doris::TUniqueId, int> const, std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&) const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 12:38:06 #9 0x559d6d665d37 in doris::ConcurrentContextMap<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, doris::pipeline::PipelineFragmentContext>::apply(std::function<doris::Status (phmap::flat_hash_map<std::pair<doris::TUniqueId, int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>, phmap::Hash<std::pair<doris::TUniqueId, int> >, phmap::EqualTo<std::pair<doris::TUniqueId, int> >, std::allocator<std::pair<std::pair<doris::TUniqueId, int> const, std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&)>&&) /root/doris/be/src/runtime/fragment_mgr.h:94:31 12:38:06 #10 0x559d6d647f93 in doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long) /root/doris/be/src/runtime/fragment_mgr.cpp:792:23 12:38:06 apache#11 0x559d7068942b in doris::LongPipelineTaskAction::handle(doris::HttpRequest*) /root/doris/be/src/http/action/pipeline_task_action.cpp:54:69 12:38:06 apache#12 0x559d82c776c6 (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x385416c6) 12:38:06 apache#13 0x559d82c5775f in bufferevent_run_readcb_ /home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/bufferevent.c:251:3 12:38:06 apache#14 0x559d82c79802 in bufferevent_trigger_nolock_ /home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/bufferevent-internal.h:411:3 12:38:06 apache#15 0x559d82c79802 in bufferevent_readcb /home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/bufferevent_sock.c:214:2 12:38:06 apache#16 0x559d82c60724 in event_persist_closure /home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/event.c:1623:9 12:38:06 apache#17 0x559d82c60724 in event_process_active_single_queue /home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/event.c:1682:4 12:38:06 apache#18 0x559d82c60d76 in event_process_active /home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/event.c:1783:9 12:38:06 + echo 'cp -r /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0//Cluster0/fe/log /home/work/pipline/backup_center/56368_04d746711764283b1cbafe40b1f849e10a5bcb6e_p0/fe/' 12:38:06 apache#19 0x559d82c6355f in event_base_loop.constprop.0 /home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/event.c:2006:12 12:38:06 apache#20 0x559d7072b08c in doris::EvHttpServer::start()::$_0::operator()() const /root/doris/be/src/http/ev_http_server.cpp:139:13 12:38:06 + cp -r /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0//Cluster0/fe/log /home/work/pipline/backup_center/56368_04d746711764283b1cbafe40b1f849e10a5bcb6e_p0/fe/ 12:38:06 apache#21 0x559d7072b08c in void std::__invoke_impl<void, doris::EvHttpServer::start()::$_0&>(std::__invoke_other, doris::EvHttpServer::start()::$_0&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 12:38:06 apache#22 0x559d7072b08c in std::enable_if<is_invocable_r_v<void, doris::EvHttpServer::start()::$_0&>, void>::type std::__invoke_r<void, doris::EvHttpServer::start()::$_0&>(doris::EvHttpServer::start()::$_0&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113:2 12:38:06 apache#23 0x559d7072b08c in std::_Function_handler<void (), doris::EvHttpServer::start()::$_0>::_M_invoke(std::_Any_data const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 12:38:06 apache#24 0x559d6e121772 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:614:24 12:38:06 apache#25 0x559d6e0fe7e6 in std::function<void ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 12:38:06 apache#26 0x559d6e0fe7e6 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:460:5 12:38:06 apache#27 0x559d6984ed26 in asan_thread_start(void*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x1f118d26) 12:38:06 apache#28 0x7f83909ee608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 12:38:06 apache#29 0x7f8390901132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
xy720
pushed a commit
that referenced
this pull request
Oct 16, 2025
…sMgr::report_runtime_query_statistics (apache#56744) try to avoid asan check fail: ```cpp OpenJDK 64-Bit Server VM warning: Option CriticalJNINatives was deprecated in version 16.0 and will likely be removed in a future release. Listening for transport dt_socket at address: 8871 WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. start BE in local mode thrift error, reason=write() send(): Broken pipe================================================================= AddressSanitizer: CHECK failed: asan_thread.cpp:369 "((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0) (tid=13525) #0 0x557a503468e1 in __asan::CheckUnwind() (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f2a08e1) #1 0x557a50361182 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f2bb182) #2 0x557a5034a97b in __asan::AsanThread::GetStackFrameAccessByAddr(unsigned long, __asan::AsanThread::StackFrameAccess*) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f2a497b) #3 0x557a502a4a30 in __asan::AddressDescription::AddressDescription(unsigned long, unsigned long, bool) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f1fea30) #4 0x557a502a67a3 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f2007a3) #5 0x557a50343878 in __asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f29d878) #6 0x557a50344f75 in __asan_report_load8 (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f29ef75) #7 0x557a5039791e in std::__cxx11::basic_string, std::allocator>::_M_create(unsigned long&, unsigned long) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.tcc:148:11 #8 0x557a503984db in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, unsigned long) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.tcc:291:12 #9 0x557a503984db in std::__cxx11::basic_string, std::allocator>::basic_string(std::__cxx11::basic_string, std::allocator> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.h:617:2 #10 0x557a55b4d77d in std::pair, std::allocator> const, doris::TQueryStatistics>::pair(std::pair, std::allocator> const, doris::TQueryStatistics> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_pair.h:312:17 apache#11 0x557a55b4d77d in _ZSt12construct_atISt4pairIKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEN5doris16TQueryStatisticsEEJRKSA_EQaant20is_unbounded_array_vIT_ErqXgsnwcvPvLi0E_SD_pispclsr3stdE7declvalIT0_EEEEEPSD_SG_DpOSF_ /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_construct.h:110:22 apache#12 0x557a55b4d77d in void std::allocator_traits, std::allocator> const, doris::TQueryStatistics>>>>::construct, std::allocator> const, doris::TQueryStatistics>, std::pair, std::allocator> const, doris::TQueryStatistics> const&>(std::allocator, std::allocator> const, doris::TQueryStatistics>>>&, std::pair, std::allocator> const, doris::TQueryStatistics>*, std::pair, std::allocator> const, doris::TQueryStatistics> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/alloc_traits.h:676:4 apache#13 0x557a55b4d77d in void std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_construct_node, std::allocator> const, doris::TQueryStatistics> const&>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::pair, std::allocator> const, doris::TQueryStatistics> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:1237:8 apache#14 0x557a55b4cfe3 in std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node::operator(), std::allocator> const, doris::TQueryStatistics> const&>(std::pair, std::allocator> const, doris::TQueryStatistics> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h apache#15 0x557a55b4cbe6 in std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_clone_node, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:1287:8 apache#16 0x557a55b4c6f6 in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree_node_base*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2546:20 apache#17 0x557a55b4c7b6 in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree_node_base*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2554:3 apache#18 0x557a55b4c7b6 in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree_node_base*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2554:3 apache#19 0x557a55b4c7b6 in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree_node_base*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2554:3 apache#20 0x557a55b4c7b6 in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree_node_base*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2554:3 apache#21 0x557a55b4c27a in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>> const&, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:1523:6 apache#22 0x557a55740d4f in std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::operator=(std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2453:18 apache#23 0x557a55740d4f in std::map, std::allocator>, doris::TQueryStatistics, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::operator=(std::map, std::allocator>, doris::TQueryStatistics, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_map.h:354:37 apache#24 0x557a55740d4f in doris::TReportWorkloadRuntimeStatusParams::__set_query_statistics_map(std::map, std::allocator>, doris::TQueryStatistics, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>> const&) /home/zcp/repo_center/doris_branch-4.0/doris/gensrc/build/gen_cpp/FrontendService_types.cpp:4391:30 apache#25 0x557a5447e13a in doris::RuntimeQueryStatisticsMgr::report_runtime_query_statistics() /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/runtime_query_statistics_mgr.cpp:403:31 apache#26 0x557a504f1192 in doris::Daemon::report_runtime_query_statistics_thread() /home/zcp/repo_center/doris_branch-4.0/doris/be/src/common/daemon.cpp:457:65 apache#27 0x557a54bf2eb6 in std::function::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#28 0x557a54bf2eb6 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_branch-4.0/doris/be/src/util/thread.cpp:460:5 apache#29 0x557a50336d26 in asan_thread_start(void*) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f290d26) apache#30 0x7f12a399eac2 in start_thread nptl/pthread_create.c:442:8 apache#31 0x7f12a3a3084f in __closefrom_fallback misc/../sysdeps/unix/sysv/linux/closefrom_fallback.c:45:19 172.20.59.34 last coredump sql: last SQL query not found ```
xy720
pushed a commit
that referenced
this pull request
Oct 16, 2025
…e#56935) ``` (gdb) bt #0 doris::cloud::FdbTxnKv::get_partition_boundaries (this=0x7f11ac6e3950, boundaries=0x7f1130017f20) at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/meta-store/txn_kv.cpp:212 #1 0x000055e494de9276 in doris::cloud::export_fdb_kv_ranges_details (kv=0x7f11ac6e3950) at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/common/metric.cpp:358 #2 doris::cloud::FdbMetricExporter::export_fdb_metrics (txn_kv=0x7f11ac6e3950) at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/common/metric.cpp:393 #3 0x000055e494dec77d in doris::cloud::FdbMetricExporter::start()::$_0::operator()() const (this=<optimized out>) at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/common/metric.cpp:415 #4 std::__invoke_impl<void, doris::cloud::FdbMetricExporter::start()::$_0>(std::__invoke_other, doris::cloud::FdbMetricExporter::start()::$_0&&) (__f=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63 #5 std::__invoke<doris::cloud::FdbMetricExporter::start()::$_0>(doris::cloud::FdbMetricExporter::start()::$_0&&) (__fn=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:98 #6 std::thread::_Invoker<std::tuple<doris::cloud::FdbMetricExporter::start()::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_thread.h:303 #7 std::thread::_Invoker<std::tuple<doris::cloud::FdbMetricExporter::start()::$_0> >::operator()() (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_thread.h:310 #8 std::thread::_State_impl<std::thread::_Invoker<std::tuple<doris::cloud::FdbMetricExporter::start()::$_0> > >::_M_run() (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_thread.h:255 #9 0x000055e495e547b0 in execute_native_thread_routine () #10 0x00007f11b1dfeac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 apache#11 0x00007f11b1e90850 in __closefrom_fallback (from=-1266337808, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45 apache#12 0x0000000000000000 in ?? () ```
xy720
pushed a commit
that referenced
this pull request
Oct 16, 2025
…56755) ### What problem does this PR solve? ```text #0 __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ./nptl/pthread_sigmask.c:43 #1 0x00007f91fd6c471e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so #2 0x00007f91fd6c5206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so #3 <signal handler called> #4 __gnu_cxx::__exchange_and_add (__mem=0xe, __val=-1) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/ext/atomicity.h:68 #5 __gnu_cxx::__exchange_and_add_dispatch (__mem=0xe, __val=-1) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/ext/atomicity.h:103 #6 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_weak_release (this=0x2) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:211 #7 std::__weak_count<(__gnu_cxx::_Lock_policy)2>::~__weak_count (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1168 #8 std::__weak_ptr<doris::QueryContext, (__gnu_cxx::_Lock_policy)2>::~__weak_ptr (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:2003 #9 doris::QueryTaskController::revoke_memory()::$_1::~$_1() (this=<optimized out>) at /root/doris/be/src/runtime/workload_management/query_task_controller.cpp:168 #10 std::_Function_base::_Base_manager<doris::QueryTaskController::revoke_memory()::$_1>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) (__victim=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:177 apache#11 std::_Function_base::_Base_manager<doris::QueryTaskController::revoke_memory()::$_1>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) (__dest=..., __op=std::__destroy_functor, __source=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:205 apache#12 std::_Function_handler<void (doris::pipeline::SpillContext*), doris::QueryTaskController::revoke_memory()::$_1>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) (__dest=..., __source=..., __op=std::__destroy_functor) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:284 apache#13 0x000055f37fba60c5 in std::_Function_base::~_Function_base (this=0x7f8dbb0c9fe0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:246 apache#14 doris::pipeline::SpillContext::~SpillContext (this=0x7f91536f0150) at /root/doris/be/src/pipeline/exec/spill_utils.h:57 apache#15 0x000055f384259e4b in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7f91536f0140) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:345 apache#16 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::operator= (this=0x7f8d61302518, __r=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1088 apache#17 std::__shared_ptr<doris::pipeline::SpillContext, (__gnu_cxx::_Lock_policy)2>::operator= (this=0x7f8d61302510) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1530 apache#18 std::shared_ptr<doris::pipeline::SpillContext>::operator= (this=0x7f8d61302510) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr.h:413 apache#19 doris::pipeline::PipelineTask::revoke_memory (this=0x7f8d61302490, spill_context=...) at /root/doris/be/src/pipeline/pipeline_task.cpp:812 apache#20 0x000055f37fba3c2d in doris::QueryTaskController::revoke_memory (this=<optimized out>) at /root/doris/be/src/runtime/workload_management/query_task_controller.cpp:185 apache#21 0x000055f37fb99d08 in doris::WorkloadGroupMgr::handle_single_query_ (this=<optimized out>, requestor=..., size_to_reserve=size_to_reserve@entry=1024000, time_in_queue=time_in_queue@entry=27, paused_reason=...) at /root/doris/be/src/runtime/workload_group/workload_group_manager.cpp:820 apache#22 0x000055f37fb981d0 in doris::WorkloadGroupMgr::handle_paused_queries (this=0x7f9141d8a800) at /root/doris/be/src/runtime/workload_group/workload_group_manager.cpp:381 apache#23 0x000055f37eb97297 in doris::Daemon::memory_maintenance_thread (this=0x7ffe63cad730) at /root/doris/be/src/common/daemon.cpp:354 apache#24 0x000055f37fd812fc in std::function<void ()>::operator()() const (this=0x7f8dbb0c9fe0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593 apache#25 doris::Thread::supervise_thread (arg=0x7f914ba25e10) at /root/doris/be/src/util/thread.cpp:460 apache#26 0x00007f91fc75fac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 apache#27 0x00007f91fc7f1850 in __closefrom_fallback (from=1674236160, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45 apache#28 0x0000000000000000 in ?? () ``` Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
xy720
pushed a commit
that referenced
this pull request
Oct 23, 2025
…icate (apache#57100) ### What problem does this PR solve? ```text #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140649357411904) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=140649357411904) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=140649357411904, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x00007feca2a42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x00007feca2a287f3 in __GI_abort () at ./stdlib/abort.c:79 #5 0x000055cd63341b52 in __gnu_cxx::__verbose_terminate_handler() () #6 0x000055cd6333fe26 in __cxxabiv1::__terminate(void (*)()) () #7 0x000055cd6333fe79 in std::terminate() () #8 0x000055cd6333ffc3 in __cxa_throw () #9 0x000055cd6339556a in std::__throw_out_of_range(char const*) () #10 0x000055cd5a9a7a68 in __gnu_cxx::__stoa<double, double, char> (__str=0x7fe842726fc0 "9.", '9' <repeats 15 times>, "1e-309", __idx=0x0, __convf=<optimized out>, __name=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/ext/string_conversions.h:90 apache#11 std::__cxx11::stod (__str=..., __idx=0x0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.h:4490 apache#12 doris::IntegerPredicateCreator<(doris::PrimitiveType)9, (doris::PredicateType)1, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::convert (condition=...) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:68 apache#13 doris::IntegerPredicateCreator<(doris::PrimitiveType)9, (doris::PredicateType)1, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::create (this=<optimized out>, column=..., index=1, conditions=..., opposite=false, arena=...) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:59 apache#14 0x000055cd5a3dd63b in doris::create_predicate<(doris::PredicateType)1, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (column=..., index=1, conditions=..., opposite=false, arena=...) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:270 apache#15 doris::create_comparison_predicate<(doris::PredicateType)1> (column=..., index=1, condition=..., opposite=false, arena=...) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:278 apache#16 0x000055cd5a3dd435 in doris::parse_to_predicate (column=..., index=1, condition=..., arena=..., opposite=false) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:326 apache#17 0x000055cd5afc2759 in doris::TabletReader::_init_conditions_param (this=this@entry=0x7fe842e1a800, read_params=...) at /root/doris_branch-4.0/doris/be/src/olap/tablet_reader.cpp:549 apache#18 0x000055cd5afc1022 in doris::TabletReader::_init_params (this=this@entry=0x7fe842e1a800, read_params=...) at /root/doris_branch-4.0/doris/be/src/olap/tablet_reader.cpp:304 apache#19 0x000055cd5afbfe00 in doris::TabletReader::init (this=0x7fe842e1a800, read_params=...) at /root/doris_branch-4.0/doris/be/src/olap/tablet_reader.cpp:127 apache#20 0x000055cd5f019d0f in doris::vectorized::BlockReader::init (this=0x4c1b, read_params=...) at /root/doris_branch-4.0/doris/be/src/vec/olap/block_reader.cpp:206 apache#21 0x000055cd5f79a3f4 in doris::vectorized::OlapScanner::open (this=0x7febe2c16010, state=<optimized out>) at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/olap_scanner.cpp:278 apache#22 0x000055cd5e05303a in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=..., scan_task=...) at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182 apache#23 0x000055cd5e0549e5 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (this=<optimized out>) at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96 apache#24 doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const (this=0x7febe321dac0) at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95 apache#25 std::__invoke_impl<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__f=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63 apache#26 std::__invoke_r<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__fn=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116 apache#27 std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) ( __functor=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 apache#28 0x000055cd5b4ab5d5 in doris::ThreadPool::dispatch_thread (this=0x7fec35e01800) at /root/doris_branch-4.0/doris/be/src/util/threadpool.cpp:614 apache#29 0x000055cd5b4a06fc in std::function<void ()>::operator()() const (this=0x4a17) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593 apache#30 doris::Thread::supervise_thread (arg=0x7fec35872110) at /root/doris_branch-4.0/doris/be/src/util/thread.cpp:460 apache#31 0x00007feca2a94ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 apache#32 0x00007feca2b26850 in __closefrom_fallback (from=-823527648, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45 apache#33 0x0000000000000000 in ?? () ``` Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
xy720
pushed a commit
that referenced
this pull request
Oct 23, 2025
### What problem does this PR solve? ```text #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=136182904088128) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=136182904088128) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=136182904088128, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x00007fdee8ffb476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x00007fdee8fe17f3 in __GI_abort () at ./stdlib/abort.c:79 #5 0x000055cc31fb9bcd in ?? () #6 0x000055cc31fadeac in google::LogMessage::SendToLog() () #7 0x000055cc31fae538 in google::LogMessage::Flush() () #8 0x000055cc31fb2116 in google::LogMessageFatal::~LogMessageFatal() () #9 0x000055cc1977f798 in doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>::operator[] (this=<optimized out>, n=2) at /root/doris/be/src/vec/common/pod_array.h:466 #10 0x000055cc19777580 in doris::vectorized::ColumnStr<unsigned int>::size_at (this=0x7c5eea94dda0, i=2) at /root/doris/be/src/vec/columns/column_string.h:96 apache#11 doris::vectorized::ColumnStr<unsigned int>::compare_at (this=0x7c5eea94dda0, n=2, m=1, rhs_=...) at /root/doris/be/src/vec/columns/column_string.h:482 apache#12 0x000055cc2f2f1138 in doris::vectorized::VCollectIterator::_topn_next (this=<optimized out>, block=<optimized out>) at /root/doris/be/src/vec/olap/vcollect_iterator.cpp:343 apache#13 0x000055cc2f2efa08 in doris::vectorized::VCollectIterator::next (this=0x898f, block=0x6) at /root/doris/be/src/vec/olap/vcollect_iterator.cpp:247 apache#14 0x000055cc2f28e3ea in doris::vectorized::BlockReader::_direct_next_block (this=0x7dbeed736880, block=<optimized out>, eof=<optimized out>) at /root/doris/be/src/vec/olap/block_reader.cpp:267 apache#15 0x000055cc2f28778b in doris::vectorized::BlockReader::next_block_with_aggregation (this=0x7dbeed736880, block=<optimized out>, eof=<optimized out>) at /root/doris/be/src/vec/olap/block_reader.cpp:66 apache#16 0x000055cc2b33300e in doris::vectorized::OlapScanner::_get_block_impl (this=0x7daeee690690, state=<optimized out>, block=0x7daeee6907b8, eof=<optimized out>) at /root/doris/be/src/vec/exec/scan/olap_scanner.cpp:584 apache#17 0x000055cc2b2c990d in doris::vectorized::Scanner::get_block (this=<optimized out>, state=<optimized out>, block=0x7daeee6907b8, eof=<optimized out>) at /root/doris/be/src/vec/exec/scan/scanner.cpp:116 apache#18 0x000055cc2b2c8fa0 in doris::vectorized::Scanner::get_block_after_projects (this=0x7daeee690690, state=0x7dcee9e6ec80, block=0x7caee9b74fe0, eos=0x7bdb809a11f0) at /root/doris/be/src/vec/exec/scan/scanner.cpp:82 apache#19 0x000055cc2b2e51b4 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=..., scan_task=...) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182 apache#20 0x000055cc2b2ed92c in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (this=<optimized out>) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96 apache#21 doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const (this=0x7c0eeec42540) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95 apache#22 std::__invoke_impl<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__f=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63 apache#23 std::__invoke_r<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__fn=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116 apache#24 std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) ( __functor=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 apache#25 0x000055cc1dd09d33 in doris::ThreadPool::dispatch_thread (this=0x7d3ee8680f80) at /root/doris/be/src/util/threadpool.cpp:614 apache#26 0x000055cc1dce6da7 in std::function<void ()>::operator()() const (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593 apache#27 doris::Thread::supervise_thread (arg=<optimized out>) at /root/doris/be/src/util/thread.cpp:460 apache#28 0x000055cc19364d27 in asan_thread_start(void*) () apache#29 0x00007fdee904dac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 apache#30 0x00007fdee90df850 in __closefrom_fallback (from=877593024, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45 apache#31 0x0000000000000000 in ?? () ``` Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
xy720
pushed a commit
that referenced
this pull request
Nov 14, 2025
…ntains window function (apache#55066) ### What problem does this PR solve? Support window function rewrite when materialized view contains window function Such as mv def is as following: CREATE MATERIALIZED VIEW mv1 BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL DISTRIBUTED BY RANDOM BUCKETS 2 PROPERTIES ('replication_num' = '1') AS select * from ( select o_orderkey, FIRST_VALUE(o_custkey) OVER ( PARTITION BY o_orderdate ORDER BY o_totalprice NULLS LAST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS first_value, RANK() OVER ( PARTITION BY o_orderdate, o_orderstatus ORDER BY o_totalprice NULLS LAST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS rank_value, LAG(l_extendedprice, 1, 0) over (partition by o_orderdate, l_shipdate order by l_quantity) AS lag_value from lineitem2 left join orders2 on l_orderkey = o_orderkey and l_shipdate = o_orderdate ) t where o_orderkey > 1; if query as following, this can use mv to represent query select * from ( select o_orderkey, FIRST_VALUE(o_custkey) OVER ( PARTITION BY o_orderdate ORDER BY o_totalprice NULLS LAST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS first_value, RANK() OVER ( PARTITION BY o_orderdate, o_orderstatus ORDER BY o_totalprice NULLS LAST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS rank_value, LAG(l_extendedprice, 1, 0) over (partition by o_orderdate, l_shipdate order by l_quantity) AS lag_value from lineitem2 left join orders2 on l_orderkey = o_orderkey and l_shipdate = o_orderdate ) t where o_orderkey > 2; explain result is as follwing: +----------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | o_orderkey[#4] | | first_value[#5] | | rank_value[#6] | | lag_value[#7] | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 0:VOlapScanNode(445) | | TABLE: regression_test_nereids_rules_p0_mv_window.mv1(mv1), PREAGGREGATION: ON | | PREDICATES: (o_orderkey[#0] > 2) | | partitions=1/1 (mv1) | | tablets=2/2, tabletList=1755678381149,1755678381151 | | cardinality=1, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | final projections: o_orderkey[#0], first_value[#1], rank_value[#2], lag_value[#3] | | final project output tuple id: 1 | | | | | | ========== MATERIALIZATIONS ========== | | | | MaterializedView | | MaterializedViewRewriteSuccessAndChose: | | internal.regression_test_nereids_rules_p0_mv_window.mv1 chose | | | | MaterializedViewRewriteSuccessButNotChose: | | | | MaterializedViewRewriteFail: | | | | | | ========== STATISTICS ========== | | planed with unknown column statistics | +----------------------------------------------------------------------------------------+
xy720
added a commit
that referenced
this pull request
Nov 17, 2025
…ich belongs to an agg materialized view (apache#58038) ### What problem does this PR solve? Issue Number: close apache#58037 Problem Summary: ``` #0 0x00007f9aca4a3f8c in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x00007f9aca454a26 in raise () from /lib64/libc.so.6 #2 0x00007f9aca43d87c in abort () from /lib64/libc.so.6 #3 0x0000561dc3d1ea1d in ?? () #4 0x0000561dc3d1105a in google::LogMessage::Fail() () #5 0x0000561dc3d14146 in google::LogMessage::SendToLog() () #6 0x0000561dc3d10b90 in google::LogMessage::Flush() () #7 0x0000561dc3d14989 in google::LogMessageFatal::~LogMessageFatal() () #8 0x0000561db854c996 in assert_cast<doris::vectorized::ColumnStr<unsigned int> const&, (TypeCheckOnRelease)1, doris::vectorized::IColumn const&>(doris::vectorized::IColumn const&)::{lambda(auto:1&&)#1}::operator()<doris::vectorized::IColumn const&>(doris::vectorized::IColumn const&) const (this=this@entry=0x7f9658ccc1f8, from=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/common/assert_cast.h:58 #9 0x0000561db854c7d7 in assert_cast<doris::vectorized::ColumnStr<unsigned int> const&, (TypeCheckOnRelease)1, doris::vectorized::IColumn const&> (from=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/common/assert_cast.h:73 #10 0x0000561db854bb0b in doris::vectorized::ColumnStr<unsigned int>::compare_at (this=0x7f957a14e2c0, n=1159288, m=6, rhs_=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/columns/column_string.h:526 apache#11 0x0000561dbe108c6b in doris::vectorized::GenericComparisonImpl<doris::vectorized::EqualsOp<int, int> >::vector_constant (a=..., b=..., c=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:112 apache#12 doris::vectorized::FunctionComparison<doris::vectorized::EqualsOp, doris::vectorized::NameEquals>::execute_generic_identical_types ( this=<optimized out>, block=..., result=result@entry=10, c0=0x7f957a14e2c0, c1=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:506 apache#13 0x0000561dbdf9e97e in doris::vectorized::FunctionComparison<doris::vectorized::EqualsOp, doris::vectorized::NameEquals>::execute_generic ( this=0x7f96d6fb1b90, block=..., result=10, c0=..., c1=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:517 apache#14 doris::vectorized::FunctionComparison<doris::vectorized::EqualsOp, doris::vectorized::NameEquals>::execute_impl (this=0x7f96d6fb1b90, context=<optimized out>, block=..., arguments=..., result=10, input_rows_count=104) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:707 apache#15 0x0000561dbdcf1b8f in doris::vectorized::DefaultExecutable::execute_impl (this=<optimized out>, context=0x6, block=..., arguments=..., result=1, input_rows_count=104) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.h:472 apache#16 0x0000561dbeea76ae in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal (this=this@entry=0x7f99f62a65d0, context=context@entry=0x7f99f6442b00, block=..., args=..., result=result@entry=10, input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:121 apache#17 0x0000561dbeea4ce8 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns (this=0x7f99f62a65d0, context=0x7f99f6442b00, block=..., args=..., result=10, input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:246 apache#18 doris::vectorized::PreparedFunctionImpl::default_implementation_for_nulls (this=this@entry=0x7f99f62a65d0, context=context@entry=0x7f99f6442b00, block=..., args=..., result=result@entry=10, input_rows_count=104, dry_run=<optimized out>, executed=0x7f9658ccc666) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:218 apache#19 0x0000561dbeea4e9c in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal (this=0x7f99f62a65d0, context=0x7f99f6442b00, block=..., args=..., result=10, input_rows_count=<optimized out>, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:112 apache#20 doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns (this=0x7f99f62a65d0, context=0x7f99f6442b00, block=..., args=..., result=10, input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:246 apache#21 0x0000561dbeea4f66 in doris::vectorized::PreparedFunctionImpl::execute (this=0x11b078, context=0x6, block=..., args=..., result=1, input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:252 apache#22 0x0000561dbdcf1500 in doris::vectorized::IFunctionBase::execute (this=<optimized out>, context=0x7f99f6442b00, block=..., arguments=..., result=10, input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.h:195 --Type <RET> for more, q to quit, c to continue without paging--c apache#23 0x0000561dbdceccad in doris::vectorized::VectorizedFnCall::_do_execute (this=0x7f96f0fec510, context=0x7f957f09cdf0, block=0x7f957b02a3b0, result_column_id=0x7f9658ccca14, args=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vectorized_fn_call.cpp:197 apache#24 0x0000561dbdced2c6 in doris::vectorized::VectorizedFnCall::execute (this=0x11b078, context=0x6, block=0x7f9aca4a3f8c <__pthread_kill_implementation+268>, result_column_id=0x7f9658ccbe10) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vectorized_fn_call.cpp:212 apache#25 0x0000561dbdd1e51b in doris::vectorized::VExprContext::execute (this=0x7f957f09cdf0, block=0x7f9aca4a3f8c <__pthread_kill_implementation+268>, block@entry=0x7f957b02a3b0, result_column_id=result_column_id@entry=0x7f9658ccca14) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vexpr_context.cpp:55 apache#26 0x0000561dbdd1fcb5 in doris::vectorized::VExprContext::execute_conjuncts (ctxs=..., filters=filters@entry=0x0, accept_null=false, block=block@entry=0x7f957b02a3b0, result_filter=result_filter@entry=0x7f9658ccccc0, can_filter_all=0x7f9658cccbc7) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vexpr_context.cpp:174 apache#27 0x0000561dbdd2131f in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block (ctxs=..., block=0x7f957b02a3b0, columns_to_filter=..., column_to_keep=6, filter=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vexpr_context.cpp:354 apache#28 0x0000561db8f49450 in doris::segment_v2::SegmentIterator::_execute_common_expr (this=this@entry=0x7f954e20a000, sel_rowid_idx=0x7f955c0d2000, selected_size=@0x7f9658ccce6e: 104, block=block@entry=0x7f957b02a3b0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2338 apache#29 0x0000561db8f482f8 in doris::segment_v2::SegmentIterator::_next_batch_internal (this=0x7f954e20a000, block=0x7f957b02a3b0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2230 apache#30 0x0000561db8f45212 in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const ( this=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1953 apache#31 doris::segment_v2::SegmentIterator::next_batch (this=0x7f954e20a000, block=0x6) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1952 apache#32 0x0000561db8ee49bc in doris::segment_v2::LazyInitSegmentIterator::next_batch (this=0x7f953bc71f80, block=0x7f957b02a3b0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44 apache#33 0x0000561db8dab844 in doris::BetaRowsetReader::next_block (this=0x7f9a4e215800, block=0x7f957b02a3b0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/beta_rowset_reader.cpp:377 apache#34 0x0000561dc2c9413d in doris::vectorized::VCollectIterator::Level0Iterator::_refresh (this=0x7f953ba137a0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.h:256 apache#35 doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row (this=this@entry=0x7f953ba137a0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.cpp:509 apache#36 0x0000561dc2c93bf4 in doris::vectorized::VCollectIterator::Level0Iterator::init (this=0x7f953ba137a0, get_data_by_ref=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.cpp:461 apache#37 0x0000561dc2c91002 in doris::vectorized::VCollectIterator::build_heap (this=0x7f957a52bb30, rs_readers=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.cpp:125 apache#38 0x0000561dc2c7e1f2 in doris::vectorized::BlockReader::_init_collect_iter (this=this@entry=0x7f957a52b400, read_params=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/block_reader.cpp:153 apache#39 0x0000561dc2c7f191 in doris::vectorized::BlockReader::init (this=<optimized out>, read_params=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/block_reader.cpp:226 apache#40 0x0000561dc3937869 in doris::vectorized::NewOlapScanner::open (this=0x7f9a56270210, state=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:252 apache#41 0x0000561dbdcc5413 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=..., scan_task=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:221 apache#42 0x0000561dbdcc62bd in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (this=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:154 apache#43 doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}::operator()() const (this=0x7f954e25d3c0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:153 apache#44 std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&) (__f=...) at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 apache#45 std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&) (__fn=...) at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111 apache#46 std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 apache#47 0x0000561db962137a in doris::ThreadPool::dispatch_thread (this=0x7f9a50f9d380) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/util/threadpool.cpp:602 apache#48 0x0000561db96159a1 in std::function<void ()>::operator()() const (this=0x11a791) at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 apache#49 doris::Thread::supervise_thread (arg=0x7f969f569ce0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/util/thread.cpp:498 apache#50 0x00007f9aca4a2215 in start_thread () from /lib64/libc.so.6 apache#51 0x00007f9aca524bdc in clone3 () from /lib64/libc.so.6 ``` Assume that 0,1,2,3,4, is key columns of an AGG mv, because the PreAgg is OFF at scan node, the block will contain all key columns to merge data in storage layer. if we select 0,1 column, with 3,4 column in where clause, then the slot ids should be 0,1,3,4, and column ids in conjuncts is the index of slot ids.(which is 2 and 3) But the plan use the key type of base table which is DUP key, treating the AGG mv as a DUP mv, so these conjuncts are pushed down to the scan node which belongs to an AGG mv, these conjuncts will pick the wrong column 2 and 3 (which shoud be 4 and 5) in block to exucute. So we should use the key type of mv but not the key type of base table. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
xy720
pushed a commit
that referenced
this pull request
Nov 17, 2025
…57888) We previously had a crash. The cause is that we should not access the request after calling add_block(...) because add_block may enqueue a closure that runs on another thread and frees the request ``` ==730145==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7be1efd803a0 at pc 0x556b38d9d625 bp 0x7b16bf0193f0 sp 0x7b16bf0193e8 READ of size 4 at 0x7be1efd803a0 thread T1559 #0 0x556b38d9d624 in google::protobuf::internal::RepeatedPtrFieldBase::size() const /home/zcp/repo_center/doris_master/doris/thirdparty/installed/include/google/protobuf/repeated_ptr_field.h:185:29 #1 0x556b408ab062 in google::protobuf::RepeatedPtrField<doris::PBlock>::size() const /home/zcp/repo_center/doris_master/doris/thirdparty/installed/include/google/protobuf/repeated_ptr_field.h:1248:32 #2 0x556b408aaff4 in doris::PTransmitDataParams::_internal_blocks_size() const /home/zcp/repo_center/doris_master/doris/be/../gensrc/build/gen_cpp/internal_service.pb.h:32149:25 #3 0x556b4089731c in doris::PTransmitDataParams::blocks_size() const /home/zcp/repo_center/doris_master/doris/be/../gensrc/build/gen_cpp/internal_service.pb.h:32152:10 #4 0x556b60a83c17 in doris::vectorized::VDataStreamMgr::transmit_block(doris::PTransmitDataParams const*, google::protobuf::Closure**, long) /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_mgr.cpp:150:38 #5 0x556b407f7408 in doris::PInternalService::_transmit_block(google::protobuf::RpcController*, doris::PTransmitDataParams const*, doris::PTransmitDataResult*, google::protobuf::Closure*, doris::Status const&, long) /home/zcp/repo_center/doris_master/doris/be/src/service/internal_service.cpp:1673:40 #6 0x556b407f52bb in doris::PInternalService::transmit_block(google::protobuf::RpcController*, doris::PTransmitDataParams const*, doris::PTransmitDataResult*, google::protobuf::Closure*) /home/zcp/repo_center/doris_master/doris/be/src/service/internal_service.cpp:1610:9 #7 0x556b43fceba2 in doris::PBackendService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*) /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/internal_service.pb.cc:49452:7 #8 0x556b6736273e in brpc::policy::ProcessRpcRequest(brpc::InputMessageBase*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770bd73e) #9 0x556b67357426 in brpc::ProcessInputMessage(void*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770b2426) #10 0x556b67357f20 in brpc::InputMessenger::InputMessageClosure::~InputMessageClosure() (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770b2f20) apache#11 0x556b673588dd in brpc::InputMessenger::OnNewMessages(brpc::Socket*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770b38dd) apache#12 0x556b674a0adc in brpc::Socket::ProcessEvent(void*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x771fbadc) apache#13 0x556b672e0f76 in bthread::TaskGroup::task_runner(long) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x7703bf76) apache#14 0x556b672cbbe0 in bthread_make_fcontext (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x77026be0) ``` ``` Note: The done pointer will be saved in add_block and may be called in another thread via done->Run(). For example, when blocks_size == 1, the process is as follows: transmit_block (i=0) └─> recvr->add_block(..., done, ...) // Pass done └─> SenderQueue::add_block └─> _pending_closures.push(done) // done is saved get_batch() [another thread] └─> closure_pair.first->Run() //⚠️ done->Run() is called └─> brpc releases request and response transmit_block (i=1) [original thread continues] └─> request->blocks_size() //⚠️ request has already been released! At this point, a use-after-free issue occurs. TODO: We should consider refactoring this part because add_block may release the request. We should not access the request after calling add_block. ``` apache#50113
xy720
pushed a commit
that referenced
this pull request
Jan 6, 2026
Related issue: apache#57884 ``` MySQL [demo]> show frontends; +-----------------------------------------+--------------+-------------+----------+-----------+---------+--------------------+----------+----------+-----------+------+-------+-------------------+---------------------+---------------------+----------+--------+------------------------+------------------+---------------------+ | Name | Host | EditLogPort | HttpPort | QueryPort | RpcPort | ArrowFlightSqlPort | Role | IsMaster | ClusterId | Join | Alive | ReplayedJournalId | LastStartTime | LastHeartbeat | IsHelper | ErrMsg | Version | CurrentConnected | LiveSince | +-----------------------------------------+--------------+-------------+----------+-----------+---------+--------------------+----------+----------+-----------+------+-------+-------------------+---------------------+---------------------+----------+--------+------------------------+------------------+---------------------+ | fe_a7c0b6d8_82c2_48f0_8220_fb65dd18be69 | 10.37.75.124 | 9010 | 8030 | 9030 | 9020 | 8070 | FOLLOWER | true | 742250121 | true | true | 2409 | 2025-11-11 14:42:16 | 2025-11-11 14:44:06 | true | | doris-0.0.0-009c3b552a | Yes | 2025-11-11 14:42:16 | +-----------------------------------------+--------------+-------------+----------+-----------+---------+--------------------+----------+----------+-----------+------+-------+-------------------+---------------------+---------------------+----------+--------+------------------------+------------------+---------------------+ 1 row in set (0.016 sec) MySQL [demo]> show catalog edoris; +-----------------+-------------------------------+ | Key | Value | +-----------------+-------------------------------+ | create_time | 2025-11-11 11:25:33.488106853 | | fe_arrow_hosts | 10.37.103.28:8070 | | fe_http_hosts | 10.37.103.28:8030 | | fe_thrift_hosts | 10.37.103.28:9020 | | password | *XXX | | type | doris | | use_meta_cache | true | | user | test | +-----------------+-------------------------------+ 8 rows in set (0.002 sec) MySQL [demo]> select * from inner_table; +----------+--------+ | log_type | reason | +----------+--------+ | 2 | inner2 | | 3 | inner3 | | 4 | inner4 | +----------+--------+ 3 rows in set (0.032 sec) MySQL [demo]> select * from edoris.external.example_tbl_duplicate; +---------------------+----------+------------+-----------+-------+---------------------+ | log_time | log_type | error_code | error_msg | op_id | op_time | +---------------------+----------+------------+-----------+-------+---------------------+ | 2024-11-01 00:00:00 | 2 | 2 | timeout | 12 | 2024-11-01 01:00:00 | +---------------------+----------+------------+-----------+-------+---------------------+ 1 row in set (0.059 sec) MySQL [demo]> select * from inner_table a join edoris.external.example_tbl_duplicate b on (a.log_type = b.log_type); +----------+--------+---------------------+----------+------------+-----------+-------+---------------------+ | log_type | reason | log_time | log_type | error_code | error_msg | op_id | op_time | +----------+--------+---------------------+----------+------------+-----------+-------+---------------------+ | 2 | inner2 | 2024-11-01 00:00:00 | 2 | 2 | timeout | 12 | 2024-11-01 01:00:00 | +----------+--------+---------------------+----------+------------+-----------+-------+---------------------+ 1 row in set (0.050 sec) MySQL [demo]> explain select * from inner_table a join edoris.external.example_tbl_duplicate b on (a.log_type = b.log_type) where error_code=2; +-------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +-------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | log_type[apache#16] | | reason[apache#17] | | log_time[apache#18] | | log_type[apache#19] | | error_code[apache#20] | | error_msg[apache#21] | | op_id[apache#22] | | op_time[apache#23] | | PARTITION: HASH_PARTITIONED: log_type[#6] | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCOL | | | | 3:VHASH JOIN(200) | | | join op: INNER JOIN(BROADCAST)[] | | | equal join conjunct: (log_type[#6] = log_type[#1]) | | | cardinality=3 | | | vec output tuple id: 3 | | | output tuple id: 3 | | | vIntermediate tuple ids: 2 | | | hash output slot ids: 0 1 2 3 4 5 6 7 | | | runtime filters: RF000[min_max] <- log_type[#1](1/1/1048576), RF001[in_or_bloom] <- log_type[#1](1/1/1048576) | | | final projections: log_type[#8], reason[#9], log_time[#10], log_type[apache#11], error_code[apache#12], error_msg[apache#13], op_id[apache#14], op_time[apache#15] | | | final project output tuple id: 3 | | | distribute expr lists: log_type[#6] | | | distribute expr lists: | | | | | |----1:VEXCHANGE | | | offset: 0 | | | distribute expr lists: log_type[#1] | | | | | 2:VOlapScanNode(187) | | TABLE: demo.inner_table(inner_table), PREAGGREGATION: ON | | partitions=1/1 (inner_table) | | tablets=1/1, tabletList=1762832514491 | | cardinality=3, avgRowSize=901.6666, numNodes=1 | | pushAggOp=NONE | | runtime filters: RF000[min_max] -> log_type[#6], RF001[in_or_bloom] -> log_type[#6] | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: log_type[#1] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | UNPARTITIONED | | | | 0:VOlapScanNode(188) | | TABLE: external.example_tbl_duplicate(example_tbl_duplicate), PREAGGREGATION: ON | | PREDICATES: (error_code[#2] = 2) | | partitions=1/1 (example_tbl_duplicate) | | tablets=1/1, tabletList=1762481736238 | | cardinality=1, avgRowSize=7425.0, numNodes=1 | | pushAggOp=NONE | | | | | | | | ========== STATISTICS ========== | | planed with unknown column statistics | +-------------------------------------------------------------------------------------------------------------------------------------------+ 65 rows in set (0.040 sec) ```
xy720
pushed a commit
that referenced
this pull request
Jan 6, 2026
…shard_ptr not initialized (apache#58751) ``` #0 0x000055f11470ff29 in std::__shared_ptr<doris::TabletMeta, (__gnu_cxx::_Lock_policy)2>::get (this=0x148) at /home/work/env/ldb_toolchain_master/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1673 #1 std::__shared_ptr_access<doris::TabletMeta, (__gnu_cxx::_Lock_policy)2, false, false>::_M_get (this=0x148) at /home/work/env/ldb_toolchain_master/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1370 #2 std::__shared_ptr_access<doris::TabletMeta, (__gnu_cxx::_Lock_policy)2, false, false>::operator-> (this=0x148) at /home/work/env/ldb_toolchain_master/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1364 #3 doris::BaseTablet::tablet_id (this=0x0) at /home/work/doris/be/src/olap/base_tablet.h:73 #4 doris::segment_v2::SegmentWriter::finalize (this=0x7fa160bde000, segment_file_size=0x7fa564dcc668, index_size=0x7fa564dcc5b8) at /home/work/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1044 #5 0x000055f11442578c in doris::SegmentFlusher::_flush_segment_writer (this=0x7fa160b8a420, writer=..., flush_size=flush_size@entry=0x0) at /home/work/doris/be/src/olap/rowset/segment_creator.cpp:304 #6 0x000055f114426c4e in doris::SegmentFlusher::Writer::flush (this=<optimized out>) at /home/work/doris/be/src/olap/rowset/segment_creator.cpp:376 #7 doris::SegmentCreator::flush (this=0x7fa160b8a418) at /home/work/doris/be/src/olap/rowset/segment_creator.cpp:422 #8 0x000055f1143d4785 in doris::BaseBetaRowsetWriter::flush (this=<optimized out>) at /home/work/doris/be/src/olap/rowset/beta_rowset_writer.cpp:723 #9 0x000055f11436bf47 in doris::Merger::vmerge_rowsets (tablet=..., reader_type=<optimized out>, cur_tablet_schema=..., src_rowset_readers=..., dst_rowset_writer=0x7fa160b8a000, stats_output=0x7fa1df108298) at /home/work/doris/be/src/olap/merger.cpp:159 #10 0x000055f114345595 in doris::Compaction::merge_input_rowsets (this=this@entry=0x7fa1df108210) at /home/work/doris/be/src/olap/compaction.cpp:220 apache#11 0x000055f1143550d2 in doris::CloudCompactionMixin::execute_compact_impl (this=this@entry=0x7fa1df108210, permits=permits@entry=6) at /home/work/doris/be/src/olap/compaction.cpp:1491 apache#12 0x000055f114342bc1 in doris::CloudCompactionMixin::execute_compact (this=0x7fa1df108210) at /home/work/doris/be/src/olap/compaction.cpp:1620 apache#13 0x000055f11a22c6d8 in doris::CloudCumulativeCompaction::execute_compact (this=0x7fa1df108210) at /home/work/doris/be/src/cloud/cloud_cumulative_compaction.cpp:203 ```
xy720
pushed a commit
that referenced
this pull request
Jan 6, 2026
…e#59098) ### What problem does this PR solve? Introduced by apache#58905 ==2037076==ERROR: AddressSanitizer: heap-use-after-free on address 0x7baaae908730 at pc 0x561b769a1fd0 bp 0x7b3caf4ebdf0 sp 0x7b3caf4ebde8 22:30:08 READ of size 1 at 0x7baaae908730 thread T12303 (rs_normal [work) 22:30:08 #0 0x561b769a1fcf in doris::(anonymous namespace)::string_compare(char const*, long, char const*, long, long) /root/doris/be/src/vec/common/string_ref.h:170:29 22:30:08 #1 0x561b769a1fcf in doris::StringRef::compare(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:259:30 22:30:08 #2 0x561b76f537cd in doris::StringRef::ge(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:282:52 22:30:08 #3 0x561b76f537cd in doris::StringRef::operator>=(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:292:60 22:30:08 #4 0x561b76f537cd in bool doris::Compare::greater_equal<doris::StringRef>(doris::StringRef const&, doris::StringRef const&) /root/doris/be/src/common/compare.h:42:18 22:30:08 #5 0x561b76f537cd in doris::ComparisonPredicateBase<(doris::PrimitiveType)23, (doris::PredicateType)6>::camp_field(doris::vectorized::Field const&, doris::vectorized::Field const&) const /root/doris/be/src/olap/comparison_predicate.h:192:20 22:30:08 #6 0x561b76f4baa4 in doris::ComparisonPredicateBase<(doris::PrimitiveType)23, (doris::PredicateType)6>::evaluate_and(doris::vectorized::ParquetPredicate::ColumnStat*) const /root/doris/be/src/olap/comparison_predicate.h:207:26 22:30:08 #7 0x561b76765284 in doris::AndBlockColumnPredicate::evaluate_and(doris::vectorized::ParquetPredicate::ColumnStat*) const /root/doris/be/src/olap/block_column_predicate.h:251:42 22:30:08 #8 0x561b89acd735 in doris::vectorized::ParquetReader::_process_column_stat_filter(tparquet::RowGroup const&, std::vector<std::unique_ptr<doris::MutilColumnBlockPredicate, std::default_delete<doris::MutilColumnBlockPredicate> >, std::allocator<std::unique_ptr<doris::MutilColumnBlockPredicate, std::default_delete<doris::MutilColumnBlockPredicate> > > > const&, bool*, bool*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:1225:25 22:30:08 #9 0x561b89ac8dd7 in doris::vectorized::ParquetReader::_process_min_max_bloom_filter(doris::vectorized::RowGroupReader::RowGroupIndex const&, tparquet::RowGroup const&, std::vector<std::unique_ptr<doris::MutilColumnBlockPredicate, std::default_delete<doris::MutilColumnBlockPredicate> >, std::allocator<std::unique_ptr<doris::MutilColumnBlockPredicate, std::default_delete<doris::MutilColumnBlockPredicate> > > > const&, doris::segment_v2::RowRanges*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:1108:9 22:30:08 #10 0x561b89ac3e73 in doris::vectorized::ParquetReader::_next_row_group_reader() /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:718:9 22:30:08 apache#11 0x561b89ac008f in doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:607:21 22:30:08 apache#12 0x561b8a07c6f7 in doris::vectorized::HiveReader::get_next_block_inner(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/table/hive_reader.cpp:32:5 22:30:08 apache#13 0x561b89fee256 in doris::vectorized::TableFormatReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/table/table_format_reader.h:81:16 22:30:08 apache#14 0x561b89f71b97 in doris::vectorized::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/file_scanner.cpp:472:13 22:30:08 apache#15 0x561b89f7086f in doris::vectorized::FileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/file_scanner.cpp:409:17 22:30:08 apache#16 0x561b8a19f86e in doris::vectorized::Scanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/scanner.cpp:109:17 22:30:08 apache#17 0x561b8a19f0a6 in doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/scanner.cpp:85:16 22:30:08 apache#18 0x561b8a1ccd0f in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:173:5 22:30:08 apache#19 0x561b8a1d6875 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:76:17 22:30:08 apache#20 0x561b8a1d6875 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:75:27 22:30:08 apache#21 0x561b8a1d6875 in bool std::__invoke_impl<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 22:30:08 apache#22 0x561b8a1d6875 in std::enable_if<is_invocable_r_v<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>, bool>::type std::__invoke_r<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9 22:30:08 apache#23 0x561b8a1d6875 in std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 22:30:08 apache#24 0x561b8a1d5f07 in std::function<bool ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 22:30:08 apache#25 0x561b8a1d5f07 in doris::vectorized::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:407:25 22:30:08 apache#26 0x561b8a2c56d4 in doris::vectorized::PrioritizedSplitRunner::process() /root/doris/be/src/vec/exec/executor/time_sharing/prioritized_split_runner.cpp:103:35 22:30:08 apache#27 0x561b8a29045c in doris::vectorized::TimeSharingTaskExecutor::_dispatch_thread() /root/doris/be/src/vec/exec/executor/time_sharing/time_sharing_task_executor.cpp:570:77 22:30:08 apache#28 0x561b7b9fecb6 in std::function<void ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 22:30:08 apache#29 0x561b7b9fecb6 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:460:5 22:30:08 apache#30 0x561b76044d26 in asan_thread_start(void*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P1/Cluster0/be/lib/doris_be+0x23962d26) 22:30:08 apache#31 0x7f4aaae68608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 22:30:08 apache#32 0x7f4aaad7b132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.