Skip to content

Conversation

@mrhhsg
Copy link
Member

@mrhhsg mrhhsg commented Nov 22, 2023

Proposed changes

*** SIGSEGV address not mapped to object (@0x0) received by PID 428538 (TID 429389 OR 0x7fa30a60a700) from PID 0; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:417
 1# os::Linux::chained_handler(int, siginfo*, void*) in /root/ldb/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /root/ldb/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /root/ldb/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007FA45F2FBB50 in /lib64/libc.so.6
 5# doris::vectorized::is_column_const(doris::vectorized::IColumn const&) at /root/doris/be/src/vec/columns/column.cpp:80
 6# doris::vectorized::PreparedFunctionImpl::default_implementation_for_constant_arguments(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool, bool*) at /root/doris/be/src/vec/functions/function.cpp:171
 7# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) in /root/doris/be/lib/doris_be
 8# doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) at /root/doris/be/src/vec/functions/function.cpp:268
 9# doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) at /root/doris/be/src/vec/functions/function.h:172
10# doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) at /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:159
11# doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) at /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:140
12# doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) at /root/doris/be/src/vec/exprs/vexpr_context.cpp:60
13# doris::vectorized::VExprContext::execute_conjuncts(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, std::vector<doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false>, 15ul, 16ul>*, std::allocator<doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false>, 15ul, 16ul>*> > const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false>, 15ul, 16ul>*, bool*) at /root/doris/be/src/vec/exprs/vexpr_context.cpp:169
14# doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, doris::vectorized::Block*, std::vector<unsigned int, std::allocator<unsigned int> >&, int, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false>, 15ul, 16ul>&) at /root/doris/be/src/vec/exprs/vexpr_context.cpp:278
15# doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) at /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2028
16# doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1922
17# doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1781
18# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) at /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:287
19# doris::vectorized::VCollectIterator::_topn_next(doris::vectorized::Block*) at /root/doris/be/src/vec/olap/vcollect_iterator.cpp:294
20# doris::vectorized::VCollectIterator::next(doris::vectorized::Block*) in /root/doris/be/lib/doris_be
21# doris::vectorized::BlockReader::_direct_next_block(doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/olap/block_reader.cpp:258
22# doris::vectorized::BlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/olap/block_reader.h:52
23# doris::vectorized::NewOlapScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:471
24# doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) in /root/doris/be/lib/doris_be
25# doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::VScanner>) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:337
26# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::{lambda()#2}>::_M_invoke(std::_Any_data const&) at /root/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
27# doris::WorkThreadPool<true>::work_thread(int) at /root/doris/be/src/util/work_thread_pool.hpp:160
28# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
29# start_thread in /lib64/libpthread.so.0
30# __clone in /lib64/libc.so.6 

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@mrhhsg
Copy link
Member Author

mrhhsg commented Nov 22, 2023

run buildall

@wm1581066 wm1581066 added dev/2.0.3 usercase Important user case type label labels Nov 22, 2023
@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.41 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 30 seconds loaded 2358488459 Bytes, about 74 MB/s
stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17101636323 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit cc6efff948d7540be1252bec8ec8e8469d0cf59a, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4861	4652	4648	4648
q2	355	173	154	154
q3	2036	1948	1951	1948
q4	1381	1322	1248	1248
q5	3977	3917	3972	3917
q6	242	131	134	131
q7	1431	885	895	885
q8	2733	2775	2742	2742
q9	9645	9665	9574	9574
q10	3488	3522	3529	3522
q11	374	253	240	240
q12	436	290	291	290
q13	4540	3815	3780	3780
q14	319	292	300	292
q15	594	530	521	521
q16	661	585	582	582
q17	1133	984	943	943
q18	7795	7387	7309	7309
q19	1665	1672	1664	1664
q20	568	311	308	308
q21	4348	3910	3983	3910
q22	484	366	373	366
Total cold run time: 53066 ms
Total hot run time: 48974 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4589	4563	4605	4563
q2	343	229	281	229
q3	4024	3998	3996	3996
q4	2687	2671	2684	2671
q5	9709	9586	9510	9510
q6	241	121	123	121
q7	2985	2473	2451	2451
q8	4486	4472	4464	4464
q9	13164	13155	13054	13054
q10	4087	4189	4207	4189
q11	786	676	651	651
q12	989	812	814	812
q13	4269	3585	3552	3552
q14	382	346	334	334
q15	584	515	531	515
q16	730	660	669	660
q17	3954	3879	3835	3835
q18	9613	9096	8958	8958
q19	1778	1759	1758	1758
q20	2364	2066	2043	2043
q21	8739	8645	8484	8484
q22	846	805	795	795
Total cold run time: 81349 ms
Total hot run time: 77645 ms

Copy link
Contributor

@starocean999 starocean999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 22, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 0302a9d into apache:master Nov 22, 2023
@mrhhsg mrhhsg deleted the fix_having_with_topn branch November 23, 2023 01:43
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Nov 27, 2023
eldenmoon added a commit that referenced this pull request Nov 27, 2023
* [fix](stats) Fix update rows for unique table didn't get updated properly #26968 (#27337)

* [FIX](jsonb) fix jsonb in predict column #27325 (#27424)

* [fix](fe) slots in having clause should be set to need materialized(#27412) (#27429)

* [Bug](insert)fix insert wrong data on mv when stmt have multiple values (#27297) (#27382)

fix insert wrong data on mv when stmt have multiple values

* [fix](fe ut) Fix OlapQueryCacheTest failed (#27305) (#27406)

1.
```
java.lang.NullPointerException: null
        at org.apache.doris.catalog.Env.getCurrentSystemInfo(Env.java:793) ~[classes/:?]
        at org.apache.doris.qe.SimpleScheduler$UpdateBlacklistThread.run(SimpleScheduler.java:206) ~[classes/:?]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]

java.lang.NullPointerException
        at org.apache.doris.qe.OlapQueryCacheTest.setUp(OlapQueryCacheTest.java:226)
```

2.
```
[ERROR] testSqlCacheKeyWithNestedViewForNereids  Time elapsed: 1.962 s  <<< FAILURE!
java.lang.AssertionError: SELECT command denied to user 'testCluster:testUser'@'192.168.1.1' for table 'internal: testCluster:testDb: appevent'
	at org.apache.doris.qe.OlapQueryCacheTest.parseSqlByNereids(OlapQueryCacheTest.java:579)
	at org.apache.doris.qe.OlapQueryCacheTest.testSqlCacheKeyWithNestedViewForNereids(OlapQueryCacheTest.java:1338)
```

3.
```
[ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 113.63 s <<< FAILURE! - in org.apache.doris.qe.OlapQueryCacheTest
[ERROR] testCacheModeTable  Time elapsed: 1.657 s  <<< ERROR!
java.lang.IllegalArgumentException: Value of type org.apache.doris.qe.QueryState incompatible with return type org.apache.doris.system.SystemInfoService of org.apache.doris.catalog.Env#getCurrentSystemInfo()
        at org.apache.doris.qe.OlapQueryCacheTest.setUp(OlapQueryCacheTest.java:156)
```

* [regression test](schema change) add some schema change regression cases (#27112) (#27418)

* [fix](Nereids) result type of add precision is 1 more than expected (#27136) (#27426)

* [fix](Nereids): fill miss slot in having subquery (#27177) (#27394)

* [fix](memory) Fix make_top_consumption_snapshots heap-use-after-free #27434 (#27465)

* [fix](function) make TIMESTAMP function DEPEND_ON_ARGUMENT (#27343) (#27458)

* [fix](test) order by clause in test_map(#27390) (#27391)

pick #27390

* [performance](Planner): optimize getStringValue() in DateLiteral (#27363) (#27470)

- reduce cost of `getStringValue()`
- original code don't consider `microsecond` part in `getStringValue()`

(cherry picked from commit 044a295)

* [Chore](pick) do not push down agg on aggregate column (#27356) (#27498)

* [fix](stats) table not exists error msg not print objects name #27074 (#27463)

* [improve](nereids) support agg function of count(const value) pushdown #26677 (#27499)

support sql: select count(1)-count(not null) from table, the agg of count could push down.

* [test](fe-ut) fix unstable MysqlServerTest (#27459)

Need to find a unbind port for MysqlServerTest

* [opt](MergedIO) no need to merge large columns (#27315) (#27497)

1. Fix a profile bug of `MergeRangeFileReader`, and add a profile `ApplyBytes` to show the total bytes  of ranges.
2. There's no need to merge large columns, because `MergeRangeFileReader` will increase the copy time.

* [improvement](drop tablet)  impr gc shutdown tablet lock (#26151) (#27478)

* [doc](stats) SQL manual for stats (#27461)

* [chore](merge-on-write) disable rowid conversion check for mow table by default (#27482) (#27508)

* [fix](regression)Fix hive p2 case (#27466) (#27511)

* [fix](statistics)Fix auto analyze remove finished job bug #27486 (#27510)

* [Bug](bitmap) Fix heap-use-after-free in the bitmap functions #27411 (#27521)

* [Pick](nereids) Pick: partition prune fails in case of NOT expression (#27047) (#27507)

* [fix](clone) Fix engine_clone file exist (#27361) (#27536)

* [chore](case) adjust timeout of broker load case #27540

* Fix auto analyze doesn't filter unsupported type bug. (#27547)

Fix auto analyze doesn't filter unsupported type bug.
Catch throwable in auto analyze thread for each database, otherwise the thread will quit when one database failed to create jobs and all other databases will not get analyzed.
change FE config item full_auto_analyze_simultaneously_running_task_num to auto_analyze_simultaneously_running_task_num
backport #27559

* [chore](fe plugin) Upgrade dependency to doris 2.0-SNAPSHOT #27522 (#27558)

* [Bug](materialized-view) add limitation for duplicate expr on materialized view (#27523) (#27562)

* [fix](planner)join node should output required slot from parent node #27526 (#27551)

* [branch-2.0](hive) enable hive view by default (#27550)

* [pick](nereids) adjust bc join and shuffle join #27113 (#27566)

* [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. (#27567)

---------

Co-authored-by: AKIRA <33112463+Kikyou1997@users.noreply.github.com>
Co-authored-by: amory <wangqiannan@selectdb.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: Pxl <pxl290@qq.com>
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
Co-authored-by: 谢健 <jianxie0@gmail.com>
Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com>
Co-authored-by: jakevin <jakevingoo@gmail.com>
Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com>
Co-authored-by: yujun <yu.jun.reach@gmail.com>
Co-authored-by: Xin Liao <liaoxinbit@126.com>
Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>
Co-authored-by: minghong <englefly@gmail.com>
Co-authored-by: Jack Drogon <jack.xsuperman@gmail.com>
Co-authored-by: Dongyang Li <hello_stephen@qq.com>
Co-authored-by: zhiqiang <seuhezhiqiang@163.com>
Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com>
Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.3-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants