Skip to content

Conversation

@BiteTheDDDDt
Copy link
Contributor

Proposed changes

select min(lo_extendedprice),max(lo_extendedprice) from lineorder_flat;
*** Query id: 8369862682bc40c4-881e7c3e1daf6d58 ***
*** tablet id: 0 ***
*** Aborted at 1700547343 (unix time) try "date -d @1700547343" if you are using GNU date ***
*** Current BE git commitID: 7fe4b58fc3 ***
*** SIGSEGV address not mapped to object (@0x29) received by PID 2022939 (TID 2023553 OR 0x7f342920c700) from PID 41; stack trace: ***
 0# 0x000055D6B92CEB6D in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
 1# 0x00007F3570FF5400 in /lib64/libc.so.6
 2# doris::segment_v2::ColumnReader::_parse_zone_map_skip_null(doris::segment_v2::ZoneMapPB const&, doris::WrapperField*, doris::WrapperField*) const in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
 3# doris::segment_v2::ColumnReader::next_batch_of_zone_map(unsigned long*, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&) const in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
 4# doris::segment_v2::FileColumnIterator::next_batch_of_zone_map(unsigned long*, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&) in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
 5# doris::vectorized::VStatisticsIterator::next_batch(doris::vectorized::Block*) in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
 6# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
 7# doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
 8# doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
 9# doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
10# doris::vectorized::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > >&) in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
11# doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
12# doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
13# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
14# doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::VScanner>) in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
15# 0x000055D6BEDF8C91 in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
16# doris::WorkThreadPool<true>::work_thread(int) in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
17# 0x000055D6C651E360 in /mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be
18# start_thread in /lib64/libpthread.so.0
19# __GI___clone in /lib64/libc.so.6

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 99212a385588030f9cdd4f08c16f24c75e2d4cc2, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4908	4669	4682	4669
q2	370	181	197	181
q3	2024	1932	1907	1907
q4	1367	1260	1237	1237
q5	3976	3912	3992	3912
q6	246	131	134	131
q7	1390	883	911	883
q8	2726	2784	2752	2752
q9	45497	13131	9556	9556
q10	10275	3536	3518	3518
q11	398	252	239	239
q12	731	288	295	288
q13	4564	3777	3807	3777
q14	326	288	300	288
q15	586	534	536	534
q16	684	603	583	583
q17	1135	984	909	909
q18	7787	7315	7380	7315
q19	1685	1663	1658	1658
q20	535	290	306	290
q21	6308	3971	3961	3961
q22	486	373	371	371
Total cold run time: 98004 ms
Total hot run time: 48959 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4646	4554	4543	4543
q2	346	229	270	229
q3	4003	3983	3969	3969
q4	2694	2684	2691	2684
q5	9692	9585	9584	9584
q6	242	125	125	125
q7	3022	2462	2520	2462
q8	4452	4448	4461	4448
q9	13231	13146	13170	13146
q10	4098	4180	4209	4180
q11	776	699	657	657
q12	979	816	832	816
q13	4281	3570	3558	3558
q14	391	342	350	342
q15	570	523	518	518
q16	729	669	678	669
q17	3870	3895	3848	3848
q18	9478	8913	8923	8913
q19	1811	1747	1780	1747
q20	2390	2095	2040	2040
q21	8761	8805	8801	8801
q22	921	811	804	804
Total cold run time: 81383 ms
Total hot run time: 78083 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.3 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17098781312 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.54% (8449/23124)
Line Coverage: 28.88% (68712/237939)
Region Coverage: 27.84% (35532/127641)
Branch Coverage: 24.59% (18120/73684)
Coverage Report: http://coverage.selectdb-in.cc/coverage/99212a385588030f9cdd4f08c16f24c75e2d4cc2_99212a385588030f9cdd4f08c16f24c75e2d4cc2/report/index.html

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 21, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@BiteTheDDDDt BiteTheDDDDt changed the title [Bug](scan) do not ppush down agg on aggregate column when query match mv [Bug](scan) do not ppush down agg on aggregate column Nov 21, 2023
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BiteTheDDDDt BiteTheDDDDt changed the title [Bug](scan) do not ppush down agg on aggregate column [Bug](scan) do not push down agg on aggregate column Nov 22, 2023
@BiteTheDDDDt BiteTheDDDDt merged commit b541de7 into apache:master Nov 22, 2023
@wm1581066 wm1581066 added the p0_c label Nov 22, 2023
BiteTheDDDDt added a commit that referenced this pull request Nov 23, 2023
do not push down agg on aggregate column
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Nov 27, 2023
eldenmoon added a commit that referenced this pull request Nov 27, 2023
* [fix](stats) Fix update rows for unique table didn't get updated properly #26968 (#27337)

* [FIX](jsonb) fix jsonb in predict column #27325 (#27424)

* [fix](fe) slots in having clause should be set to need materialized(#27412) (#27429)

* [Bug](insert)fix insert wrong data on mv when stmt have multiple values (#27297) (#27382)

fix insert wrong data on mv when stmt have multiple values

* [fix](fe ut) Fix OlapQueryCacheTest failed (#27305) (#27406)

1.
```
java.lang.NullPointerException: null
        at org.apache.doris.catalog.Env.getCurrentSystemInfo(Env.java:793) ~[classes/:?]
        at org.apache.doris.qe.SimpleScheduler$UpdateBlacklistThread.run(SimpleScheduler.java:206) ~[classes/:?]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]

java.lang.NullPointerException
        at org.apache.doris.qe.OlapQueryCacheTest.setUp(OlapQueryCacheTest.java:226)
```

2.
```
[ERROR] testSqlCacheKeyWithNestedViewForNereids  Time elapsed: 1.962 s  <<< FAILURE!
java.lang.AssertionError: SELECT command denied to user 'testCluster:testUser'@'192.168.1.1' for table 'internal: testCluster:testDb: appevent'
	at org.apache.doris.qe.OlapQueryCacheTest.parseSqlByNereids(OlapQueryCacheTest.java:579)
	at org.apache.doris.qe.OlapQueryCacheTest.testSqlCacheKeyWithNestedViewForNereids(OlapQueryCacheTest.java:1338)
```

3.
```
[ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 113.63 s <<< FAILURE! - in org.apache.doris.qe.OlapQueryCacheTest
[ERROR] testCacheModeTable  Time elapsed: 1.657 s  <<< ERROR!
java.lang.IllegalArgumentException: Value of type org.apache.doris.qe.QueryState incompatible with return type org.apache.doris.system.SystemInfoService of org.apache.doris.catalog.Env#getCurrentSystemInfo()
        at org.apache.doris.qe.OlapQueryCacheTest.setUp(OlapQueryCacheTest.java:156)
```

* [regression test](schema change) add some schema change regression cases (#27112) (#27418)

* [fix](Nereids) result type of add precision is 1 more than expected (#27136) (#27426)

* [fix](Nereids): fill miss slot in having subquery (#27177) (#27394)

* [fix](memory) Fix make_top_consumption_snapshots heap-use-after-free #27434 (#27465)

* [fix](function) make TIMESTAMP function DEPEND_ON_ARGUMENT (#27343) (#27458)

* [fix](test) order by clause in test_map(#27390) (#27391)

pick #27390

* [performance](Planner): optimize getStringValue() in DateLiteral (#27363) (#27470)

- reduce cost of `getStringValue()`
- original code don't consider `microsecond` part in `getStringValue()`

(cherry picked from commit 044a295)

* [Chore](pick) do not push down agg on aggregate column (#27356) (#27498)

* [fix](stats) table not exists error msg not print objects name #27074 (#27463)

* [improve](nereids) support agg function of count(const value) pushdown #26677 (#27499)

support sql: select count(1)-count(not null) from table, the agg of count could push down.

* [test](fe-ut) fix unstable MysqlServerTest (#27459)

Need to find a unbind port for MysqlServerTest

* [opt](MergedIO) no need to merge large columns (#27315) (#27497)

1. Fix a profile bug of `MergeRangeFileReader`, and add a profile `ApplyBytes` to show the total bytes  of ranges.
2. There's no need to merge large columns, because `MergeRangeFileReader` will increase the copy time.

* [improvement](drop tablet)  impr gc shutdown tablet lock (#26151) (#27478)

* [doc](stats) SQL manual for stats (#27461)

* [chore](merge-on-write) disable rowid conversion check for mow table by default (#27482) (#27508)

* [fix](regression)Fix hive p2 case (#27466) (#27511)

* [fix](statistics)Fix auto analyze remove finished job bug #27486 (#27510)

* [Bug](bitmap) Fix heap-use-after-free in the bitmap functions #27411 (#27521)

* [Pick](nereids) Pick: partition prune fails in case of NOT expression (#27047) (#27507)

* [fix](clone) Fix engine_clone file exist (#27361) (#27536)

* [chore](case) adjust timeout of broker load case #27540

* Fix auto analyze doesn't filter unsupported type bug. (#27547)

Fix auto analyze doesn't filter unsupported type bug.
Catch throwable in auto analyze thread for each database, otherwise the thread will quit when one database failed to create jobs and all other databases will not get analyzed.
change FE config item full_auto_analyze_simultaneously_running_task_num to auto_analyze_simultaneously_running_task_num
backport #27559

* [chore](fe plugin) Upgrade dependency to doris 2.0-SNAPSHOT #27522 (#27558)

* [Bug](materialized-view) add limitation for duplicate expr on materialized view (#27523) (#27562)

* [fix](planner)join node should output required slot from parent node #27526 (#27551)

* [branch-2.0](hive) enable hive view by default (#27550)

* [pick](nereids) adjust bc join and shuffle join #27113 (#27566)

* [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. (#27567)

---------

Co-authored-by: AKIRA <33112463+Kikyou1997@users.noreply.github.com>
Co-authored-by: amory <wangqiannan@selectdb.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: Pxl <pxl290@qq.com>
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
Co-authored-by: 谢健 <jianxie0@gmail.com>
Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com>
Co-authored-by: jakevin <jakevingoo@gmail.com>
Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com>
Co-authored-by: yujun <yu.jun.reach@gmail.com>
Co-authored-by: Xin Liao <liaoxinbit@126.com>
Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>
Co-authored-by: minghong <englefly@gmail.com>
Co-authored-by: Jack Drogon <jack.xsuperman@gmail.com>
Co-authored-by: Dongyang Li <hello_stephen@qq.com>
Co-authored-by: zhiqiang <seuhezhiqiang@163.com>
Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com>
Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
do not push down agg on aggregate column
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
do not push down agg on aggregate column
@BiteTheDDDDt BiteTheDDDDt deleted the fix_1121 branch January 20, 2025 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.3-merged p0_c reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants