Skip to content

Conversation

@JackDrogon
Copy link
Contributor

@JackDrogon JackDrogon commented Nov 21, 2023

Proposed changes

Fix engine_clone file exist

  • not use RETURN_IF_ERROR directly, because defer(!status.ok()) remove linked files
  • if to binlog exist, compare md5sum

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@JackDrogon
Copy link
Contributor Author

run buildall

@JackDrogon JackDrogon force-pushed the hotfix/engine_clone_file_exist branch from d09cbc0 to 10b5844 Compare November 21, 2023 10:47
@JackDrogon

This comment was marked as duplicate.

@JackDrogon JackDrogon force-pushed the hotfix/engine_clone_file_exist branch from 10b5844 to 0024800 Compare November 21, 2023 10:48
@JackDrogon
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
@JackDrogon JackDrogon force-pushed the hotfix/engine_clone_file_exist branch from 0024800 to 957523e Compare November 21, 2023 10:59
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@JackDrogon
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 957523e282e7f2e6b2e1d04b11f575d31ca8ea40, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4940	4622	4644	4622
q2	355	170	156	156
q3	2030	1938	1895	1895
q4	1388	1289	1227	1227
q5	3946	3904	4009	3904
q6	253	128	130	128
q7	1428	875	892	875
q8	2725	2761	2756	2756
q9	10090	14704	9542	9542
q10	5053	3568	3534	3534
q11	374	236	251	236
q12	457	298	290	290
q13	20462	3783	3783	3783
q14	323	283	279	279
q15	587	526	522	522
q16	660	592	577	577
q17	1134	963	957	957
q18	7712	7351	7258	7258
q19	1662	1672	1665	1665
q20	518	307	313	307
q21	4377	3926	3963	3926
q22	478	374	379	374
Total cold run time: 70952 ms
Total hot run time: 48813 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4589	4585	4561	4561
q2	351	231	263	231
q3	3998	3979	3993	3979
q4	2692	2682	2673	2673
q5	9644	9627	9648	9627
q6	244	117	124	117
q7	3017	2489	2444	2444
q8	4501	4456	4453	4453
q9	13161	13107	13137	13107
q10	4092	4191	4217	4191
q11	756	649	727	649
q12	987	827	818	818
q13	4295	3582	3558	3558
q14	370	362	339	339
q15	577	522	524	522
q16	728	675	659	659
q17	3914	3837	3866	3837
q18	9627	9110	9016	9016
q19	1802	1780	1741	1741
q20	2388	2041	2032	2032
q21	8761	8818	8586	8586
q22	867	780	820	780
Total cold run time: 81361 ms
Total hot run time: 77920 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.58 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.9 seconds inserted 10000000 Rows, about 358K ops/s
storage size: 17099132600 Bytes

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 21, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.52% (8446/23126)
Line Coverage: 28.85% (68662/237990)
Region Coverage: 27.81% (35499/127670)
Branch Coverage: 24.56% (18102/73708)
Coverage Report: http://coverage.selectdb-in.cc/coverage/00248004dd45ca8ebccc80d5e090c215991608bc_00248004dd45ca8ebccc80d5e090c215991608bc/report/index.html

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.53% (8447/23126)
Line Coverage: 28.86% (68691/237991)
Region Coverage: 27.82% (35518/127670)
Branch Coverage: 24.57% (18113/73708)
Coverage Report: http://coverage.selectdb-in.cc/coverage/957523e282e7f2e6b2e1d04b11f575d31ca8ea40_957523e282e7f2e6b2e1d04b11f575d31ca8ea40/report/index.html

@dataroaring dataroaring merged commit 2f04873 into apache:master Nov 21, 2023
@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.79 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17099645781 Bytes

@xiaokang xiaokang added the p0_b label Nov 22, 2023
JackDrogon added a commit to JackDrogon/doris that referenced this pull request Nov 24, 2023
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
JackDrogon added a commit to JackDrogon/doris that referenced this pull request Nov 24, 2023
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
JackDrogon added a commit to JackDrogon/doris that referenced this pull request Nov 24, 2023
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Nov 27, 2023
eldenmoon added a commit that referenced this pull request Nov 27, 2023
* [fix](stats) Fix update rows for unique table didn't get updated properly #26968 (#27337)

* [FIX](jsonb) fix jsonb in predict column #27325 (#27424)

* [fix](fe) slots in having clause should be set to need materialized(#27412) (#27429)

* [Bug](insert)fix insert wrong data on mv when stmt have multiple values (#27297) (#27382)

fix insert wrong data on mv when stmt have multiple values

* [fix](fe ut) Fix OlapQueryCacheTest failed (#27305) (#27406)

1.
```
java.lang.NullPointerException: null
        at org.apache.doris.catalog.Env.getCurrentSystemInfo(Env.java:793) ~[classes/:?]
        at org.apache.doris.qe.SimpleScheduler$UpdateBlacklistThread.run(SimpleScheduler.java:206) ~[classes/:?]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]

java.lang.NullPointerException
        at org.apache.doris.qe.OlapQueryCacheTest.setUp(OlapQueryCacheTest.java:226)
```

2.
```
[ERROR] testSqlCacheKeyWithNestedViewForNereids  Time elapsed: 1.962 s  <<< FAILURE!
java.lang.AssertionError: SELECT command denied to user 'testCluster:testUser'@'192.168.1.1' for table 'internal: testCluster:testDb: appevent'
	at org.apache.doris.qe.OlapQueryCacheTest.parseSqlByNereids(OlapQueryCacheTest.java:579)
	at org.apache.doris.qe.OlapQueryCacheTest.testSqlCacheKeyWithNestedViewForNereids(OlapQueryCacheTest.java:1338)
```

3.
```
[ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 113.63 s <<< FAILURE! - in org.apache.doris.qe.OlapQueryCacheTest
[ERROR] testCacheModeTable  Time elapsed: 1.657 s  <<< ERROR!
java.lang.IllegalArgumentException: Value of type org.apache.doris.qe.QueryState incompatible with return type org.apache.doris.system.SystemInfoService of org.apache.doris.catalog.Env#getCurrentSystemInfo()
        at org.apache.doris.qe.OlapQueryCacheTest.setUp(OlapQueryCacheTest.java:156)
```

* [regression test](schema change) add some schema change regression cases (#27112) (#27418)

* [fix](Nereids) result type of add precision is 1 more than expected (#27136) (#27426)

* [fix](Nereids): fill miss slot in having subquery (#27177) (#27394)

* [fix](memory) Fix make_top_consumption_snapshots heap-use-after-free #27434 (#27465)

* [fix](function) make TIMESTAMP function DEPEND_ON_ARGUMENT (#27343) (#27458)

* [fix](test) order by clause in test_map(#27390) (#27391)

pick #27390

* [performance](Planner): optimize getStringValue() in DateLiteral (#27363) (#27470)

- reduce cost of `getStringValue()`
- original code don't consider `microsecond` part in `getStringValue()`

(cherry picked from commit 044a295)

* [Chore](pick) do not push down agg on aggregate column (#27356) (#27498)

* [fix](stats) table not exists error msg not print objects name #27074 (#27463)

* [improve](nereids) support agg function of count(const value) pushdown #26677 (#27499)

support sql: select count(1)-count(not null) from table, the agg of count could push down.

* [test](fe-ut) fix unstable MysqlServerTest (#27459)

Need to find a unbind port for MysqlServerTest

* [opt](MergedIO) no need to merge large columns (#27315) (#27497)

1. Fix a profile bug of `MergeRangeFileReader`, and add a profile `ApplyBytes` to show the total bytes  of ranges.
2. There's no need to merge large columns, because `MergeRangeFileReader` will increase the copy time.

* [improvement](drop tablet)  impr gc shutdown tablet lock (#26151) (#27478)

* [doc](stats) SQL manual for stats (#27461)

* [chore](merge-on-write) disable rowid conversion check for mow table by default (#27482) (#27508)

* [fix](regression)Fix hive p2 case (#27466) (#27511)

* [fix](statistics)Fix auto analyze remove finished job bug #27486 (#27510)

* [Bug](bitmap) Fix heap-use-after-free in the bitmap functions #27411 (#27521)

* [Pick](nereids) Pick: partition prune fails in case of NOT expression (#27047) (#27507)

* [fix](clone) Fix engine_clone file exist (#27361) (#27536)

* [chore](case) adjust timeout of broker load case #27540

* Fix auto analyze doesn't filter unsupported type bug. (#27547)

Fix auto analyze doesn't filter unsupported type bug.
Catch throwable in auto analyze thread for each database, otherwise the thread will quit when one database failed to create jobs and all other databases will not get analyzed.
change FE config item full_auto_analyze_simultaneously_running_task_num to auto_analyze_simultaneously_running_task_num
backport #27559

* [chore](fe plugin) Upgrade dependency to doris 2.0-SNAPSHOT #27522 (#27558)

* [Bug](materialized-view) add limitation for duplicate expr on materialized view (#27523) (#27562)

* [fix](planner)join node should output required slot from parent node #27526 (#27551)

* [branch-2.0](hive) enable hive view by default (#27550)

* [pick](nereids) adjust bc join and shuffle join #27113 (#27566)

* [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. (#27567)

---------

Co-authored-by: AKIRA <33112463+Kikyou1997@users.noreply.github.com>
Co-authored-by: amory <wangqiannan@selectdb.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: Pxl <pxl290@qq.com>
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
Co-authored-by: 谢健 <jianxie0@gmail.com>
Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com>
Co-authored-by: jakevin <jakevingoo@gmail.com>
Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com>
Co-authored-by: yujun <yu.jun.reach@gmail.com>
Co-authored-by: Xin Liao <liaoxinbit@126.com>
Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>
Co-authored-by: minghong <englefly@gmail.com>
Co-authored-by: Jack Drogon <jack.xsuperman@gmail.com>
Co-authored-by: Dongyang Li <hello_stephen@qq.com>
Co-authored-by: zhiqiang <seuhezhiqiang@163.com>
Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com>
Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
@xiaokang xiaokang mentioned this pull request Dec 4, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.3-merged p0_b reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants