Skip to content

[opt](Nereids) let inverted index work with top opt (#29000) #33200

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Apr 3, 2024
Merged

[opt](Nereids) let inverted index work with top opt (#29000) #33200

morrySnow merged 1 commit into from
Apr 3, 2024

Conversation

@morrySnow
Copy link
Contributor

@morrySnow morrySnow commented Apr 3, 2024

pick from master #29000
commit id 3ea9ff7

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Apr 3, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morrySnow
Copy link
Contributor Author

morrySnow commented Apr 3, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Apr 3, 2024

TPC-H: Total hot run time: 49697 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 21b7a252d418e39e7646beb82cd2c75d04768394, data reload: false

------ Round 1 ----------------------------------
q1	18192	4333	4307	4307
q2	2029	145	146	145
q3	10278	1884	1933	1884
q4	10357	1263	1313	1263
q5	8459	3884	3983	3884
q6	233	122	122	122
q7	2033	1600	1613	1600
q8	9513	2706	2710	2706
q9	11125	10510	10301	10301
q10	8644	3501	3505	3501
q11	416	245	237	237
q12	465	306	306	306
q13	18338	4099	4003	4003
q14	369	330	323	323
q15	509	448	467	448
q16	696	590	587	587
q17	1117	968	1023	968
q18	7318	6822	6803	6803
q19	1660	1586	1524	1524
q20	520	311	310	310
q21	4441	4099	4087	4087
q22	503	388	396	388
Total cold run time: 117215 ms
Total hot run time: 49697 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4327	4300	4239	4239
q2	315	213	223	213
q3	4189	4146	4143	4143
q4	2739	2747	2723	2723
q5	7264	7166	7187	7166
q6	229	116	119	116
q7	3257	2871	2838	2838
q8	4369	4426	4476	4426
q9	17155	16942	17061	16942
q10	4232	4233	4288	4233
q11	752	701	672	672
q12	1020	835	855	835
q13	7014	3750	3732	3732
q14	451	430	419	419
q15	491	463	451	451
q16	765	689	700	689
q17	3838	3860	3847	3847
q18	8724	8894	8776	8776
q19	1710	1685	1645	1645
q20	2387	2155	2114	2114
q21	8576	8558	8527	8527
q22	1020	956	937	937
Total cold run time: 84824 ms
Total hot run time: 79683 ms

@doris-robot
Copy link

doris-robot commented Apr 3, 2024

TPC-DS: Total hot run time: 202483 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 21b7a252d418e39e7646beb82cd2c75d04768394, data reload: false

query1	942	392	375	375
query2	6517	2177	2248	2177
query3	6913	206	211	206
query4	20047	17988	18108	17988
query5	19706	6511	6512	6511
query6	297	228	232	228
query7	4159	305	302	302
query8	258	246	252	246
query9	3159	2755	2656	2656
query10	414	302	316	302
query11	11360	10683	10725	10683
query12	123	82	70	70
query13	5581	685	632	632
query14	17454	13538	13676	13538
query15	364	230	233	230
query16	6453	269	253	253
query17	1734	1468	888	888
query18	2293	405	415	405
query19	207	155	145	145
query20	80	75	72	72
query21	185	90	96	90
query22	5250	5078	4918	4918
query23	32642	31859	31927	31859
query24	6991	6572	6597	6572
query25	527	429	414	414
query26	520	166	161	161
query27	1800	302	300	300
query28	6092	2268	2250	2250
query29	2956	2657	2640	2640
query30	238	164	163	163
query31	898	734	739	734
query32	66	63	57	57
query33	391	265	250	250
query34	842	481	490	481
query35	1128	896	894	894
query36	1257	1229	1223	1223
query37	89	63	57	57
query38	3080	2969	2936	2936
query39	1347	1308	1309	1308
query40	206	98	100	98
query41	37	33	34	33
query42	88	84	79	79
query43	581	566	601	566
query44	1137	734	754	734
query45	239	227	221	221
query46	1234	1014	992	992
query47	1725	1771	1844	1771
query48	998	693	690	690
query49	612	365	381	365
query50	857	624	605	605
query51	4797	4701	4743	4701
query52	94	87	88	87
query53	443	336	323	323
query54	2672	2496	2473	2473
query55	88	88	99	88
query56	223	219	210	210
query57	1318	1121	1064	1064
query58	207	205	192	192
query59	3703	3425	3294	3294
query60	222	194	185	185
query61	87	83	88	83
query62	826	495	485	485
query63	472	337	336	336
query64	2337	1397	1455	1397
query65	3648	3531	3568	3531
query66	757	382	367	367
query67	17641	17768	15569	15569
query68	6020	675	672	672
query69	554	329	353	329
query70	1563	1357	1345	1345
query71	400	296	328	296
query72	6432	3440	3465	3440
query73	720	326	328	326
query74	6273	5896	5791	5791
query75	4673	3767	3756	3756
query76	3760	1148	1226	1148
query77	546	257	254	254
query78	12547	11545	14188	11545
query79	8449	659	662	659
query80	2919	400	398	398
query81	530	238	237	237
query82	257	93	103	93
query83	237	139	132	132
query84	256	69	70	69
query85	1402	295	290	290
query86	358	311	338	311
query87	3282	3035	2966	2966
query88	4354	2396	2398	2396
query89	346	311	293	293
query90	1729	213	215	213
query91	149	116	117	116
query92	63	53	51	51
query93	1758	568	589	568
query94	889	204	198	198
query95	1118	1098	1066	1066
query96	653	340	334	334
query97	6422	6400	6385	6385
query98	183	163	166	163
query99	3059	932	890	890
Total cold run time: 303510 ms
Total hot run time: 202483 ms

@doris-robot
Copy link

doris-robot commented Apr 3, 2024

ClickBench: Total hot run time: 30.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 21b7a252d418e39e7646beb82cd2c75d04768394, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.25	0.05	0.05
query4	1.79	0.07	0.07
query5	0.54	0.52	0.52
query6	1.28	0.62	0.62
query7	0.01	0.01	0.01
query8	0.04	0.03	0.02
query9	0.53	0.46	0.48
query10	0.55	0.53	0.53
query11	0.12	0.09	0.09
query12	0.13	0.10	0.09
query13	0.62	0.61	0.62
query14	0.78	0.80	0.80
query15	0.78	0.76	0.75
query16	0.38	0.39	0.36
query17	0.99	0.98	1.04
query18	0.23	0.24	0.25
query19	1.86	1.85	1.88
query20	0.01	0.01	0.01
query21	15.46	0.57	0.56
query22	2.06	1.88	1.49
query23	17.17	1.06	0.98
query24	7.93	0.81	0.69
query25	0.37	0.09	0.06
query26	0.82	0.14	0.16
query27	0.05	0.04	0.04
query28	5.76	0.77	0.75
query29	12.68	2.24	2.15
query30	0.60	0.52	0.52
query31	2.80	0.39	0.38
query32	3.40	0.50	0.49
query33	3.09	3.05	3.10
query34	15.28	4.79	4.80
query35	4.87	4.83	4.87
query36	1.05	1.01	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 104.75 s
Total hot run time: 30.27 s

@doris-robot
Copy link

doris-robot commented Apr 3, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 21b7a252d418e39e7646beb82cd2c75d04768394 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.1 seconds inserted 10000000 Rows, about 452K ops/s

@morrySnow
Copy link
Contributor Author

morrySnow commented Apr 3, 2024

run p0

@morrySnow morrySnow merged commit d138108 into apache:branch-2.0 Apr 3, 2024
@morrySnow morrySnow deleted the 2.0_29000 branch April 30, 2024 06:22
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morrySnow
[opt](Nereids) let inverted index work with top opt (apache#29000) (a…
f0090a8 
…pache#33200)

pick from master apache#29000
commit id 3ea9ff7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/nereids

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@morrySnow @doris-robot