Skip to content

Revert "[feature](pipelineX) control exchange sink by memory usage (#… #29652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Gabriel39 merged 1 commit into from
Jan 8, 2024
Merged

Revert "[feature](pipelineX) control exchange sink by memory usage (#… #29652

Gabriel39 merged 1 commit into from
Jan 8, 2024

Conversation

@Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Jan 8, 2024

…28814)"

This reverts commit e326ebb.

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions
Copy link
Contributor

github-actions bot commented Jan 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

zclllyybb
zclllyybb approved these changes Jan 8, 2024
View reviewed changes
Copy link
Contributor

@zclllyybb zclllyybb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zclllyybb
Copy link
Contributor

zclllyybb commented Jan 8, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jan 8, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Jan 8, 2024

TeamCity be ut coverage result:
Function Coverage: 36.67% (8638/23559)
Line Coverage: 28.67% (70191/244809)
Region Coverage: 27.62% (36298/131439)
Branch Coverage: 24.30% (18548/76332)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9b2d33c210420782be6a71ed0af7728e184e80ab_9b2d33c210420782be6a71ed0af7728e184e80ab/report/index.html

@doris-robot
Copy link

doris-robot commented Jan 8, 2024

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.55 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.1 seconds inserted 10000000 Rows, about 369K ops/s
storage size: 17183498642 Bytes

@github-actions
Copy link
Contributor

github-actions bot commented Jan 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 8, 2024
@Gabriel39 Gabriel39 merged commit 8fd69ac into apache:master Jan 8, 2024
@doris-robot
Copy link

doris-robot commented Jan 8, 2024

TPC-H: Total hot run time: 38542 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9b2d33c210420782be6a71ed0af7728e184e80ab, data reload: false

------ Round 1 ----------------------------------
q1	17682	4975	4921	4921
q2	2014	155	144	144
q3	10639	1143	1154	1143
q4	10468	780	756	756
q5	7815	2962	2931	2931
q6	205	124	127	124
q7	926	525	494	494
q8	9274	1984	1984	1984
q9	6462	6420	6320	6320
q10	8278	3055	3042	3042
q11	441	209	211	209
q12	393	229	225	225
q13	17996	3396	3404	3396
q14	241	209	206	206
q15	548	513	495	495
q16	462	408	396	396
q17	975	691	543	543
q18	7360	6703	6682	6682
q19	1580	1520	1514	1514
q20	714	310	298	298
q21	2777	2425	2409	2409
q22	347	310	338	310
Total cold run time: 107597 ms
Total hot run time: 38542 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4967	4930	4857	4857
q2	336	230	243	230
q3	3297	3311	3288	3288
q4	2112	2015	2030	2015
q5	5817	5803	5781	5781
q6	205	116	120	116
q7	2323	1913	1874	1874
q8	3369	3503	3466	3466
q9	8761	8722	8672	8672
q10	3786	3840	3865	3840
q11	542	453	447	447
q12	799	628	656	628
q13	5597	3208	3174	3174
q14	290	254	254	254
q15	543	508	501	501
q16	559	484	487	484
q17	1869	1850	1861	1850
q18	8640	8336	8180	8180
q19	1608	1641	1655	1641
q20	2185	1932	1944	1932
q21	5625	5328	5225	5225
q22	502	468	429	429
Total cold run time: 63732 ms
Total hot run time: 58884 ms

@doris-robot
Copy link

doris-robot commented Jan 8, 2024

TPC-DS: Total hot run time: 184079 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9b2d33c210420782be6a71ed0af7728e184e80ab, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	940	344	331	331
query2	6562	1917	2007	1917
query3	6719	231	230	230
query4	26836	22472	22442	22442
query5	5312	637	621	621
query6	363	210	207	207
query7	4641	302	296	296
query8	269	226	207	207
query9	8785	3003	3000	3000
query10	582	294	313	294
query11	16166	15624	15978	15624
query12	168	104	94	94
query13	1701	364	339	339
query14	13226	7692	7709	7692
query15	251	221	221	221
query16	6495	331	337	331
query17	1698	523	508	508
query18	1974	302	314	302
query19	242	166	168	166
query20	112	108	108	108
query21	193	98	90	90
query22	4905	4698	4842	4698
query23	32496	30928	31708	30928
query24	10849	2942	2935	2935
query25	609	357	361	357
query26	1603	169	176	169
query27	2752	322	324	322
query28	7030	2168	2163	2163
query29	1554	429	413	413
query30	283	145	149	145
query31	1008	826	811	811
query32	135	87	85	85
query33	840	364	310	310
query34	855	476	500	476
query35	988	853	856	853
query36	1379	1278	1312	1278
query37	119	85	80	80
query38	3489	3368	3350	3350
query39	1361	1290	1464	1290
query40	186	87	87	87
query41	42	36	37	36
query42	104	104	116	104
query43	563	498	507	498
query44	1169	811	812	811
query45	224	215	221	215
query46	1052	693	732	693
query47	1768	1722	1656	1656
query48	364	281	286	281
query49	1073	323	318	318
query50	758	359	368	359
query51	5418	5263	5341	5263
query52	121	91	96	91
query53	240	164	173	164
query54	1425	663	654	654
query55	105	94	99	94
query56	256	244	260	244
query57	1037	1030	1030	1030
query58	310	294	294	294
query59	3060	2914	2744	2744
query60	336	303	285	285
query61	151	155	142	142
query62	561	516	496	496
query63	184	171	177	171
query64	4983	1686	1727	1686
query65	3429	3337	3355	3337
query66	1225	388	381	381
query67	15458	15515	15495	15495
query68	12924	576	552	552
query69	594	297	309	297
query70	1776	1630	1562	1562
query71	619	263	259	259
query72	5333	3452	3422	3422
query73	2547	352	335	335
query74	7078	6509	6507	6507
query75	4955	2317	2246	2246
query76	6398	1130	1285	1130
query77	764	332	334	332
query78	9085	8729	8614	8614
query79	2125	530	522	522
query80	561	344	344	344
query81	484	211	218	211
query82	227	113	100	100
query83	187	141	140	140
query84	256	62	59	59
query85	932	281	268	268
query86	429	442	452	442
query87	3638	3405	3463	3405
query88	3510	2537	2511	2511
query89	374	273	297	273
query90	1964	258	254	254
query91	122	95	98	95
query92	91	81	78	78
query93	1530	497	495	495
query94	856	245	221	221
query95	597	518	508	508
query96	631	345	345	345
query97	4296	4208	4247	4208
query98	234	215	231	215
query99	1146	836	817	817
Total cold run time: 297355 ms
Total hot run time: 184079 ms

yiguolei pushed a commit that referenced this pull request Jan 8, 2024
@Gabriel39 @Doris-Extras
Revert "[feature](pipelineX) control exchange sink by memory usage (#…
767de7a 
…28814)" (#29652)

This reverts commit e326ebb.
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@Gabriel39 @HappenLee
Revert "[feature](pipelineX) control exchange sink by memory usage (a…
55056b2 
…pache#28814)" (apache#29652)

This reverts commit e326ebb.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@zclllyybb zclllyybb zclllyybb approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.0-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Gabriel39 @zclllyybb @doris-robot @yiguolei @dataroaring