Skip to content

branch-3.1: [fix](union) Local shuffle for union operator #56048 #56449 #56556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Sep 29, 2025
Merged

branch-3.1: [fix](union) Local shuffle for union operator #56048 #56449 #56556

morrySnow merged 2 commits into from
Sep 29, 2025

Conversation

@Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Sep 28, 2025 •
edited by morrySnow
Loading

picked from #56048 #56449

@Thearas
Copy link
Contributor

Thearas commented Sep 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow morrySnow changed the title Pick 0928 branch-3.1: [fix](union) Local shuffle for union operator #56048 #56449 Sep 28, 2025
@morrySnow
Copy link
Contributor

morrySnow commented Sep 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 28, 2025

TPC-H: Total hot run time: 32617 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3a9124e849c10351a624145e09ed95729a5ca47e, data reload: false

------ Round 1 ----------------------------------
q1	17588	5566	5435	5435
q2	2057	281	171	171
q3	10424	1312	727	727
q4	10233	876	458	458
q5	8254	2376	2148	2148
q6	197	166	142	142
q7	913	738	625	625
q8	9363	1459	1172	1172
q9	5365	4964	4917	4917
q10	6847	2307	1815	1815
q11	491	275	261	261
q12	342	368	210	210
q13	17776	3632	3043	3043
q14	221	225	213	213
q15	531	454	460	454
q16	414	422	377	377
q17	612	871	380	380
q18	6796	6489	6294	6294
q19	1086	975	528	528
q20	328	342	213	213
q21	3031	2202	2069	2069
q22	1052	1038	965	965
Total cold run time: 103921 ms
Total hot run time: 32617 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5600	5534	5551	5534
q2	238	330	229	229
q3	2304	2632	2362	2362
q4	1358	1784	1383	1383
q5	4444	4858	5210	4858
q6	177	166	128	128
q7	2098	1973	1826	1826
q8	2655	2844	2707	2707
q9	7238	7171	7185	7171
q10	3027	3266	2788	2788
q11	582	519	505	505
q12	662	794	567	567
q13	3366	3810	3167	3167
q14	296	308	283	283
q15	538	480	452	452
q16	458	497	446	446
q17	1252	1741	1267	1267
q18	7640	7401	7303	7303
q19	829	1057	1125	1057
q20	2012	2019	1932	1932
q21	5510	4954	4696	4696
q22	1125	1082	1025	1025
Total cold run time: 53409 ms
Total hot run time: 51686 ms

@doris-robot
Copy link

doris-robot commented Sep 28, 2025

TPC-DS: Total hot run time: 190747 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3a9124e849c10351a624145e09ed95729a5ca47e, data reload: false

query1	969	392	392	392
query2	6540	1992	1919	1919
query3	6712	215	220	215
query4	34191	23933	23563	23563
query5	4331	614	486	486
query6	287	192	179	179
query7	4625	509	321	321
query8	302	256	253	253
query9	9371	2612	2609	2609
query10	462	326	280	280
query11	18258	15699	15301	15301
query12	153	107	107	107
query13	1654	548	423	423
query14	9443	6955	7322	6955
query15	222	196	183	183
query16	7936	665	459	459
query17	1940	800	599	599
query18	2123	427	330	330
query19	252	202	186	186
query20	123	121	118	118
query21	226	130	107	107
query22	4380	4809	4521	4521
query23	35311	34133	33286	33286
query24	7611	2661	2646	2646
query25	489	476	448	448
query26	1196	285	173	173
query27	2125	471	347	347
query28	4960	2198	2148	2148
query29	630	605	455	455
query30	251	194	157	157
query31	960	911	796	796
query32	71	65	60	60
query33	517	359	308	308
query34	740	827	516	516
query35	790	800	735	735
query36	981	1047	972	972
query37	108	96	75	75
query38	3983	3867	3880	3867
query39	1492	1453	1432	1432
query40	212	120	109	109
query41	56	54	55	54
query42	120	107	110	107
query43	533	531	489	489
query44	1338	812	804	804
query45	184	180	174	174
query46	878	1047	674	674
query47	1873	1901	1827	1827
query48	428	427	358	358
query49	816	534	411	411
query50	682	688	420	420
query51	7240	7142	7037	7037
query52	106	103	92	92
query53	226	255	184	184
query54	548	546	473	473
query55	84	78	84	78
query56	255	268	249	249
query57	1218	1223	1171	1171
query58	240	221	213	213
query59	3071	3125	2861	2861
query60	299	294	279	279
query61	119	112	119	112
query62	808	760	671	671
query63	231	210	195	195
query64	4130	976	647	647
query65	3305	3208	3248	3208
query66	1067	416	311	311
query67	15957	15697	15499	15499
query68	7132	856	540	540
query69	473	309	265	265
query70	1155	1164	1130	1130
query71	385	297	269	269
query72	5639	3817	3913	3817
query73	648	763	364	364
query74	10658	9408	8963	8963
query75	3301	3136	2671	2671
query76	3310	1174	777	777
query77	743	382	291	291
query78	10360	10482	9553	9553
query79	3193	876	593	593
query80	705	536	438	438
query81	501	257	225	225
query82	611	121	93	93
query83	179	170	152	152
query84	280	115	92	92
query85	793	362	293	293
query86	369	326	278	278
query87	4335	4288	4196	4196
query88	5006	2414	2406	2406
query89	398	331	294	294
query90	1971	184	184	184
query91	139	147	111	111
query92	66	55	52	52
query93	1898	875	542	542
query94	692	409	299	299
query95	341	265	260	260
query96	504	608	290	290
query97	3227	3318	3182	3182
query98	227	203	201	201
query99	1510	1420	1292	1292
Total cold run time: 293488 ms
Total hot run time: 190747 ms

@doris-robot
Copy link

doris-robot commented Sep 28, 2025

ClickBench: Total hot run time: 28.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3a9124e849c10351a624145e09ed95729a5ca47e, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.11	0.12
query5	0.51	0.50	0.51
query6	1.12	0.74	0.74
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.52	0.52
query10	0.56	0.55	0.55
query11	0.15	0.10	0.10
query12	0.14	0.10	0.10
query13	0.61	0.59	0.60
query14	0.78	0.82	0.80
query15	0.85	0.82	0.82
query16	0.40	0.39	0.38
query17	1.02	1.05	1.03
query18	0.25	0.23	0.23
query19	1.99	1.90	1.87
query20	0.02	0.01	0.02
query21	15.38	0.90	0.56
query22	0.74	0.85	0.58
query23	15.17	1.46	0.56
query24	3.40	1.07	0.74
query25	0.19	0.07	0.06
query26	0.33	0.15	0.14
query27	0.06	0.04	0.05
query28	13.13	1.05	0.44
query29	12.59	3.99	3.26
query30	0.24	0.08	0.06
query31	2.83	0.59	0.38
query32	3.22	0.53	0.46
query33	3.00	2.99	3.06
query34	16.44	5.19	4.51
query35	4.62	4.56	4.63
query36	0.64	0.52	0.48
query37	0.09	0.06	0.05
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.06	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.43 s
Total hot run time: 28.65 s

@morrySnow morrySnow merged commit be164cf into apache:branch-3.1 Sep 29, 2025
22 of 24 checks passed
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Gabriel39 @Thearas @morrySnow @doris-robot