Skip to content

[fix](local shuffle) add distribute exprs to set operation node #56157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
starocean999 merged 1 commit into from
Sep 18, 2025
Merged

[fix](local shuffle) add distribute exprs to set operation node #56157

starocean999 merged 1 commit into from
Sep 18, 2025

Conversation

@morrySnow
Copy link
Contributor

@morrySnow morrySnow commented Sep 17, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Sep 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow
Copy link
Contributor Author

morrySnow commented Sep 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 17, 2025

TPC-H: Total hot run time: 34725 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 83784bafc086a4d1c94771e6f990a0b9200338ff, data reload: false

------ Round 1 ----------------------------------
q1	17613	5454	5250	5250
q2	2000	321	213	213
q3	10284	1298	700	700
q4	10236	1014	529	529
q5	7527	2425	2311	2311
q6	182	166	138	138
q7	918	779	635	635
q8	9362	1360	1090	1090
q9	6883	5123	5102	5102
q10	6952	2361	1992	1992
q11	491	304	291	291
q12	348	363	227	227
q13	17771	3680	3055	3055
q14	240	240	218	218
q15	586	482	499	482
q16	995	1011	946	946
q17	589	853	352	352
q18	7544	7053	7082	7053
q19	1489	951	551	551
q20	334	337	227	227
q21	3701	3200	2372	2372
q22	1033	1028	991	991
Total cold run time: 107078 ms
Total hot run time: 34725 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5267	5108	5108	5108
q2	248	329	231	231
q3	2173	2652	2309	2309
q4	1361	1765	1325	1325
q5	4263	4440	4586	4440
q6	227	176	135	135
q7	2095	1937	1893	1893
q8	2743	2649	2519	2519
q9	7276	7326	7285	7285
q10	3161	3263	2890	2890
q11	602	568	511	511
q12	715	876	635	635
q13	3537	4052	3238	3238
q14	295	316	296	296
q15	538	477	481	477
q16	1098	1144	1069	1069
q17	1188	1564	1346	1346
q18	7907	7860	7595	7595
q19	777	785	856	785
q20	2001	2276	1945	1945
q21	4860	4387	4365	4365
q22	1076	1047	995	995
Total cold run time: 53408 ms
Total hot run time: 51392 ms

@doris-robot
Copy link

doris-robot commented Sep 17, 2025

TPC-DS: Total hot run time: 188204 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 83784bafc086a4d1c94771e6f990a0b9200338ff, data reload: false

query1	1039	457	411	411
query2	6545	1736	1684	1684
query3	6764	225	227	225
query4	26064	23330	23157	23157
query5	4410	632	455	455
query6	386	229	214	214
query7	4654	499	299	299
query8	296	286	239	239
query9	8666	2592	2632	2592
query10	469	342	283	283
query11	15514	15302	14980	14980
query12	166	117	136	117
query13	1673	553	434	434
query14	10211	9286	9212	9212
query15	206	195	173	173
query16	7334	693	548	548
query17	1224	742	619	619
query18	1991	416	330	330
query19	212	193	169	169
query20	135	127	122	122
query21	219	138	116	116
query22	4029	4180	4066	4066
query23	33781	32882	33034	32882
query24	8471	2440	2476	2440
query25	632	525	469	469
query26	1249	278	165	165
query27	2750	511	354	354
query28	4367	2231	2235	2231
query29	879	601	483	483
query30	296	214	190	190
query31	906	829	739	739
query32	80	75	73	73
query33	579	380	361	361
query34	773	859	516	516
query35	819	841	730	730
query36	973	1020	932	932
query37	128	103	83	83
query38	3508	3527	3417	3417
query39	1493	1442	1412	1412
query40	226	126	121	121
query41	124	61	59	59
query42	123	114	114	114
query43	516	507	480	480
query44	1326	870	845	845
query45	189	192	179	179
query46	862	1021	647	647
query47	1752	1818	1713	1713
query48	396	425	320	320
query49	764	505	406	406
query50	647	684	397	397
query51	4039	3945	3899	3899
query52	118	115	101	101
query53	233	269	196	196
query54	599	588	522	522
query55	82	82	91	82
query56	324	312	300	300
query57	1202	1201	1123	1123
query58	301	262	260	260
query59	2572	2652	2549	2549
query60	352	364	330	330
query61	167	161	159	159
query62	823	761	675	675
query63	225	193	189	189
query64	4500	1159	814	814
query65	4086	3972	3971	3971
query66	1179	495	335	335
query67	15204	15255	15113	15113
query68	8050	925	573	573
query69	491	318	278	278
query70	1396	1243	1325	1243
query71	572	342	307	307
query72	5945	4966	4907	4907
query73	670	578	357	357
query74	9081	9026	8906	8906
query75	3898	3292	2874	2874
query76	3635	1179	720	720
query77	816	403	310	310
query78	9604	9582	8812	8812
query79	2055	845	605	605
query80	625	596	535	535
query81	486	262	235	235
query82	471	167	147	147
query83	317	343	247	247
query84	303	116	99	99
query85	895	474	434	434
query86	348	290	286	286
query87	3718	3716	3622	3622
query88	3378	2242	2234	2234
query89	469	334	308	308
query90	1974	220	215	215
query91	170	172	131	131
query92	82	64	64	64
query93	1680	960	655	655
query94	700	433	308	308
query95	393	316	312	312
query96	493	561	278	278
query97	2928	2964	2845	2845
query98	235	215	215	215
query99	1441	1417	1321	1321
Total cold run time: 274141 ms
Total hot run time: 188204 ms

@doris-robot
Copy link

doris-robot commented Sep 17, 2025

ClickBench: Total hot run time: 29.94 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 83784bafc086a4d1c94771e6f990a0b9200338ff, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.05
query3	0.25	0.08	0.08
query4	1.61	0.11	0.11
query5	0.27	0.28	0.25
query6	1.18	0.65	0.64
query7	0.03	0.03	0.03
query8	0.06	0.05	0.05
query9	0.62	0.53	0.52
query10	0.57	0.57	0.57
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.63	0.64	0.61
query14	1.01	1.02	1.06
query15	0.86	0.84	0.87
query16	0.41	0.43	0.40
query17	1.02	1.05	1.06
query18	0.22	0.19	0.20
query19	1.91	1.78	1.81
query20	0.02	0.01	0.01
query21	15.44	0.93	0.59
query22	0.75	1.20	0.75
query23	14.88	1.37	0.66
query24	6.99	0.83	0.86
query25	0.51	0.08	0.08
query26	0.58	0.16	0.13
query27	0.07	0.05	0.06
query28	9.97	0.88	0.44
query29	12.55	3.89	3.29
query30	0.28	0.13	0.11
query31	2.84	0.61	0.39
query32	3.26	0.56	0.49
query33	3.03	3.02	3.09
query34	15.97	5.52	4.86
query35	4.87	4.88	4.94
query36	0.69	0.53	0.53
query37	0.12	0.08	0.07
query38	0.06	0.06	0.04
query39	0.03	0.03	0.03
query40	0.18	0.14	0.13
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.37 s
Total hot run time: 29.94 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 17, 2025

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 17, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 18, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 18, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 18, 2025

PR approved by anyone and no changes requested.

@starocean999 starocean999 merged commit 04cc05c into apache:master Sep 18, 2025
30 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Sep 18, 2025
@morrySnow
[fix](local shuffle) add distribute exprs to set operation node (#56157)
b18a568
@github-actions github-actions bot mentioned this pull request Sep 18, 2025
Merged
@morrySnow morrySnow mentioned this pull request Sep 25, 2025
Merged
16 tasks
morrySnow added a commit that referenced this pull request Sep 28, 2025
@github-actions @morrySnow
branch-3.1: [fix](local shuffle) add distribute exprs to set operatio…
5345313 
…n node #56157 (#56180)

Cherry-picked from #56157

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

@Jibing-Li Jibing-Li Jibing-Li approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@morrySnow @Thearas @doris-robot @hello-stephen @starocean999 @Jibing-Li