Skip to content

[fix](union) Local shuffle for union operator #56048

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Gabriel39 merged 1 commit into from
Sep 15, 2025
Merged

[fix](union) Local shuffle for union operator #56048

Gabriel39 merged 1 commit into from
Sep 15, 2025

Conversation

@Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Sep 15, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Sep 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Sep 15, 2025

run buildall

yiguolei
yiguolei reviewed Sep 15, 2025
View reviewed changes
be/src/pipeline/exec/union_sink_operator.cpp
_row_descriptor(descs, tnode.row_tuples, tnode.nullable_tuples),
_cur_child_id(child_id),
_child_size(tnode.num_children) {}
_child_size(tnode.num_children),
Copy link
Contributor

@yiguolei yiguolei Sep 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add testcases

Copy link
Contributor Author

@Gabriel39 Gabriel39 Sep 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need another fix on FE side. Maybe test cases should be added later.

@doris-robot
Copy link

doris-robot commented Sep 15, 2025

TPC-H: Total hot run time: 34931 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35de9299c43f529831d2b9c85ede47b7032d1e97, data reload: false

------ Round 1 ----------------------------------
q1	17591	5229	5094	5094
q2	2029	332	211	211
q3	10110	1355	760	760
q4	10230	1007	529	529
q5	7527	2377	2417	2377
q6	190	176	137	137
q7	917	768	638	638
q8	9342	1365	1118	1118
q9	7035	5169	5076	5076
q10	6931	2417	1962	1962
q11	494	309	301	301
q12	371	358	241	241
q13	17802	3661	3110	3110
q14	260	244	217	217
q15	560	490	508	490
q16	1011	1010	947	947
q17	615	867	369	369
q18	7799	7158	7234	7158
q19	1091	946	570	570
q20	359	358	244	244
q21	3809	3219	2418	2418
q22	1080	1036	964	964
Total cold run time: 107153 ms
Total hot run time: 34931 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5185	5101	5159	5101
q2	258	342	230	230
q3	2182	2681	2351	2351
q4	1366	1809	1350	1350
q5	4254	4554	4698	4554
q6	234	174	132	132
q7	2064	1988	1845	1845
q8	2619	2557	2592	2557
q9	7304	7385	7266	7266
q10	3186	3366	2920	2920
q11	595	560	531	531
q12	722	798	731	731
q13	3795	3958	3355	3355
q14	302	320	287	287
q15	535	491	514	491
q16	1074	1120	1121	1120
q17	1190	1660	1403	1403
q18	7850	7762	7600	7600
q19	830	770	829	770
q20	1919	1954	1812	1812
q21	4811	4337	4317	4317
q22	1092	1069	1018	1018
Total cold run time: 53367 ms
Total hot run time: 51741 ms

@doris-robot
Copy link

doris-robot commented Sep 15, 2025

TPC-DS: Total hot run time: 189398 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35de9299c43f529831d2b9c85ede47b7032d1e97, data reload: false

query1	1064	438	419	419
query2	6559	1731	1751	1731
query3	6751	227	216	216
query4	26121	23763	23653	23653
query5	4962	656	454	454
query6	328	225	230	225
query7	4678	536	295	295
query8	303	261	250	250
query9	8655	2604	2615	2604
query10	512	349	283	283
query11	15623	15150	14754	14754
query12	178	118	113	113
query13	1684	592	462	462
query14	10588	9486	9469	9469
query15	223	203	181	181
query16	7224	677	513	513
query17	1192	753	639	639
query18	2020	481	326	326
query19	211	204	174	174
query20	127	125	124	124
query21	216	133	116	116
query22	4032	4139	4008	4008
query23	33710	32946	32916	32916
query24	8453	2451	2488	2451
query25	578	503	462	462
query26	1235	273	161	161
query27	2739	527	352	352
query28	4375	2228	2205	2205
query29	818	609	526	526
query30	297	235	205	205
query31	895	822	748	748
query32	83	74	82	74
query33	587	388	351	351
query34	804	873	539	539
query35	827	836	753	753
query36	998	1011	936	936
query37	117	118	85	85
query38	3554	3574	3578	3574
query39	1486	1490	1411	1411
query40	224	137	121	121
query41	66	62	61	61
query42	129	115	130	115
query43	552	523	495	495
query44	1400	852	839	839
query45	189	188	172	172
query46	911	1050	681	681
query47	1772	1772	1719	1719
query48	418	438	317	317
query49	781	513	403	403
query50	705	710	405	405
query51	3974	3934	3849	3849
query52	119	113	106	106
query53	244	283	196	196
query54	596	614	520	520
query55	85	85	88	85
query56	327	334	313	313
query57	1187	1206	1119	1119
query58	289	301	289	289
query59	2631	2714	2599	2599
query60	378	350	356	350
query61	196	188	194	188
query62	819	749	658	658
query63	240	204	206	204
query64	4552	1159	818	818
query65	4059	4006	3996	3996
query66	1179	447	345	345
query67	15506	15109	15187	15109
query68	7859	958	581	581
query69	505	326	286	286
query70	1341	1285	1237	1237
query71	572	344	317	317
query72	6088	5037	5228	5037
query73	731	677	360	360
query74	9148	8862	8681	8681
query75	3897	3344	2847	2847
query76	3619	1164	764	764
query77	831	417	328	328
query78	9518	9754	8911	8911
query79	1612	882	598	598
query80	654	650	514	514
query81	518	277	235	235
query82	227	168	148	148
query83	278	275	269	269
query84	261	120	101	101
query85	869	474	436	436
query86	381	320	304	304
query87	3805	3760	3721	3721
query88	3126	2235	2203	2203
query89	405	349	307	307
query90	1995	225	220	220
query91	163	174	137	137
query92	82	68	71	68
query93	1842	1009	632	632
query94	642	452	328	328
query95	401	322	316	316
query96	502	579	282	282
query97	2907	3028	2856	2856
query98	246	217	212	212
query99	1324	1393	1332	1332
Total cold run time: 274684 ms
Total hot run time: 189398 ms

@doris-robot
Copy link

doris-robot commented Sep 15, 2025

ClickBench: Total hot run time: 29.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35de9299c43f529831d2b9c85ede47b7032d1e97, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.06	0.06
query3	0.26	0.09	0.08
query4	1.60	0.11	0.11
query5	0.28	0.26	0.26
query6	1.18	0.66	0.65
query7	0.03	0.02	0.04
query8	0.05	0.04	0.04
query9	0.63	0.54	0.51
query10	0.59	0.59	0.57
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.64	0.63	0.63
query14	1.03	1.05	1.03
query15	0.90	0.88	0.87
query16	0.41	0.40	0.40
query17	1.08	1.04	1.06
query18	0.21	0.22	0.21
query19	1.95	1.88	1.82
query20	0.02	0.01	0.01
query21	15.39	0.96	0.59
query22	0.78	1.20	0.76
query23	14.81	1.44	0.61
query24	7.14	1.26	0.33
query25	0.27	0.10	0.10
query26	0.62	0.16	0.13
query27	0.07	0.06	0.05
query28	9.17	0.91	0.45
query29	12.55	4.02	3.25
query30	0.29	0.13	0.11
query31	2.83	0.61	0.39
query32	3.24	0.59	0.47
query33	3.19	3.18	3.05
query34	16.08	5.49	4.85
query35	4.92	4.90	4.97
query36	0.70	0.52	0.50
query37	0.10	0.08	0.07
query38	0.07	0.05	0.05
query39	0.04	0.03	0.03
query40	0.20	0.15	0.14
query41	0.08	0.03	0.03
query42	0.03	0.03	0.04
query43	0.04	0.04	0.04
Total cold run time: 103.94 s
Total hot run time: 29.52 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 15, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/12) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.21% (17439/33401)
Line Coverage 37.42% (158460/423450)
Region Coverage 32.00% (120874/377678)
Branch Coverage 33.36% (53026/158940)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 15, 2025

BE Regression && UT Coverage Report

Increment line coverage 83.33% (10/12) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.83% (23131/32656)
Line Coverage 57.24% (241778/422412)
Region Coverage 52.64% (201445/382653)
Branch Coverage 54.34% (86748/159639)

@github-actions
Copy link
Contributor

github-actions bot commented Sep 15, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 15, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 15, 2025

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 merged commit 8aef7c1 into apache:master Sep 15, 2025
29 of 34 checks passed
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Sep 28, 2025
@Gabriel39
[fix](union) Local shuffle for union operator (apache#56048)
3983fe0
@Gabriel39 Gabriel39 mentioned this pull request Sep 28, 2025
Merged
morrySnow pushed a commit that referenced this pull request Sep 29, 2025
@Gabriel39
branch-3.1: [fix](union) Local shuffle for union operator #56048 #56449…
be164cf 
… (#56556)

picked from #56048 #56449
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Gabriel39 @Thearas @doris-robot @hello-stephen @BiteTheDDDDt @yiguolei @morrySnow