Skip to content

branch-3.1: [fix](coordinator) fix cte with local shuffle throw exception #52870 #52876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
924060929 merged 2 commits into from
Jul 8, 2025
Merged

branch-3.1: [fix](coordinator) fix cte with local shuffle throw exception #52870 #52876

924060929 merged 2 commits into from
Jul 8, 2025

Conversation

@924060929
Copy link
Contributor

@924060929 924060929 commented Jul 7, 2025 •
edited
Loading

cherry pick from #52870

@924060929
cherry pick from apache#52870
55b397d 
the CTE use MultiCastDataSink.destinations to store the destination instead of FragmentExecParams.destinations, but the old Coordinator use FragmentExecParams.destinations for local shuffle, so throw the exception:
```
[CANCELLED]Rows mismatched! Data may be lost. [Expected enqueue rows=14, Real enqueue rows=0, Detail: Type: HASH_SHUFFLE, Local Exchange Id: -10, Shuffled Map: [0:6], ]
```
@924060929 924060929 requested a review from morrySnow as a code owner July 7, 2025 08:02
@Thearas
Copy link
Contributor

Thearas commented Jul 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929
Copy link
Contributor Author

924060929 commented Jul 7, 2025

run buildall

@morrySnow morrySnow changed the title [fix](coordinator) fix cte with local shuffle throw exception (#52870) branch-3.1: [fix](coordinator) fix cte with local shuffle throw exception #52870 Jul 8, 2025
@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-H: Total hot run time: 39848 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40ac8e3a1d9098b82e04f5224ec1962ab2b0dba7, data reload: false

------ Round 1 ----------------------------------
q1	17618	6891	6598	6598
q2	2076	206	157	157
q3	10590	1127	1115	1115
q4	10234	731	743	731
q5	7767	2833	2786	2786
q6	220	132	131	131
q7	965	619	618	618
q8	9343	1930	1967	1930
q9	6624	6405	6409	6405
q10	7051	2275	2303	2275
q11	461	265	264	264
q12	432	209	212	209
q13	17797	2994	2972	2972
q14	230	212	220	212
q15	508	482	470	470
q16	486	382	369	369
q17	959	560	521	521
q18	7183	6665	6634	6634
q19	1318	1003	1010	1003
q20	494	204	200	200
q21	4039	3233	3236	3233
q22	1134	1015	1025	1015
Total cold run time: 107529 ms
Total hot run time: 39848 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6646	6507	6639	6507
q2	346	232	232	232
q3	2888	2816	2904	2816
q4	2072	1750	1836	1750
q5	5704	5711	5682	5682
q6	212	126	131	126
q7	2216	1829	1815	1815
q8	3352	3486	3473	3473
q9	8787	8874	8871	8871
q10	3572	3561	3504	3504
q11	613	489	503	489
q12	802	648	611	611
q13	8847	3198	3121	3121
q14	315	266	272	266
q15	496	456	477	456
q16	484	459	419	419
q17	1821	1615	1585	1585
q18	8257	7714	7724	7714
q19	1681	1582	1623	1582
q20	2076	1795	1813	1795
q21	5289	5010	5007	5007
q22	1107	1047	1044	1044
Total cold run time: 67583 ms
Total hot run time: 58865 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-DS: Total hot run time: 196506 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40ac8e3a1d9098b82e04f5224ec1962ab2b0dba7, data reload: false

query1	1264	903	898	898
query2	6218	1890	1911	1890
query3	10968	4405	4591	4405
query4	33353	23512	24000	23512
query5	4828	447	440	440
query6	294	197	193	193
query7	4011	322	325	322
query8	288	237	233	233
query9	9441	2580	2568	2568
query10	482	266	251	251
query11	18113	15146	15769	15146
query12	167	104	104	104
query13	1547	431	409	409
query14	10453	6963	6767	6767
query15	266	173	180	173
query16	7980	523	477	477
query17	1557	586	585	585
query18	2150	321	312	312
query19	359	152	151	151
query20	120	114	118	114
query21	209	108	108	108
query22	4889	4433	4434	4433
query23	34911	34172	34310	34172
query24	11179	2927	2937	2927
query25	659	418	425	418
query26	1270	174	168	168
query27	2725	380	362	362
query28	7432	2188	2193	2188
query29	888	468	454	454
query30	260	175	160	160
query31	1021	828	851	828
query32	96	64	59	59
query33	776	309	327	309
query34	916	509	507	507
query35	874	756	739	739
query36	1093	927	966	927
query37	138	71	68	68
query38	4145	3909	4008	3909
query39	1522	1441	1480	1441
query40	217	105	107	105
query41	51	52	52	52
query42	119	107	99	99
query43	544	472	473	472
query44	1314	810	800	800
query45	189	175	172	172
query46	1161	712	737	712
query47	2067	1917	1972	1917
query48	435	344	346	344
query49	963	390	397	390
query50	831	424	432	424
query51	7447	7341	7169	7169
query52	102	92	87	87
query53	256	183	184	183
query54	1350	487	473	473
query55	80	77	81	77
query56	264	267	244	244
query57	1346	1223	1188	1188
query58	232	214	220	214
query59	3328	3053	3109	3053
query60	275	259	259	259
query61	165	147	143	143
query62	855	704	713	704
query63	221	192	193	192
query64	5173	695	644	644
query65	3329	3268	3278	3268
query66	1423	302	293	293
query67	16036	15740	15519	15519
query68	4904	573	566	566
query69	434	257	262	257
query70	1127	1056	1067	1056
query71	335	252	250	250
query72	6312	4162	3981	3981
query73	757	350	351	350
query74	10250	9404	8907	8907
query75	3380	2619	2683	2619
query76	2644	1077	1144	1077
query77	389	266	273	266
query78	10603	9505	9553	9505
query79	2626	626	614	614
query80	1124	424	424	424
query81	550	223	215	215
query82	933	85	88	85
query83	243	143	140	140
query84	229	86	85	85
query85	1347	306	295	295
query86	432	301	289	289
query87	4419	4268	4206	4206
query88	4290	2387	2390	2387
query89	415	289	300	289
query90	2122	222	181	181
query91	144	105	108	105
query92	70	51	53	51
query93	2475	553	551	551
query94	934	281	306	281
query95	365	256	255	255
query96	607	277	274	274
query97	3305	3123	3126	3123
query98	222	212	215	212
query99	1538	1326	1286	1286
Total cold run time: 308160 ms
Total hot run time: 196506 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-H: Total hot run time: 40312 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40ac8e3a1d9098b82e04f5224ec1962ab2b0dba7, data reload: false

------ Round 1 ----------------------------------
q1	17597	6947	6680	6680
q2	2076	195	202	195
q3	10614	1169	1233	1169
q4	10442	754	711	711
q5	7759	3336	2835	2835
q6	224	138	136	136
q7	997	622	639	622
q8	9376	2006	2062	2006
q9	6685	6476	6467	6467
q10	7065	2303	2310	2303
q11	484	279	289	279
q12	409	228	215	215
q13	17799	2985	3049	2985
q14	265	212	212	212
q15	502	468	469	468
q16	482	390	388	388
q17	1004	640	603	603
q18	7336	6573	6719	6573
q19	1311	1066	1041	1041
q20	508	211	208	208
q21	3925	3238	3293	3238
q22	1106	991	978	978
Total cold run time: 107966 ms
Total hot run time: 40312 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6702	6664	6662	6662
q2	333	231	233	231
q3	3015	2930	2974	2930
q4	2158	1941	1839	1839
q5	5752	5754	5778	5754
q6	219	127	128	127
q7	2262	1864	1850	1850
q8	3400	3549	3567	3549
q9	8897	8965	8862	8862
q10	3599	3557	3523	3523
q11	580	494	489	489
q12	819	578	613	578
q13	10423	3172	3228	3172
q14	306	270	282	270
q15	504	477	467	467
q16	501	464	457	457
q17	1924	1653	1598	1598
q18	8410	7789	7641	7641
q19	1680	1674	1498	1498
q20	2074	1878	1842	1842
q21	5280	5186	5062	5062
q22	1112	1056	1013	1013
Total cold run time: 69950 ms
Total hot run time: 59414 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-DS: Total hot run time: 198217 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40ac8e3a1d9098b82e04f5224ec1962ab2b0dba7, data reload: false

query1	1324	906	894	894
query2	6254	1930	1926	1926
query3	10813	4269	4264	4264
query4	33061	24007	23552	23552
query5	4824	478	458	458
query6	286	187	199	187
query7	4006	326	334	326
query8	305	238	233	233
query9	9336	2625	2636	2625
query10	486	281	282	281
query11	18183	15257	15334	15257
query12	155	109	103	103
query13	1549	426	430	426
query14	8962	7485	7055	7055
query15	246	185	201	185
query16	8024	495	474	474
query17	1616	639	589	589
query18	2141	326	323	323
query19	223	172	171	171
query20	133	119	115	115
query21	210	112	108	108
query22	4646	4357	4384	4357
query23	35382	34329	34155	34155
query24	11256	2966	3004	2966
query25	662	448	446	446
query26	962	179	179	179
query27	2292	365	376	365
query28	7042	2215	2244	2215
query29	858	478	477	477
query30	267	159	161	159
query31	1047	863	870	863
query32	120	58	58	58
query33	782	314	305	305
query34	990	532	546	532
query35	869	741	739	739
query36	1128	955	942	942
query37	153	70	69	69
query38	4115	4011	4023	4011
query39	1540	1481	1513	1481
query40	207	106	109	106
query41	50	52	51	51
query42	117	110	110	110
query43	539	490	495	490
query44	1361	823	820	820
query45	188	177	177	177
query46	1176	746	756	746
query47	1984	1936	1870	1870
query48	462	336	349	336
query49	980	429	406	406
query50	857	446	433	433
query51	7389	7296	7248	7248
query52	111	96	90	90
query53	271	187	187	187
query54	1141	476	481	476
query55	87	80	81	80
query56	281	266	272	266
query57	1315	1206	1204	1204
query58	231	213	222	213
query59	3255	3036	2971	2971
query60	297	282	276	276
query61	122	153	156	153
query62	865	730	703	703
query63	231	208	201	201
query64	4119	713	656	656
query65	3524	3327	3336	3327
query66	820	305	307	305
query67	15795	15566	15463	15463
query68	4433	599	604	599
query69	443	280	275	275
query70	1194	1130	1117	1117
query71	331	265	262	262
query72	6320	4105	4054	4054
query73	794	366	367	366
query74	10495	9060	8987	8987
query75	3418	2655	2734	2655
query76	2735	1107	1212	1107
query77	405	304	288	288
query78	10500	9647	9698	9647
query79	2401	639	637	637
query80	1186	448	446	446
query81	568	224	227	224
query82	922	97	99	97
query83	236	143	142	142
query84	233	80	82	80
query85	1309	325	305	305
query86	432	303	294	294
query87	4373	4283	4214	4214
query88	4208	2503	2401	2401
query89	425	291	307	291
query90	1975	191	193	191
query91	145	111	108	108
query92	69	52	52	52
query93	2443	568	573	568
query94	861	311	291	291
query95	366	264	267	264
query96	637	279	284	279
query97	3285	3195	3208	3195
query98	239	208	201	201
query99	1576	1354	1345	1345
Total cold run time: 302980 ms
Total hot run time: 198217 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

ClickBench: Total hot run time: 30.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40ac8e3a1d9098b82e04f5224ec1962ab2b0dba7, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.06	0.07
query4	1.62	0.10	0.10
query5	0.51	0.52	0.54
query6	1.12	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.56	0.49	0.50
query10	0.56	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.12	0.12
query13	0.62	0.60	0.60
query14	0.79	0.79	0.80
query15	0.85	0.84	0.84
query16	0.38	0.37	0.39
query17	1.05	1.10	1.08
query18	0.23	0.22	0.22
query19	2.01	1.86	1.90
query20	0.01	0.02	0.01
query21	15.39	0.59	0.56
query22	2.46	1.97	1.33
query23	16.74	1.19	0.82
query24	2.89	1.17	1.18
query25	0.24	0.15	0.13
query26	0.36	0.15	0.14
query27	0.03	0.03	0.04
query28	10.17	0.49	0.51
query29	12.63	3.25	3.22
query30	0.25	0.08	0.06
query31	2.85	0.40	0.38
query32	3.23	0.47	0.46
query33	2.97	3.01	3.04
query34	16.84	4.50	4.53
query35	4.51	4.51	4.53
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.17	0.13	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 103.71 s
Total hot run time: 30.14 s

@924060929 924060929 merged commit 82452c7 into apache:branch-3.1 Jul 8, 2025
21 of 22 checks passed
@924060929 924060929 deleted the branch-3.1-fix-cte-with-local-shuffle branch July 8, 2025 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@924060929 @Thearas @doris-robot @morrySnow