Skip to content

branch-3.1: [fix](iceberg) Fix transaction issues #52716 #53157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 15, 2025
Merged

branch-3.1: [fix](iceberg) Fix transaction issues #52716 #53157

morrySnow merged 1 commit into from
Jul 15, 2025
+14 −10

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jul 13, 2025

Cherry-picked from #52716

@wuwenchi
[fix](iceberg) Fix transaction issues (#52716)
7d8219b 
### What problem does this PR solve?

Problem Summary:

1. In a transaction, the transaction API should be used, and commits
should not be made when finishInsert is called.
2. The ExecutorService for ReplacePartitions was omitted.

Co-authored-by: wuwenchi.wwc <wuwenchi.wwc@oceanbase.com>
@github-actions github-actions bot requested a review from morrySnow as a code owner July 13, 2025 00:55
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jul 13, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 13, 2025

TPC-H: Total hot run time: 39618 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7d8219bdc127ae971a246449ca1cc6c3936e133e, data reload: false

------ Round 1 ----------------------------------
q1	17573	6779	6617	6617
q2	2068	185	172	172
q3	10646	1124	1170	1124
q4	10352	739	701	701
q5	7758	2880	2819	2819
q6	215	134	136	134
q7	979	617	617	617
q8	9365	1915	2007	1915
q9	6609	6372	6377	6372
q10	7001	2268	2258	2258
q11	459	259	258	258
q12	481	211	210	210
q13	17773	2979	2982	2979
q14	236	207	201	201
q15	499	474	469	469
q16	467	375	377	375
q17	973	681	657	657
q18	7246	6586	6665	6586
q19	1331	1042	946	946
q20	472	202	200	200
q21	4026	3062	3098	3062
q22	1071	946	959	946
Total cold run time: 107600 ms
Total hot run time: 39618 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6627	6545	6569	6545
q2	331	231	237	231
q3	2885	2903	2909	2903
q4	2077	1809	1788	1788
q5	5729	5683	5706	5683
q6	202	130	130	130
q7	2198	1807	1832	1807
q8	3372	3493	3559	3493
q9	8791	8863	8816	8816
q10	3554	3561	3519	3519
q11	587	506	506	506
q12	848	629	642	629
q13	7481	3187	3285	3187
q14	299	290	284	284
q15	519	466	476	466
q16	513	450	439	439
q17	1897	1650	1662	1650
q18	8329	7841	7680	7680
q19	1691	1539	1572	1539
q20	2064	1813	1802	1802
q21	5168	5017	4960	4960
q22	1101	1021	1045	1021
Total cold run time: 66263 ms
Total hot run time: 59078 ms

@doris-robot
Copy link

doris-robot commented Jul 13, 2025

TPC-DS: Total hot run time: 197353 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7d8219bdc127ae971a246449ca1cc6c3936e133e, data reload: false

query1	1276	933	900	900
query2	6290	1908	1945	1908
query3	10796	4344	4301	4301
query4	33270	23654	23594	23594
query5	3840	481	466	466
query6	281	187	185	185
query7	3993	314	317	314
query8	304	233	227	227
query9	9742	2581	2578	2578
query10	481	266	272	266
query11	17979	15613	15477	15477
query12	169	107	105	105
query13	1567	423	436	423
query14	9475	7316	7135	7135
query15	250	186	193	186
query16	7938	482	494	482
query17	1697	605	600	600
query18	2162	318	337	318
query19	316	171	183	171
query20	126	118	113	113
query21	209	115	109	109
query22	4635	4423	4454	4423
query23	34864	33993	34357	33993
query24	11517	2906	2913	2906
query25	736	415	398	398
query26	2048	179	165	165
query27	2698	346	353	346
query28	7734	2152	2165	2152
query29	1047	444	446	444
query30	276	166	162	162
query31	1058	817	829	817
query32	99	58	55	55
query33	765	313	307	307
query34	1018	517	533	517
query35	923	731	724	724
query36	1074	942	948	942
query37	170	72	67	67
query38	4143	3936	4004	3936
query39	1518	1504	1454	1454
query40	265	104	110	104
query41	48	47	52	47
query42	117	109	101	101
query43	528	472	493	472
query44	1289	822	806	806
query45	185	171	180	171
query46	1147	740	732	732
query47	2008	1880	1915	1880
query48	447	353	336	336
query49	1034	400	401	400
query50	823	432	418	418
query51	7309	7167	7314	7167
query52	102	94	95	94
query53	261	187	187	187
query54	1389	503	503	503
query55	84	85	85	85
query56	276	263	266	263
query57	1344	1200	1200	1200
query58	237	219	213	213
query59	3203	3062	3034	3034
query60	291	257	250	250
query61	110	130	112	112
query62	870	684	695	684
query63	229	187	190	187
query64	4874	685	683	683
query65	3412	3286	3295	3286
query66	1407	309	334	309
query67	16085	15377	15445	15377
query68	4924	584	590	584
query69	442	269	261	261
query70	1131	1141	1125	1125
query71	341	268	261	261
query72	6211	4072	3961	3961
query73	741	345	358	345
query74	10327	9223	9339	9223
query75	3373	2599	2694	2599
query76	2645	1135	1112	1112
query77	379	273	288	273
query78	10438	9596	9514	9514
query79	1447	598	577	577
query80	1048	438	443	438
query81	546	220	220	220
query82	945	90	85	85
query83	237	144	143	143
query84	236	81	76	76
query85	1261	322	303	303
query86	395	274	303	274
query87	4348	4207	4238	4207
query88	3436	2414	2382	2382
query89	418	302	288	288
query90	1937	187	185	185
query91	141	108	105	105
query92	63	50	57	50
query93	1423	553	552	552
query94	893	297	301	297
query95	365	260	251	251
query96	610	280	288	280
query97	3295	3194	3111	3111
query98	218	203	202	202
query99	1538	1339	1302	1302
Total cold run time: 303610 ms
Total hot run time: 197353 ms

@doris-robot
Copy link

doris-robot commented Jul 13, 2025

ClickBench: Total hot run time: 29.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7d8219bdc127ae971a246449ca1cc6c3936e133e, data reload: false

query1	0.03	0.04	0.02
query2	0.06	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.11	0.10
query5	0.49	0.50	0.54
query6	1.13	0.73	0.74
query7	0.02	0.02	0.02
query8	0.03	0.03	0.03
query9	0.56	0.50	0.50
query10	0.55	0.55	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	0.78	0.79	0.83
query15	0.83	0.84	0.82
query16	0.38	0.38	0.38
query17	0.99	1.05	1.00
query18	0.23	0.21	0.22
query19	1.93	1.89	1.88
query20	0.01	0.01	0.01
query21	15.42	0.61	0.57
query22	2.52	1.69	1.87
query23	16.94	0.92	0.94
query24	2.65	0.47	1.60
query25	0.31	0.14	0.12
query26	0.32	0.14	0.13
query27	0.05	0.04	0.04
query28	10.75	0.51	0.46
query29	12.58	3.21	3.18
query30	0.24	0.06	0.07
query31	2.86	0.38	0.39
query32	3.25	0.46	0.46
query33	2.97	3.03	2.96
query34	17.03	4.45	4.47
query35	4.50	4.52	4.46
query36	0.67	0.48	0.48
query37	0.09	0.06	0.05
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 104.3 s
Total hot run time: 29.55 s

@morrySnow
Copy link
Contributor

morrySnow commented Jul 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 14, 2025

TPC-H: Total hot run time: 39479 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7d8219bdc127ae971a246449ca1cc6c3936e133e, data reload: false

------ Round 1 ----------------------------------
q1	17571	6914	6579	6579
q2	2081	189	171	171
q3	10595	1148	1110	1110
q4	10430	806	685	685
q5	7743	2867	2804	2804
q6	211	131	135	131
q7	972	612	583	583
q8	9357	1932	2053	1932
q9	6667	6387	6437	6387
q10	7003	2258	2259	2258
q11	470	263	261	261
q12	403	204	209	204
q13	17776	2968	2969	2968
q14	230	212	206	206
q15	515	468	463	463
q16	479	368	378	368
q17	985	580	534	534
q18	7249	6579	6713	6579
q19	1318	994	1030	994
q20	487	201	201	201
q21	3849	3106	3184	3106
q22	1093	955	978	955
Total cold run time: 107484 ms
Total hot run time: 39479 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6650	6548	6609	6548
q2	325	232	229	229
q3	2873	2848	2959	2848
q4	2008	1757	1843	1757
q5	5686	5747	5782	5747
q6	211	130	132	130
q7	2251	1842	1803	1803
q8	3416	3559	3550	3550
q9	8927	9039	9041	9039
q10	3551	3573	3575	3573
q11	616	499	487	487
q12	806	650	622	622
q13	9429	3198	3119	3119
q14	304	270	265	265
q15	509	466	452	452
q16	506	459	443	443
q17	1829	1621	1573	1573
q18	8193	7751	7806	7751
q19	1691	1594	1563	1563
q20	2079	1789	1800	1789
q21	5194	4923	5018	4923
q22	1161	1050	1022	1022
Total cold run time: 68215 ms
Total hot run time: 59233 ms

@doris-robot
Copy link

doris-robot commented Jul 14, 2025

TPC-DS: Total hot run time: 195647 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7d8219bdc127ae971a246449ca1cc6c3936e133e, data reload: false

query1	1287	898	882	882
query2	6300	1868	1913	1868
query3	10802	4325	4184	4184
query4	33077	23646	23432	23432
query5	3509	468	443	443
query6	257	173	181	173
query7	3988	312	328	312
query8	295	232	240	232
query9	9288	2596	2589	2589
query10	458	258	257	257
query11	17857	15354	15220	15220
query12	159	106	108	106
query13	1585	434	432	432
query14	9637	7244	6858	6858
query15	244	187	199	187
query16	8033	491	482	482
query17	1621	610	613	610
query18	2194	328	319	319
query19	262	168	171	168
query20	128	114	130	114
query21	207	106	109	106
query22	4693	4352	4344	4344
query23	34818	34111	33591	33591
query24	11404	2985	2922	2922
query25	668	429	441	429
query26	1590	182	172	172
query27	2919	357	344	344
query28	8147	2170	2145	2145
query29	1009	469	489	469
query30	283	170	165	165
query31	1045	822	872	822
query32	93	57	56	56
query33	763	301	298	298
query34	1008	502	537	502
query35	838	731	714	714
query36	1121	959	930	930
query37	124	73	75	73
query38	4149	3957	4012	3957
query39	1523	1515	1467	1467
query40	257	99	96	96
query41	48	49	48	48
query42	117	102	106	102
query43	530	487	483	483
query44	1293	820	812	812
query45	184	171	177	171
query46	1183	728	781	728
query47	2031	1906	1894	1894
query48	434	344	355	344
query49	994	403	377	377
query50	858	424	433	424
query51	7615	7219	7300	7219
query52	107	93	96	93
query53	259	196	182	182
query54	1270	464	489	464
query55	78	81	79	79
query56	280	255	244	244
query57	1332	1208	1204	1204
query58	228	216	212	212
query59	3209	3067	2945	2945
query60	293	262	274	262
query61	145	111	107	107
query62	861	716	677	677
query63	222	197	197	197
query64	4983	673	640	640
query65	3335	3279	3343	3279
query66	1308	298	308	298
query67	15747	15393	15311	15311
query68	4813	594	589	589
query69	437	268	265	265
query70	1186	1151	1139	1139
query71	347	258	262	258
query72	6208	3989	3925	3925
query73	755	352	362	352
query74	10493	9013	9118	9013
query75	3393	2677	2652	2652
query76	2623	1084	973	973
query77	375	278	276	276
query78	10459	9583	9645	9583
query79	2503	620	605	605
query80	1104	428	426	426
query81	554	223	218	218
query82	704	87	90	87
query83	250	148	143	143
query84	240	83	85	83
query85	1668	306	294	294
query86	478	301	289	289
query87	4357	4246	4220	4220
query88	3989	2388	2380	2380
query89	414	293	295	293
query90	2097	188	184	184
query91	140	107	107	107
query92	63	53	51	51
query93	2406	562	571	562
query94	961	283	294	283
query95	362	251	259	251
query96	626	282	281	281
query97	3283	3149	3117	3117
query98	226	213	200	200
query99	1495	1282	1286	1282
Total cold run time: 305593 ms
Total hot run time: 195647 ms

@doris-robot
Copy link

doris-robot commented Jul 14, 2025

ClickBench: Total hot run time: 28.49 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7d8219bdc127ae971a246449ca1cc6c3936e133e, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.52	0.50	0.50
query6	1.15	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.56	0.50	0.50
query10	0.54	0.55	0.56
query11	0.14	0.11	0.11
query12	0.13	0.11	0.11
query13	0.60	0.59	0.59
query14	0.77	0.79	0.78
query15	0.84	0.83	0.81
query16	0.39	0.39	0.40
query17	1.08	1.05	1.03
query18	0.24	0.20	0.20
query19	1.97	1.90	1.78
query20	0.02	0.01	0.01
query21	15.42	0.59	0.58
query22	2.14	2.04	0.89
query23	17.01	1.11	0.71
query24	2.93	1.80	0.59
query25	0.24	0.11	0.08
query26	0.52	0.13	0.14
query27	0.04	0.04	0.07
query28	10.25	0.48	0.44
query29	12.60	3.14	3.15
query30	0.25	0.06	0.05
query31	2.86	0.40	0.37
query32	3.25	0.46	0.45
query33	2.93	3.03	3.01
query34	16.81	4.44	4.42
query35	4.47	4.46	4.48
query36	0.67	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.15	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.85 s
Total hot run time: 28.49 s

@morrySnow morrySnow merged commit f920b94 into branch-3.1 Jul 15, 2025
21 of 22 checks passed
@github-actions github-actions bot deleted the auto-pick-52716-branch-3.1 branch July 15, 2025 02:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @morrySnow @dataroaring @wuwenchi