Skip to content

branch-3.0: [fix](iceberg) Fix transaction issues (#52716) #53237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jul 15, 2025
Merged

branch-3.0: [fix](iceberg) Fix transaction issues (#52716) #53237

dataroaring merged 1 commit into from
Jul 15, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Jul 14, 2025

bp #52716

@wuwenchi @morningman
[fix](iceberg) Fix transaction issues (apache#52716)
9d7fb0f 
Problem Summary:

1. In a transaction, the transaction API should be used, and commits
should not be made when finishInsert is called.
2. The ExecutorService for ReplacePartitions was omitted.

Co-authored-by: wuwenchi.wwc <wuwenchi.wwc@oceanbase.com>
@morningman morningman requested a review from dataroaring as a code owner July 14, 2025 18:59
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Jul 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 14, 2025

TPC-H: Total hot run time: 39473 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9d7fb0fbafc677284e7064fbe811295b217167fd, data reload: false

------ Round 1 ----------------------------------
q1	17583	6964	6634	6634
q2	2055	209	170	170
q3	10648	1131	1114	1114
q4	10488	730	700	700
q5	7758	2782	2758	2758
q6	209	133	131	131
q7	969	612	591	591
q8	9340	1908	2049	1908
q9	6664	6405	6427	6405
q10	7029	2295	2239	2239
q11	462	253	261	253
q12	401	210	208	208
q13	17796	2955	2989	2955
q14	232	199	206	199
q15	504	487	469	469
q16	492	398	385	385
q17	971	513	582	513
q18	7300	6573	6765	6573
q19	1405	1092	1013	1013
q20	464	199	199	199
q21	3869	3205	3075	3075
q22	1131	983	981	981
Total cold run time: 107770 ms
Total hot run time: 39473 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6620	6609	6705	6609
q2	328	229	236	229
q3	2948	2976	2964	2964
q4	2127	1847	1879	1847
q5	5782	5779	5733	5733
q6	210	132	132	132
q7	2264	1866	1855	1855
q8	3387	3563	3559	3559
q9	8826	8994	8925	8925
q10	3583	3524	3563	3524
q11	609	516	501	501
q12	824	621	642	621
q13	7228	3186	3106	3106
q14	297	272	274	272
q15	513	459	470	459
q16	480	443	451	443
q17	1836	1612	1584	1584
q18	8188	7794	7667	7667
q19	1726	1555	1615	1555
q20	2049	1820	1830	1820
q21	5198	5001	4935	4935
q22	1112	1089	1032	1032
Total cold run time: 66135 ms
Total hot run time: 59372 ms

@doris-robot
Copy link

doris-robot commented Jul 14, 2025

TPC-DS: Total hot run time: 196258 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9d7fb0fbafc677284e7064fbe811295b217167fd, data reload: false

query1	1271	914	886	886
query2	6290	1928	1893	1893
query3	10974	4355	4293	4293
query4	33313	23542	23507	23507
query5	3687	464	450	450
query6	285	179	177	177
query7	3982	316	321	316
query8	287	222	221	221
query9	9523	2597	2570	2570
query10	467	260	259	259
query11	17989	15250	15197	15197
query12	158	100	99	99
query13	1548	426	415	415
query14	8634	7197	6647	6647
query15	239	190	198	190
query16	8037	485	489	485
query17	1578	581	556	556
query18	2116	309	305	305
query19	214	153	152	152
query20	123	111	112	111
query21	204	101	108	101
query22	5004	4414	4401	4401
query23	34620	34283	34356	34283
query24	11183	2959	2869	2869
query25	670	391	405	391
query26	1251	164	168	164
query27	2459	353	352	352
query28	7609	2183	2205	2183
query29	874	457	418	418
query30	271	158	161	158
query31	1040	823	825	823
query32	97	59	55	55
query33	755	307	288	288
query34	1148	512	507	507
query35	869	747	729	729
query36	1079	940	972	940
query37	122	69	67	67
query38	4050	3954	3952	3952
query39	1546	1475	1476	1475
query40	215	101	102	101
query41	49	50	49	49
query42	112	103	102	102
query43	516	478	481	478
query44	1302	834	818	818
query45	180	169	167	167
query46	1160	730	720	720
query47	1986	1931	1946	1931
query48	492	411	390	390
query49	953	394	386	386
query50	821	422	427	422
query51	7450	7306	7127	7127
query52	95	88	91	88
query53	261	181	182	181
query54	1149	486	469	469
query55	76	76	77	76
query56	261	243	248	243
query57	1322	1238	1192	1192
query58	247	208	209	208
query59	3221	3112	2975	2975
query60	287	260	264	260
query61	112	109	112	109
query62	848	693	676	676
query63	232	203	186	186
query64	4177	682	658	658
query65	3348	3305	3278	3278
query66	785	312	304	304
query67	15772	15549	15418	15418
query68	4464	612	612	612
query69	432	267	267	267
query70	1154	1071	1124	1071
query71	334	268	258	258
query72	6485	4144	4114	4114
query73	745	342	362	342
query74	10162	9052	8881	8881
query75	3362	2673	2594	2594
query76	2737	1097	1135	1097
query77	385	265	261	261
query78	10510	9619	9606	9606
query79	2416	603	623	603
query80	1186	423	414	414
query81	564	216	214	214
query82	692	86	85	85
query83	238	149	145	145
query84	245	77	78	77
query85	1555	292	288	288
query86	487	281	301	281
query87	4335	4298	4175	4175
query88	4028	2395	2345	2345
query89	412	295	292	292
query90	1924	182	183	182
query91	182	144	144	144
query92	65	48	46	46
query93	2241	544	558	544
query94	852	293	292	292
query95	359	249	249	249
query96	607	275	289	275
query97	3303	3129	3159	3129
query98	218	194	197	194
query99	1520	1284	1323	1284
Total cold run time: 301194 ms
Total hot run time: 196258 ms

@doris-robot
Copy link

doris-robot commented Jul 14, 2025

ClickBench: Total hot run time: 29.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9d7fb0fbafc677284e7064fbe811295b217167fd, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.11	0.10
query5	0.53	0.54	0.52
query6	1.13	0.73	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.55	0.49	0.50
query10	0.56	0.55	0.55
query11	0.13	0.10	0.10
query12	0.14	0.12	0.11
query13	0.60	0.59	0.60
query14	0.78	0.79	0.80
query15	0.85	0.82	0.82
query16	0.39	0.40	0.39
query17	1.05	0.97	1.02
query18	0.23	0.22	0.21
query19	1.90	1.83	1.74
query20	0.01	0.01	0.01
query21	15.39	0.58	0.60
query22	2.29	1.79	2.02
query23	17.08	0.97	0.79
query24	3.00	0.27	1.51
query25	0.31	0.16	0.11
query26	0.29	0.14	0.14
query27	0.05	0.04	0.04
query28	10.69	0.49	0.45
query29	12.58	3.26	3.24
query30	0.25	0.06	0.06
query31	2.86	0.39	0.37
query32	3.25	0.45	0.46
query33	2.93	2.99	2.97
query34	17.12	4.56	4.60
query35	4.54	4.55	4.52
query36	0.66	0.47	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 104.63 s
Total hot run time: 29.38 s

dataroaring
dataroaring approved these changes Jul 15, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 121641e into apache:branch-3.0 Jul 15, 2025
23 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morningman @hello-stephen @doris-robot @dataroaring @wuwenchi