Skip to content

[fix](iceberg) Fix transaction issues #52716

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Jul 13, 2025
Merged

[fix](iceberg) Fix transaction issues #52716

morningman merged 2 commits into from
Jul 13, 2025

Conversation

@wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Jul 3, 2025

What problem does this PR solve?

Problem Summary:

  1. In a transaction, the transaction API should be used, and commits should not be made when finishInsert is called.
  2. The ExecutorService for ReplacePartitions was omitted.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jul 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Jul 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 3, 2025

TPC-H: Total hot run time: 34031 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3622287916843747d3787e66cb1731ee28bf3567, data reload: false

------ Round 1 ----------------------------------
q1	17597	5215	5008	5008
q2	1950	301	176	176
q3	10487	1282	716	716
q4	10311	1028	513	513
q5	8924	2550	2348	2348
q6	178	163	128	128
q7	885	744	597	597
q8	9320	1262	1042	1042
q9	6978	5106	5091	5091
q10	6880	2390	1967	1967
q11	476	300	298	298
q12	340	350	214	214
q13	17779	3671	3350	3350
q14	217	224	204	204
q15	563	490	482	482
q16	415	417	377	377
q17	614	842	401	401
q18	7494	7244	7047	7047
q19	1212	963	537	537
q20	331	350	223	223
q21	3853	2533	2336	2336
q22	1044	1023	976	976
Total cold run time: 107848 ms
Total hot run time: 34031 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5106	5040	5014	5014
q2	239	327	216	216
q3	2174	2717	2262	2262
q4	1430	1753	1359	1359
q5	4209	4541	4463	4463
q6	223	169	132	132
q7	2041	1957	1845	1845
q8	2642	2582	2582	2582
q9	7416	7269	7307	7269
q10	3146	3331	2903	2903
q11	582	529	512	512
q12	663	784	603	603
q13	3548	3884	3348	3348
q14	292	283	309	283
q15	522	488	509	488
q16	649	487	435	435
q17	1166	1581	1353	1353
q18	7953	7735	7577	7577
q19	781	746	765	746
q20	1977	1945	1830	1830
q21	4741	4366	4297	4297
q22	1057	1039	987	987
Total cold run time: 52557 ms
Total hot run time: 50504 ms

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Jul 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 3, 2025

TPC-H: Total hot run time: 33854 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 012ef67864c79fae878671747b9e027adecbab8b, data reload: false

------ Round 1 ----------------------------------
q1	17593	5213	5050	5050
q2	1930	275	187	187
q3	10320	1278	701	701
q4	10218	1022	514	514
q5	7522	2399	2325	2325
q6	186	160	129	129
q7	874	776	586	586
q8	9299	1262	1059	1059
q9	6870	5106	5074	5074
q10	6899	2381	1976	1976
q11	469	300	276	276
q12	342	349	215	215
q13	17757	3724	3063	3063
q14	220	220	226	220
q15	545	484	489	484
q16	418	424	378	378
q17	618	860	379	379
q18	7941	7164	7164	7164
q19	1204	980	545	545
q20	340	344	223	223
q21	3735	3202	2337	2337
q22	1054	1013	969	969
Total cold run time: 106354 ms
Total hot run time: 33854 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5118	5119	5137	5119
q2	250	338	219	219
q3	2165	2668	2312	2312
q4	1364	1770	1327	1327
q5	4221	4518	4439	4439
q6	204	161	126	126
q7	2135	1951	1864	1864
q8	2597	2731	2562	2562
q9	7329	7312	7248	7248
q10	3111	3249	2889	2889
q11	587	512	496	496
q12	673	768	881	768
q13	3462	3943	3562	3562
q14	290	311	274	274
q15	528	470	477	470
q16	458	490	441	441
q17	1185	1447	1439	1439
q18	8121	7693	7441	7441
q19	794	845	1085	845
q20	2063	2100	1803	1803
q21	4670	4395	4212	4212
q22	1077	1032	991	991
Total cold run time: 52402 ms
Total hot run time: 50847 ms

@doris-robot
Copy link

doris-robot commented Jul 3, 2025

TPC-DS: Total hot run time: 184123 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 012ef67864c79fae878671747b9e027adecbab8b, data reload: false

query1	989	392	377	377
query2	6556	1753	1682	1682
query3	6749	216	208	208
query4	26053	23920	22956	22956
query5	4361	577	426	426
query6	308	217	203	203
query7	4632	490	294	294
query8	280	224	225	224
query9	8626	2632	2659	2632
query10	481	345	273	273
query11	15352	15048	14817	14817
query12	158	109	100	100
query13	1646	525	421	421
query14	8495	5645	5751	5645
query15	228	193	176	176
query16	7361	634	445	445
query17	1185	709	555	555
query18	1972	396	288	288
query19	187	195	153	153
query20	118	119	136	119
query21	214	123	103	103
query22	4308	4048	4030	4030
query23	33999	33034	32764	32764
query24	8442	2361	2376	2361
query25	540	474	385	385
query26	1231	266	144	144
query27	2762	516	351	351
query28	4278	2119	2094	2094
query29	773	571	428	428
query30	279	228	194	194
query31	875	853	745	745
query32	74	62	58	58
query33	572	365	307	307
query34	786	822	510	510
query35	803	848	739	739
query36	911	979	884	884
query37	113	99	75	75
query38	4226	4155	4145	4145
query39	1479	1438	1418	1418
query40	209	116	106	106
query41	56	55	84	55
query42	124	108	120	108
query43	494	516	484	484
query44	1308	822	809	809
query45	176	165	163	163
query46	857	1080	634	634
query47	1746	1811	1697	1697
query48	379	409	311	311
query49	732	496	387	387
query50	643	688	411	411
query51	4104	4276	4121	4121
query52	106	106	99	99
query53	227	255	183	183
query54	570	574	500	500
query55	82	79	80	79
query56	314	305	286	286
query57	1161	1201	1127	1127
query58	268	250	261	250
query59	2595	2738	2549	2549
query60	323	326	296	296
query61	127	118	120	118
query62	808	709	672	672
query63	223	197	192	192
query64	4363	1102	796	796
query65	4238	4200	4167	4167
query66	1143	402	308	308
query67	15608	15532	15278	15278
query68	7903	897	521	521
query69	476	300	268	268
query70	1175	1144	1094	1094
query71	445	318	289	289
query72	5852	4735	4690	4690
query73	696	596	349	349
query74	8966	9123	8891	8891
query75	3760	3185	2684	2684
query76	3617	1138	719	719
query77	785	388	298	298
query78	9950	10142	9434	9434
query79	2031	789	575	575
query80	579	524	452	452
query81	466	264	226	226
query82	419	122	95	95
query83	254	266	238	238
query84	246	104	91	91
query85	788	342	306	306
query86	339	318	272	272
query87	4387	4479	4305	4305
query88	3497	2277	2264	2264
query89	389	319	282	282
query90	1917	200	200	200
query91	137	149	116	116
query92	150	57	51	51
query93	1417	937	586	586
query94	675	388	299	299
query95	367	283	280	280
query96	487	580	275	275
query97	2732	2781	2602	2602
query98	223	207	201	201
query99	1426	1392	1255	1255
Total cold run time: 271937 ms
Total hot run time: 184123 ms

@doris-robot
Copy link

doris-robot commented Jul 3, 2025

ClickBench: Total hot run time: 30.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 012ef67864c79fae878671747b9e027adecbab8b, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.05	0.04
query3	0.24	0.07	0.08
query4	1.60	0.10	0.11
query5	0.43	0.45	0.42
query6	1.16	0.66	0.66
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.52	0.52
query10	0.56	0.57	0.58
query11	0.16	0.11	0.10
query12	0.15	0.12	0.11
query13	0.62	0.62	0.60
query14	0.80	0.81	0.81
query15	0.90	0.86	0.86
query16	0.39	0.38	0.40
query17	1.08	1.05	1.06
query18	0.23	0.21	0.21
query19	1.91	1.92	1.83
query20	0.02	0.01	0.01
query21	15.39	0.89	0.54
query22	0.75	1.22	0.89
query23	14.72	1.40	0.63
query24	7.33	0.92	1.31
query25	0.48	0.22	0.17
query26	0.60	0.16	0.15
query27	0.06	0.05	0.05
query28	9.43	0.90	0.44
query29	12.56	4.09	3.44
query30	0.25	0.10	0.06
query31	2.84	0.60	0.39
query32	3.26	0.56	0.47
query33	3.14	3.06	3.18
query34	16.13	5.37	4.77
query35	4.76	4.86	4.84
query36	0.68	0.50	0.49
query37	0.10	0.07	0.07
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.19	0.14	0.13
query41	0.09	0.02	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.95 s
Total hot run time: 30.05 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 3, 2025

FE UT Coverage Report

Increment line coverage 91.67% (11/12) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 10, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jul 10, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 10, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit d0654a3 into apache:master Jul 13, 2025
30 checks passed
github-actions bot pushed a commit that referenced this pull request Jul 13, 2025
@wuwenchi
[fix](iceberg) Fix transaction issues (#52716)
7d8219b 
### What problem does this PR solve?

Problem Summary:

1. In a transaction, the transaction API should be used, and commits
should not be made when finishInsert is called.
2. The ExecutorService for ReplacePartitions was omitted.

Co-authored-by: wuwenchi.wwc <wuwenchi.wwc@oceanbase.com>
@github-actions github-actions bot mentioned this pull request Jul 13, 2025
Merged
morningman pushed a commit to morningman/doris that referenced this pull request Jul 14, 2025
@wuwenchi @morningman
[fix](iceberg) Fix transaction issues (apache#52716)
9d7fb0f 
Problem Summary:

1. In a transaction, the transaction API should be used, and commits
should not be made when finishInsert is called.
2. The ExecutorService for ReplacePartitions was omitted.

Co-authored-by: wuwenchi.wwc <wuwenchi.wwc@oceanbase.com>
@morningman morningman mentioned this pull request Jul 14, 2025
Merged
morningman pushed a commit to morningman/doris that referenced this pull request Jul 14, 2025
@wuwenchi @morningman
[fix](iceberg) Fix transaction issues (apache#52716)
5d1c08f 
Problem Summary:

1. In a transaction, the transaction API should be used, and commits
should not be made when finishInsert is called.
2. The ExecutorService for ReplacePartitions was omitted.

Co-authored-by: wuwenchi.wwc <wuwenchi.wwc@oceanbase.com>
@morningman morningman mentioned this pull request Jul 14, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jul 15, 2025
@github-actions @wuwenchi
branch-3.1: [fix](iceberg) Fix transaction issues #52716 (#53157)
f920b94 
Cherry-picked from #52716

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
Co-authored-by: wuwenchi.wwc <wuwenchi.wwc@oceanbase.com>
dataroaring pushed a commit that referenced this pull request Jul 15, 2025
@morningman @wuwenchi
branch-3.0: [fix](iceberg) Fix transaction issues (#52716) (#53237)
121641e 
bp #52716

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
Co-authored-by: wuwenchi.wwc <wuwenchi.wwc@oceanbase.com>
yiguolei pushed a commit that referenced this pull request Jul 16, 2025
@morningman @wuwenchi
branch-2.1: [fix](iceberg) Fix transaction issues (#52716) (#53238)
140fb4c 
bp #52716

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
Co-authored-by: wuwenchi.wwc <wuwenchi.wwc@oceanbase.com>
@yiguolei yiguolei mentioned this pull request Aug 11, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

+1 more reviewer

@suxiaogang223 suxiaogang223 suxiaogang223 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@wuwenchi @Thearas @doris-robot @hello-stephen @morningman @suxiaogang223 @yiguolei @dataroaring @morrySnow