Skip to content

branch-3.0: [fix](iceberg)Fix the thread pool issue used for commit. (#51508) #53155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
Jul 14, 2025
Merged

branch-3.0: [fix](iceberg)Fix the thread pool issue used for commit. (#51508) #53155

dataroaring merged 2 commits into from
Jul 14, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Jul 13, 2025

bp #51508

wuwenchi and others added 2 commits July 12, 2025 17:44
@wuwenchi @morningman
[fix](iceberg)Fix the thread pool issue used for commit. (apache#51508)
19a7c5c 
### What problem does this PR solve?

Problem Summary:

When Iceberg generates a new snapshot, it performs a merge operation
based on the previous snapshot. This operation reads manifest files, and
the file reading process uses a global thread pool. However, users may
have their own authentication information, which requires the use of
doAs to ensure context. Therefore, the thread pool provided by Iceberg
cannot be used.

### Release note

None
@morningman
patch
cfc848c
@morningman morningman requested a review from dataroaring as a code owner July 13, 2025 00:50
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Jul 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 13, 2025

TPC-H: Total hot run time: 39425 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cfc848c71e9e95e995eb1d85a61ade82b26d54ee, data reload: false

------ Round 1 ----------------------------------
q1	17584	6937	6592	6592
q2	2060	172	158	158
q3	10683	1107	1177	1107
q4	10538	713	705	705
q5	7751	3310	2807	2807
q6	213	135	130	130
q7	955	616	601	601
q8	9345	1906	1969	1906
q9	6606	6373	6364	6364
q10	7004	2277	2265	2265
q11	465	260	254	254
q12	390	220	222	220
q13	17774	3000	2962	2962
q14	245	203	209	203
q15	510	463	461	461
q16	486	379	367	367
q17	964	578	484	484
q18	7205	6656	6573	6573
q19	1405	1104	1030	1030
q20	463	194	199	194
q21	3886	3126	3050	3050
q22	1097	992	993	992
Total cold run time: 107629 ms
Total hot run time: 39425 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6675	6570	6594	6570
q2	331	234	228	228
q3	2887	2918	2885	2885
q4	1983	1784	1827	1784
q5	5683	5685	5728	5685
q6	210	131	130	130
q7	2243	1809	1845	1809
q8	3346	3504	3537	3504
q9	8712	8848	8840	8840
q10	3570	3554	3504	3504
q11	604	500	478	478
q12	820	604	596	596
q13	8007	3173	3167	3167
q14	292	270	284	270
q15	507	458	461	458
q16	517	452	448	448
q17	1835	1642	1625	1625
q18	8160	7723	7648	7648
q19	1686	1500	1414	1414
q20	2018	1807	1871	1807
q21	5200	4996	4934	4934
q22	1143	1074	1048	1048
Total cold run time: 66429 ms
Total hot run time: 58832 ms

@doris-robot
Copy link

doris-robot commented Jul 13, 2025

TPC-DS: Total hot run time: 197540 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cfc848c71e9e95e995eb1d85a61ade82b26d54ee, data reload: false

query1	1248	913	887	887
query2	6231	1968	1941	1941
query3	10971	4578	4627	4578
query4	33252	23754	23401	23401
query5	4066	493	437	437
query6	280	173	172	172
query7	3996	321	328	321
query8	310	227	225	225
query9	9418	2584	2573	2573
query10	482	274	258	258
query11	17954	15162	15589	15162
query12	156	102	104	102
query13	1556	423	417	417
query14	9977	6837	7423	6837
query15	230	188	189	188
query16	7801	502	468	468
query17	1628	618	606	606
query18	2120	322	339	322
query19	229	174	163	163
query20	121	119	121	119
query21	204	108	112	108
query22	4852	4404	4408	4404
query23	35148	34602	34856	34602
query24	12028	2984	2981	2981
query25	729	436	434	434
query26	2107	176	175	175
query27	3143	360	361	360
query28	7560	2183	2151	2151
query29	1102	471	474	471
query30	254	163	165	163
query31	1068	829	846	829
query32	95	59	55	55
query33	765	306	314	306
query34	1147	523	542	523
query35	850	732	729	729
query36	1133	949	974	949
query37	191	66	66	66
query38	4053	4028	3967	3967
query39	1526	1484	1464	1464
query40	268	109	107	107
query41	52	53	48	48
query42	125	106	102	102
query43	526	490	492	490
query44	1316	812	836	812
query45	187	169	175	169
query46	1168	756	744	744
query47	1963	1883	1930	1883
query48	505	382	384	382
query49	989	412	424	412
query50	842	438	432	432
query51	7386	7392	7198	7198
query52	98	87	88	87
query53	268	193	191	191
query54	1106	468	464	464
query55	80	86	80	80
query56	282	255	259	255
query57	1277	1214	1178	1178
query58	249	213	224	213
query59	3270	3120	3002	3002
query60	315	265	291	265
query61	119	110	163	110
query62	904	720	686	686
query63	235	200	197	197
query64	5103	696	633	633
query65	3382	3295	3252	3252
query66	1112	293	334	293
query67	15671	15455	15420	15420
query68	4556	600	584	584
query69	414	267	270	267
query70	1168	1127	1108	1108
query71	326	268	259	259
query72	6171	4017	4078	4017
query73	761	348	357	348
query74	10085	9133	8984	8984
query75	3354	2646	2689	2646
query76	2625	1110	1033	1033
query77	408	283	285	283
query78	10535	9590	9542	9542
query79	2475	622	610	610
query80	1040	442	442	442
query81	539	224	223	223
query82	940	92	89	89
query83	231	142	147	142
query84	238	84	83	83
query85	1196	312	289	289
query86	412	271	302	271
query87	4366	4219	4278	4219
query88	4330	2397	2487	2397
query89	423	291	295	291
query90	1816	187	185	185
query91	186	148	154	148
query92	62	52	52	52
query93	2268	549	550	549
query94	728	300	298	298
query95	360	274	258	258
query96	623	296	282	282
query97	3277	3104	3117	3104
query98	227	213	203	203
query99	1509	1320	1315	1315
Total cold run time: 306428 ms
Total hot run time: 197540 ms

@doris-robot
Copy link

doris-robot commented Jul 13, 2025

ClickBench: Total hot run time: 29.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cfc848c71e9e95e995eb1d85a61ade82b26d54ee, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.02	0.03
query3	0.24	0.07	0.06
query4	1.63	0.10	0.11
query5	0.52	0.52	0.51
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.56	0.50	0.50
query10	0.55	0.54	0.56
query11	0.14	0.11	0.11
query12	0.15	0.11	0.10
query13	0.61	0.60	0.59
query14	0.77	0.78	0.81
query15	0.84	0.83	0.83
query16	0.37	0.38	0.38
query17	1.04	1.02	1.02
query18	0.23	0.21	0.22
query19	1.93	1.79	1.88
query20	0.01	0.01	0.02
query21	15.39	0.58	0.56
query22	2.60	1.97	1.28
query23	17.01	1.00	0.76
query24	3.06	1.68	1.28
query25	0.17	0.08	0.20
query26	0.58	0.13	0.15
query27	0.05	0.04	0.03
query28	9.68	0.49	0.51
query29	12.58	3.24	3.22
query30	0.25	0.06	0.06
query31	2.90	0.39	0.38
query32	3.23	0.47	0.45
query33	2.94	2.99	2.98
query34	16.94	4.56	4.55
query35	4.51	4.50	4.64
query36	0.65	0.48	0.47
query37	0.09	0.06	0.06
query38	0.06	0.03	0.04
query39	0.04	0.02	0.03
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.93 s
Total hot run time: 29.86 s

dataroaring
dataroaring approved these changes Jul 14, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 6413ebc into apache:branch-3.0 Jul 14, 2025
23 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morningman @hello-stephen @doris-robot @dataroaring @wuwenchi