Skip to content

branch-3.1: [fix](catalog)Disable Hadoop FileSystem cache for multi-configurationobject storage catalogs #57063 #57153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Oct 21, 2025
Merged

branch-3.1: [fix](catalog)Disable Hadoop FileSystem cache for multi-configurationobject storage catalogs #57063 #57153

morrySnow merged 1 commit into from
Oct 21, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Oct 20, 2025

Cherry-picked from #57063

@CalvinKirs
[fix](catalog)Disable Hadoop FileSystem cache for multi-configuration…
f98f604 
…object storage catalogs (#57063)

… 

Iceberg and Paimon catalogs access object storage (OSS, OBS, etc.) using
Hadoop FileSystem libraries. In environments with multiple storage
configurations, the FileSystem cache may cause instances to be shared
across configurations, leading to potential conflicts or incorrect
access.

This PR disables the Hadoop FileSystem cache
(`fs.oss.impl.disable.cache=true`) for each OSS to ensure configuration
isolation and prevent cache overwrites.

If necessary, the cache can also be explicitly enabled for specific
catalogs
by setting `fs.oss.impl.disable.cache=false`.
@github-actions github-actions bot requested a review from morrySnow as a code owner October 20, 2025 07:45
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Oct 20, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

TPC-H: Total hot run time: 32910 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f98f6041b1d9540912fd753aebbd4db86037290f, data reload: false

------ Round 1 ----------------------------------
q1	17598	5511	5517	5511
q2	2020	394	281	281
q3	12466	1225	771	771
q4	10333	859	455	455
q5	9408	2375	2137	2137
q6	188	168	132	132
q7	883	762	639	639
q8	9328	1420	1154	1154
q9	5294	4927	5002	4927
q10	6743	2282	1839	1839
q11	481	278	259	259
q12	351	372	219	219
q13	17775	3627	3030	3030
q14	224	220	209	209
q15	529	448	468	448
q16	446	432	371	371
q17	607	860	373	373
q18	7009	6495	6433	6433
q19	1229	950	556	556
q20	328	333	212	212
q21	2927	2187	1964	1964
q22	1065	1041	990	990
Total cold run time: 107232 ms
Total hot run time: 32910 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5550	5565	5527	5527
q2	235	334	230	230
q3	2208	2630	2290	2290
q4	1399	1759	1328	1328
q5	4437	5049	4949	4949
q6	172	169	130	130
q7	2075	1948	1822	1822
q8	2625	2855	2759	2759
q9	7298	7289	7187	7187
q10	3051	3289	2690	2690
q11	579	503	512	503
q12	684	807	575	575
q13	3415	3816	3162	3162
q14	285	289	285	285
q15	509	476	469	469
q16	449	485	447	447
q17	1255	1726	1261	1261
q18	7549	7523	7455	7455
q19	819	1100	1100	1100
q20	2025	2054	1903	1903
q21	5373	4840	4671	4671
q22	1114	1056	1018	1018
Total cold run time: 53106 ms
Total hot run time: 51761 ms

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

TPC-DS: Total hot run time: 193398 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f98f6041b1d9540912fd753aebbd4db86037290f, data reload: false

query1	944	415	449	415
query2	6310	1947	1860	1860
query3	8742	203	206	203
query4	34321	24692	24182	24182
query5	4370	599	467	467
query6	306	198	181	181
query7	4210	523	328	328
query8	315	254	248	248
query9	9445	2598	2590	2590
query10	489	311	248	248
query11	18130	15518	15244	15244
query12	148	109	105	105
query13	1539	529	425	425
query14	9629	6638	7367	6638
query15	285	196	176	176
query16	8573	682	460	460
query17	1545	778	604	604
query18	2195	418	326	326
query19	219	200	171	171
query20	132	121	124	121
query21	209	131	118	118
query22	4583	4730	4428	4428
query23	35309	34113	34168	34113
query24	7216	2708	2728	2708
query25	504	479	425	425
query26	805	285	174	174
query27	2083	453	352	352
query28	5786	2208	2169	2169
query29	685	579	466	466
query30	244	197	175	175
query31	947	917	830	830
query32	83	61	59	59
query33	528	365	325	325
query34	760	857	549	549
query35	794	843	765	765
query36	1055	1064	1011	1011
query37	105	96	74	74
query38	4062	4021	3997	3997
query39	1537	1482	1479	1479
query40	202	133	108	108
query41	49	49	46	46
query42	123	110	103	103
query43	505	519	495	495
query44	1317	833	836	833
query45	188	179	171	171
query46	894	1054	694	694
query47	1962	2006	1947	1947
query48	414	429	344	344
query49	739	498	405	405
query50	678	703	440	440
query51	7413	7281	7377	7281
query52	103	104	95	95
query53	231	265	208	208
query54	551	567	479	479
query55	84	79	82	79
query56	272	279	271	271
query57	1320	1271	1209	1209
query58	262	238	228	228
query59	3120	3186	3094	3094
query60	318	304	289	289
query61	150	135	143	135
query62	812	756	710	710
query63	241	205	204	204
query64	4046	1077	744	744
query65	3413	3286	3304	3286
query66	872	433	316	316
query67	16334	15816	15748	15748
query68	7672	836	534	534
query69	509	313	281	281
query70	1209	1092	1117	1092
query71	370	305	286	286
query72	5779	3743	3767	3743
query73	627	746	346	346
query74	10292	9482	9005	9005
query75	3176	3203	2667	2667
query76	2959	1188	781	781
query77	550	372	270	270
query78	10352	10421	9663	9663
query79	4091	835	579	579
query80	754	526	449	449
query81	533	253	221	221
query82	615	117	89	89
query83	172	158	150	150
query84	287	97	79	79
query85	798	368	302	302
query86	378	310	279	279
query87	4372	4328	4280	4280
query88	5197	2408	2415	2408
query89	409	333	296	296
query90	1858	192	194	192
query91	140	143	111	111
query92	65	59	51	51
query93	2290	889	538	538
query94	700	414	292	292
query95	355	288	277	277
query96	498	615	293	293
query97	3176	3255	3157	3157
query98	221	202	204	202
query99	1546	1405	1314	1314
Total cold run time: 297705 ms
Total hot run time: 193398 ms

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

ClickBench: Total hot run time: 29.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f98f6041b1d9540912fd753aebbd4db86037290f, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.06	0.07
query4	1.63	0.11	0.10
query5	0.53	0.52	0.51
query6	1.13	0.75	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.55	0.50	0.51
query10	0.56	0.56	0.56
query11	0.15	0.10	0.13
query12	0.14	0.11	0.11
query13	0.63	0.59	0.59
query14	0.78	0.80	0.80
query15	0.86	0.83	0.84
query16	0.37	0.38	0.39
query17	1.09	1.07	1.08
query18	0.24	0.22	0.22
query19	1.88	1.90	1.90
query20	0.02	0.02	0.02
query21	15.37	0.90	0.58
query22	0.74	0.75	0.60
query23	15.24	1.43	0.52
query24	2.94	1.42	1.70
query25	0.20	0.09	0.20
query26	0.27	0.15	0.14
query27	0.06	0.05	0.06
query28	13.98	0.98	0.43
query29	12.59	3.98	3.27
query30	0.25	0.09	0.07
query31	2.82	0.62	0.39
query32	3.22	0.54	0.47
query33	3.03	3.03	3.05
query34	16.83	5.23	4.55
query35	4.64	4.58	4.57
query36	0.64	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.26 s
Total hot run time: 29.55 s

@morrySnow morrySnow merged commit 0a2fc5c into branch-3.1 Oct 21, 2025
23 checks passed
@morrySnow morrySnow deleted the auto-pick-57063-branch-3.1 branch October 21, 2025 07:59
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @morrySnow @dataroaring @CalvinKirs