Skip to content

branch-3.0: [fix](hdfs) remove cached file handle when read fails #54926 #54991

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Aug 20, 2025
Merged

branch-3.0: [fix](hdfs) remove cached file handle when read fails #54926 #54991

dataroaring merged 1 commit into from
Aug 20, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 19, 2025

Cherry-picked from #54926

@github-actions github-actions bot requested a review from dataroaring as a code owner August 19, 2025 07:05
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Aug 19, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 19, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-H: Total hot run time: 39845 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2af40c85b0c1f51a35b3055e1e0a3fdbca42d997, data reload: false

------ Round 1 ----------------------------------
q1	17614	7029	6743	6743
q2	2060	206	168	168
q3	10583	1130	1152	1130
q4	10237	764	710	710
q5	7745	2829	2816	2816
q6	213	133	135	133
q7	966	633	614	614
q8	9356	1966	2049	1966
q9	6734	6413	6421	6413
q10	7012	2272	2318	2272
q11	465	267	269	267
q12	442	218	228	218
q13	17804	2996	2993	2993
q14	233	212	204	204
q15	522	458	455	455
q16	479	386	377	377
q17	969	526	565	526
q18	7225	6703	6528	6528
q19	1391	1082	992	992
q20	461	202	204	202
q21	3936	3219	3156	3156
q22	1133	962	1001	962
Total cold run time: 107580 ms
Total hot run time: 39845 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6720	6626	6636	6626
q2	338	239	237	237
q3	2958	2941	2951	2941
q4	2056	1868	1858	1858
q5	5765	5723	5717	5717
q6	217	131	128	128
q7	2284	1825	1879	1825
q8	3413	3580	3653	3580
q9	8934	8929	8872	8872
q10	3578	3542	3565	3542
q11	606	497	493	493
q12	839	584	609	584
q13	9058	3174	3209	3174
q14	304	277	267	267
q15	518	476	471	471
q16	524	458	467	458
q17	1856	1653	1590	1590
q18	8393	7786	7748	7748
q19	1731	1414	1537	1414
q20	2092	1898	1860	1860
q21	5400	5113	5114	5113
q22	1159	1072	1033	1033
Total cold run time: 68743 ms
Total hot run time: 59531 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-DS: Total hot run time: 192165 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2af40c85b0c1f51a35b3055e1e0a3fdbca42d997, data reload: false

query1	927	403	422	403
query2	6212	1864	1869	1864
query3	8683	203	202	202
query4	33743	24018	23776	23776
query5	4289	469	454	454
query6	300	190	185	185
query7	4225	317	316	316
query8	300	236	233	233
query9	9554	2586	2582	2582
query10	484	261	258	258
query11	17884	15546	15218	15218
query12	165	101	100	100
query13	1553	429	412	412
query14	9862	6873	6711	6711
query15	254	173	179	173
query16	8059	482	493	482
query17	1607	588	587	587
query18	2141	324	332	324
query19	249	162	162	162
query20	122	122	115	115
query21	203	109	108	108
query22	4691	4444	4345	4345
query23	34804	34145	34255	34145
query24	11311	2910	2977	2910
query25	663	431	435	431
query26	1481	177	173	173
query27	2925	358	368	358
query28	7780	2178	2181	2178
query29	881	456	469	456
query30	258	187	160	160
query31	1052	811	797	797
query32	98	54	54	54
query33	786	289	300	289
query34	939	518	520	518
query35	911	758	701	701
query36	1092	953	967	953
query37	137	67	67	67
query38	4114	4001	3923	3923
query39	1716	1475	1490	1475
query40	203	101	95	95
query41	50	48	49	48
query42	115	109	99	99
query43	525	480	481	480
query44	1330	801	799	799
query45	183	174	165	165
query46	1146	754	736	736
query47	2041	1897	1930	1897
query48	462	375	382	375
query49	920	391	390	390
query50	853	438	423	423
query51	7451	7100	7221	7100
query52	100	88	88	88
query53	255	186	187	186
query54	1080	459	475	459
query55	80	77	84	77
query56	267	248	257	248
query57	1309	1205	1199	1199
query58	231	202	219	202
query59	3178	3028	3070	3028
query60	293	263	251	251
query61	106	113	113	113
query62	909	708	709	708
query63	211	191	199	191
query64	4067	666	632	632
query65	3361	3313	3333	3313
query66	797	316	298	298
query67	16336	15578	15540	15540
query68	4634	583	606	583
query69	434	260	260	260
query70	1151	1081	1112	1081
query71	353	250	255	250
query72	6318	4101	4179	4101
query73	766	346	347	346
query74	10351	9362	8891	8891
query75	3358	2601	2672	2601
query76	2776	1078	1085	1078
query77	408	270	272	270
query78	10476	9559	9535	9535
query79	2317	598	613	598
query80	1120	423	418	418
query81	562	223	223	223
query82	615	90	85	85
query83	228	142	136	136
query84	238	81	85	81
query85	1345	308	286	286
query86	425	300	304	300
query87	4392	4307	4279	4279
query88	3832	2399	2359	2359
query89	413	295	289	289
query90	2005	186	187	186
query91	177	150	147	147
query92	65	51	48	48
query93	2439	562	557	557
query94	857	292	288	288
query95	356	258	260	258
query96	604	284	284	284
query97	3331	3120	3188	3120
query98	217	209	193	193
query99	1662	1303	1296	1296
Total cold run time: 302539 ms
Total hot run time: 192165 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

ClickBench: Total hot run time: 30.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2af40c85b0c1f51a35b3055e1e0a3fdbca42d997, data reload: false

query1	0.04	0.02	0.03
query2	0.06	0.04	0.03
query3	0.23	0.06	0.06
query4	1.62	0.11	0.10
query5	0.51	0.53	0.52
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.61	0.52	0.53
query10	0.58	0.59	0.57
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	0.78	0.81	0.79
query15	0.84	0.82	0.82
query16	0.41	0.38	0.39
query17	1.00	1.00	0.98
query18	0.25	0.24	0.24
query19	1.99	1.84	1.83
query20	0.02	0.01	0.02
query21	15.38	0.59	0.57
query22	2.81	2.78	1.93
query23	16.96	0.99	0.95
query24	3.17	1.44	0.51
query25	0.30	0.18	0.09
query26	0.31	0.14	0.14
query27	0.05	0.04	0.04
query28	10.44	0.51	0.46
query29	12.59	3.25	3.25
query30	0.25	0.06	0.06
query31	2.83	0.39	0.39
query32	3.22	0.45	0.45
query33	3.00	3.02	3.03
query34	16.61	4.50	4.44
query35	4.56	4.59	4.60
query36	0.69	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 104.7 s
Total hot run time: 30.11 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 19, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 42.08% (11239/26711)
Line Coverage 32.60% (96200/295118)
Region Coverage 30.53% (55237/180898)
Branch Coverage 26.85% (27336/101792)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 19, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.09% (19713/26252)
Line Coverage 68.35% (201181/294339)
Region Coverage 66.46% (120411/181170)
Branch Coverage 59.88% (61192/102188)

dataroaring
dataroaring approved these changes Aug 20, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor Author

github-actions bot commented Aug 20, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 20, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Aug 20, 2025

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 05aa670 into branch-3.0 Aug 20, 2025
22 of 26 checks passed
@github-actions github-actions bot deleted the auto-pick-54926-branch-3.0 branch August 20, 2025 03:15
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hello-stephen @doris-robot @dataroaring