Skip to content

[fix](hdfs) remove cached file handle when read fails #54926

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 6 commits into from
Aug 19, 2025
Merged

[fix](hdfs) remove cached file handle when read fails #54926

dataroaring merged 6 commits into from
Aug 19, 2025

Conversation

@dataroaring
Copy link
Contributor

@dataroaring dataroaring commented Aug 18, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@dataroaring dataroaring requested a review from gavinchou as a code owner August 18, 2025 06:16
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring
Copy link
Contributor Author

dataroaring commented Aug 18, 2025

run buildall

gavinchou
gavinchou reviewed Aug 18, 2025
View reviewed changes
Yongqiang YANG added 2 commits August 18, 2025 14:22
@dataroaring
Copy link
Contributor Author

dataroaring commented Aug 18, 2025

run buildall

@dataroaring
Copy link
Contributor Author

dataroaring commented Aug 18, 2025

run buildall

@dataroaring
Copy link
Contributor Author

dataroaring commented Aug 18, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 18, 2025

TPC-H: Total hot run time: 33715 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f1850b32ce8961d03c51947662dbe6ab4c3c5fab, data reload: false

------ Round 1 ----------------------------------
q1	17580	5271	5124	5124
q2	1923	277	173	173
q3	10333	1249	690	690
q4	10218	1017	514	514
q5	7520	2379	2300	2300
q6	174	158	131	131
q7	895	736	613	613
q8	9292	1309	1090	1090
q9	6825	5120	5098	5098
q10	6881	2349	1969	1969
q11	471	281	266	266
q12	348	344	224	224
q13	17772	3606	3041	3041
q14	226	229	221	221
q15	550	480	478	478
q16	416	425	378	378
q17	593	844	341	341
q18	7533	6962	7065	6962
q19	1073	954	563	563
q20	353	330	223	223
q21	4110	3244	2331	2331
q22	1095	1030	985	985
Total cold run time: 106181 ms
Total hot run time: 33715 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5182	5145	5149	5145
q2	241	319	219	219
q3	2175	2648	2336	2336
q4	1355	1802	1381	1381
q5	4173	4367	4430	4367
q6	223	173	131	131
q7	1999	1908	1799	1799
q8	2696	2649	2539	2539
q9	7324	7175	7397	7175
q10	3058	3282	2906	2906
q11	616	513	501	501
q12	667	782	628	628
q13	3459	3899	3406	3406
q14	288	296	282	282
q15	507	469	454	454
q16	459	498	470	470
q17	1188	1574	1349	1349
q18	7987	7475	7626	7475
q19	897	851	852	851
q20	2088	2073	1907	1907
q21	4904	4481	4291	4291
q22	1069	1035	988	988
Total cold run time: 52555 ms
Total hot run time: 50600 ms

@doris-robot
Copy link

doris-robot commented Aug 18, 2025

TPC-DS: Total hot run time: 184472 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f1850b32ce8961d03c51947662dbe6ab4c3c5fab, data reload: false

query1	974	386	400	386
query2	6532	1724	1762	1724
query3	6749	218	211	211
query4	26922	23517	23191	23191
query5	4313	612	533	533
query6	293	198	193	193
query7	4615	496	282	282
query8	277	220	216	216
query9	8606	2808	2832	2808
query10	474	340	281	281
query11	15562	14981	14875	14875
query12	152	116	111	111
query13	1649	575	430	430
query14	8487	5683	5615	5615
query15	204	183	162	162
query16	7162	611	499	499
query17	1103	708	605	605
query18	1995	415	327	327
query19	208	188	181	181
query20	130	123	118	118
query21	215	130	112	112
query22	4107	4159	4055	4055
query23	34061	33098	33778	33098
query24	8229	2351	2373	2351
query25	560	482	412	412
query26	1227	281	172	172
query27	2734	519	364	364
query28	4379	2302	2220	2220
query29	758	567	449	449
query30	277	220	186	186
query31	856	777	715	715
query32	105	71	77	71
query33	546	361	346	346
query34	786	819	501	501
query35	812	818	749	749
query36	969	1013	911	911
query37	115	112	84	84
query38	4128	4079	3955	3955
query39	1479	1385	1433	1385
query40	216	124	112	112
query41	58	64	55	55
query42	118	110	117	110
query43	494	497	471	471
query44	1353	837	841	837
query45	175	170	172	170
query46	867	995	642	642
query47	1761	1797	1728	1728
query48	390	416	324	324
query49	738	517	391	391
query50	651	676	403	403
query51	4054	4147	4089	4089
query52	111	110	101	101
query53	253	264	189	189
query54	575	592	522	522
query55	89	88	88	88
query56	309	306	303	303
query57	1203	1199	1128	1128
query58	275	271	263	263
query59	2651	2814	2613	2613
query60	342	340	319	319
query61	124	121	124	121
query62	812	737	661	661
query63	231	190	191	190
query64	4312	1013	692	692
query65	4286	4212	4125	4125
query66	1160	422	324	324
query67	15736	15225	15061	15061
query68	8161	919	573	573
query69	493	391	281	281
query70	1202	1143	1129	1129
query71	484	325	310	310
query72	5678	4713	4744	4713
query73	674	597	351	351
query74	8851	9014	8798	8798
query75	3910	3060	2596	2596
query76	3643	1141	742	742
query77	793	389	322	322
query78	9432	9657	8865	8865
query79	2776	818	588	588
query80	659	526	463	463
query81	496	263	210	210
query82	470	132	115	115
query83	241	249	229	229
query84	269	96	84	84
query85	802	373	348	348
query86	389	327	296	296
query87	4184	4218	4189	4189
query88	3814	2276	2157	2157
query89	381	309	277	277
query90	1877	221	213	213
query91	133	143	117	117
query92	87	70	68	68
query93	2342	1000	640	640
query94	672	400	290	290
query95	392	304	299	299
query96	481	567	271	271
query97	2703	2666	2550	2550
query98	245	211	228	211
query99	1340	1375	1261	1261
Total cold run time: 274168 ms
Total hot run time: 184472 ms

@doris-robot
Copy link

doris-robot commented Aug 18, 2025

ClickBench: Total hot run time: 32.36 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f1850b32ce8961d03c51947662dbe6ab4c3c5fab, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.08	0.07
query4	1.63	0.10	0.11
query5	0.44	0.41	0.41
query6	1.18	0.63	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.51
query10	0.58	0.58	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.63	0.60	0.63
query14	0.80	0.82	0.82
query15	0.87	0.86	0.86
query16	0.39	0.41	0.40
query17	1.05	1.05	1.04
query18	0.22	0.20	0.19
query19	1.96	1.84	1.78
query20	0.01	0.01	0.01
query21	15.42	0.97	0.61
query22	0.77	1.04	0.80
query23	14.88	1.36	0.62
query24	6.50	1.33	0.65
query25	0.50	0.17	0.11
query26	0.61	0.16	0.13
query27	0.07	0.05	0.05
query28	9.73	0.90	0.42
query29	12.59	3.88	3.25
query30	3.09	2.95	2.91
query31	2.81	0.57	0.37
query32	3.24	0.54	0.49
query33	3.05	3.12	3.10
query34	16.10	5.44	4.80
query35	4.94	4.89	4.88
query36	0.68	0.51	0.50
query37	0.09	0.07	0.08
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.15	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.57 s
Total hot run time: 32.36 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 18, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 59.64% (16773/28126)
Line Coverage 48.53% (152598/314437)
Region Coverage 37.46% (115969/309599)
Branch Coverage 40.39% (51225/126835)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 18, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.07% (22616/27557)
Line Coverage 74.69% (234716/314268)
Region Coverage 61.63% (194182/315062)
Branch Coverage 65.93% (84259/127792)

@github-actions
Copy link
Contributor

github-actions bot commented Aug 19, 2025

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 19, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Aug 19, 2025

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 4774563 into apache:master Aug 19, 2025
25 of 28 checks passed
@dataroaring dataroaring mentioned this pull request Aug 19, 2025
Merged
16 tasks
github-actions bot pushed a commit that referenced this pull request Aug 19, 2025
@dataroaring
[fix](hdfs) remove cached file handle when read fails (#54926)
2af40c8
@github-actions github-actions bot mentioned this pull request Aug 19, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Aug 19, 2025
@dataroaring
[fix](hdfs) remove cached file handle when read fails (#54926)
226b2f9
@github-actions github-actions bot mentioned this pull request Aug 19, 2025
Merged
morrySnow pushed a commit that referenced this pull request Aug 20, 2025
@github-actions @dataroaring
branch-3.1: [fix](hdfs) remove cached file handle when read fails #54926
56e2269 
 (#54992)

Cherry-picked from #54926

Co-authored-by: Yongqiang YANG <yangyongqiang@selectdb.com>
dataroaring added a commit that referenced this pull request Aug 20, 2025
@github-actions @dataroaring
branch-3.0: [fix](hdfs) remove cached file handle when read fails #54926
05aa670 
 (#54991)

Cherry-picked from #54926

Co-authored-by: Yongqiang YANG <yangyongqiang@selectdb.com>
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Open
yiguolei pushed a commit that referenced this pull request Sep 12, 2025
@dataroaring
[fix](hdfs) remove cached file handle when read fails (#54988)
ba47a4a 
### What problem does this PR solve?

pick #54926

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

Co-authored-by: Yongqiang YANG <yangyogqiang@selectdb.com>
@hubgeter hubgeter mentioned this pull request Oct 10, 2025
Merged
16 tasks
morningman pushed a commit that referenced this pull request Oct 14, 2025
@hubgeter
[fix](hdfs)Fix be coredump in HDFS reader during profile collection. (#…
032ebae 
…56806)

### What problem does this PR solve?

Related PR: #54926
Issue Number: close #55747

Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read
failed, but HDFS reader still held such a dangling pointer, which led to
issue #55747
github-actions bot pushed a commit that referenced this pull request Oct 14, 2025
@hubgeter
[fix](hdfs)Fix be coredump in HDFS reader during profile collection. (#…
3f531bc 
…56806)

### What problem does this PR solve?

Related PR: #54926
Issue Number: close #55747

Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read
failed, but HDFS reader still held such a dangling pointer, which led to
issue #55747
github-actions bot pushed a commit that referenced this pull request Oct 14, 2025
@hubgeter
[fix](hdfs)Fix be coredump in HDFS reader during profile collection. (#…
33fc0ee 
…56806)

### What problem does this PR solve?

Related PR: #54926
Issue Number: close #55747

Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read
failed, but HDFS reader still held such a dangling pointer, which led to
issue #55747
morningman pushed a commit that referenced this pull request Oct 16, 2025
@hubgeter @morningman
[fix](hdfs)Fix be coredump in HDFS reader during profile collection. (#…
7e4c05e 
…56806)

### What problem does this PR solve?

Related PR: #54926
Issue Number: close #55747

Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read
failed, but HDFS reader still held such a dangling pointer, which led to
issue #55747
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

+1 more reviewer

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.12-merged dev/3.0.8-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@dataroaring @hello-stephen @doris-robot @gavinchou @Yukang-Lian @yiguolei @morrySnow