Skip to content

[fix](hdfs)Fix be coredump in HDFS reader during profile collection. #56806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Oct 14, 2025
Merged

[fix](hdfs)Fix be coredump in HDFS reader during profile collection. #56806

morningman merged 1 commit into from
Oct 14, 2025

Conversation

@hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Oct 10, 2025

What problem does this PR solve?

Related PR: #54926
Issue Number: close #55747

Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read failed, but HDFS reader still held such a dangling pointer, which led to issue #55747

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

hubgeter commented Oct 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

TPC-DS: Total hot run time: 190230 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c23464e1782cabe74ae14124319333b947bb49e5, data reload: false

query1	1048	433	403	403
query2	6580	1714	1681	1681
query3	6773	222	222	222
query4	25963	23974	23018	23018
query5	5201	643	465	465
query6	365	242	240	240
query7	4658	496	295	295
query8	305	251	255	251
query9	8702	2564	2544	2544
query10	519	339	291	291
query11	15520	15646	15524	15524
query12	196	129	125	125
query13	2337	561	441	441
query14	11566	9495	9507	9495
query15	250	213	175	175
query16	8283	738	542	542
query17	1838	804	675	675
query18	2334	443	365	365
query19	239	209	185	185
query20	141	145	134	134
query21	222	139	127	127
query22	4630	4646	4566	4566
query23	34908	33901	33151	33151
query24	8395	2350	2367	2350
query25	555	497	441	441
query26	1231	265	153	153
query27	2729	483	364	364
query28	4382	2192	2167	2167
query29	757	592	486	486
query30	291	224	196	196
query31	905	810	758	758
query32	80	70	71	70
query33	581	371	325	325
query34	794	861	520	520
query35	767	825	762	762
query36	996	999	936	936
query37	126	112	89	89
query38	3509	3596	3536	3536
query39	1488	1403	1409	1403
query40	222	128	116	116
query41	60	60	62	60
query42	123	116	108	108
query43	480	487	470	470
query44	1303	842	811	811
query45	186	178	172	172
query46	863	979	627	627
query47	1763	1836	1744	1744
query48	381	414	317	317
query49	772	489	419	419
query50	638	687	405	405
query51	3914	3979	4022	3979
query52	119	107	97	97
query53	234	268	195	195
query54	593	616	521	521
query55	85	84	87	84
query56	322	333	311	311
query57	1200	1219	1116	1116
query58	290	280	268	268
query59	2520	2658	2506	2506
query60	350	344	332	332
query61	159	159	154	154
query62	777	733	666	666
query63	231	197	192	192
query64	4405	1168	894	894
query65	4040	3955	3935	3935
query66	1083	457	346	346
query67	15408	15461	15251	15251
query68	7832	934	609	609
query69	509	344	303	303
query70	1354	1345	1281	1281
query71	455	340	332	332
query72	6137	4870	4851	4851
query73	617	586	364	364
query74	8926	9059	8969	8969
query75	3780	3329	2919	2919
query76	3498	1152	742	742
query77	803	415	321	321
query78	9569	9880	8911	8911
query79	2821	816	585	585
query80	679	565	487	487
query81	509	260	234	234
query82	452	166	134	134
query83	310	285	256	256
query84	309	120	91	91
query85	893	550	425	425
query86	386	325	302	302
query87	3772	3750	3677	3677
query88	3727	2205	2233	2205
query89	382	322	290	290
query90	2013	223	217	217
query91	163	172	148	148
query92	89	69	68	68
query93	2301	971	634	634
query94	712	454	342	342
query95	403	319	321	319
query96	481	588	277	277
query97	2947	2949	2867	2867
query98	249	214	214	214
query99	1444	1401	1289	1289
Total cold run time: 281207 ms
Total hot run time: 190230 ms

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

ClickBench: Total hot run time: 30.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c23464e1782cabe74ae14124319333b947bb49e5, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.07
query3	0.25	0.08	0.08
query4	1.61	0.12	0.12
query5	0.28	0.27	0.24
query6	1.20	0.65	0.63
query7	0.04	0.03	0.03
query8	0.06	0.04	0.04
query9	0.64	0.52	0.52
query10	0.58	0.59	0.58
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.61	0.61
query14	1.02	1.05	1.04
query15	0.87	0.87	0.87
query16	0.43	0.40	0.41
query17	1.07	1.06	1.06
query18	0.21	0.20	0.20
query19	1.97	1.89	1.84
query20	0.02	0.01	0.02
query21	15.43	0.92	0.58
query22	0.76	1.22	0.73
query23	14.83	1.39	0.69
query24	7.53	1.86	0.46
query25	0.56	0.28	0.13
query26	0.59	0.15	0.13
query27	0.07	0.05	0.05
query28	9.71	1.36	0.95
query29	12.55	3.98	3.37
query30	0.29	0.15	0.12
query31	2.83	0.60	0.39
query32	3.25	0.54	0.47
query33	3.11	3.14	3.11
query34	15.99	5.55	4.87
query35	4.92	4.90	4.93
query36	0.70	0.53	0.50
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.16	0.14
query41	0.09	0.04	0.03
query42	0.03	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 105 s
Total hot run time: 30.41 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 10, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.53% (17730/33755)
Line Coverage 37.69% (160951/427003)
Region Coverage 32.19% (122979/382045)
Branch Coverage 33.57% (53913/160576)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 10, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 10, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 10, 2025

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 10, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.09% (23511/33073)
Line Coverage 57.53% (245409/426575)
Region Coverage 52.54% (203261/386873)
Branch Coverage 54.43% (87835/161387)

dataroaring
dataroaring approved these changes Oct 13, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 032ebae into apache:master Oct 14, 2025
33 of 34 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 14, 2025
@hubgeter
[fix](hdfs)Fix be coredump in HDFS reader during profile collection. (#…
3f531bc 
…56806)

### What problem does this PR solve?

Related PR: #54926
Issue Number: close #55747

Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read
failed, but HDFS reader still held such a dangling pointer, which led to
issue #55747
@github-actions github-actions bot mentioned this pull request Oct 14, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Oct 14, 2025
@hubgeter
[fix](hdfs)Fix be coredump in HDFS reader during profile collection. (#…
33fc0ee 
…56806)

### What problem does this PR solve?

Related PR: #54926
Issue Number: close #55747

Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read
failed, but HDFS reader still held such a dangling pointer, which led to
issue #55747
@github-actions github-actions bot mentioned this pull request Oct 14, 2025
Merged
yiguolei pushed a commit that referenced this pull request Oct 16, 2025
@github-actions @hubgeter
branch-4.0: [fix](hdfs)Fix be coredump in HDFS reader during profile …
44a73f0 
…collection. #56806 (#56951)

Cherry-picked from #56806

Co-authored-by: daidai <changyuwei@selectdb.com>
morningman pushed a commit that referenced this pull request Oct 16, 2025
@hubgeter @morningman
[fix](hdfs)Fix be coredump in HDFS reader during profile collection. (#…
7e4c05e 
…56806)

### What problem does this PR solve?

Related PR: #54926
Issue Number: close #55747

Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read
failed, but HDFS reader still held such a dangling pointer, which led to
issue #55747
morningman pushed a commit that referenced this pull request Oct 16, 2025
@github-actions @hubgeter
branch-3.1: [fix](hdfs)Fix be coredump in HDFS reader during profile …
8b8425c 
…collection. #56806 (#56950)

Cherry-picked from #56806

Co-authored-by: daidai <changyuwei@selectdb.com>
@yiguolei yiguolei mentioned this pull request Nov 5, 2025
Closed
Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Nov 12, 2025
@selectdb-robot @hubgeter
branch-selectdb-doris-3.0: [fix](hdfs)Fix be coredump in HDFS reader …
59ca790 
…during profile collection. apache#56806 (apache#5192)

cherry-picks from apache#56806
Related to apache#5076

Co-authored-by: daidai <changyuwei@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.x dev/3.1.2-merged dev/4.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] be crash when reading hive table from different format with same table

6 participants

@hubgeter @hello-stephen @doris-robot @morningman @dataroaring @yiguolei