Skip to content

branch-3.1: [fix](hdfs)Fix be coredump in HDFS reader during profile collection. #56806 #56950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Oct 16, 2025
Merged

branch-3.1: [fix](hdfs)Fix be coredump in HDFS reader during profile collection. #56806 #56950

morningman merged 1 commit into from
Oct 16, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Oct 14, 2025

Cherry-picked from #56806

@github-actions github-actions bot requested a review from morrySnow as a code owner October 14, 2025 07:56
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Oct 14, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 14, 2025

TPC-H: Total hot run time: 32634 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f531bce511a25da67414ca013f7ea01126f45fe, data reload: false

------ Round 1 ----------------------------------
q1	17677	5405	5343	5343
q2	2015	423	295	295
q3	12094	1250	749	749
q4	10255	911	460	460
q5	8926	2368	2154	2154
q6	183	162	132	132
q7	887	768	627	627
q8	9346	1455	1123	1123
q9	5259	4989	4956	4956
q10	6781	2281	1847	1847
q11	486	284	267	267
q12	339	366	210	210
q13	17780	3586	3017	3017
q14	241	237	221	221
q15	526	460	466	460
q16	419	441	365	365
q17	625	858	374	374
q18	6888	6333	6281	6281
q19	1206	945	542	542
q20	326	345	217	217
q21	2873	2191	1993	1993
q22	1053	1026	1001	1001
Total cold run time: 106185 ms
Total hot run time: 32634 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5550	5520	5545	5520
q2	231	330	236	236
q3	2253	2637	2330	2330
q4	1315	1784	1368	1368
q5	4439	4931	4984	4931
q6	170	158	125	125
q7	2075	1966	1804	1804
q8	2658	2827	2723	2723
q9	7274	7281	7219	7219
q10	3051	3341	2704	2704
q11	559	496	483	483
q12	728	763	615	615
q13	3378	3748	3158	3158
q14	274	302	271	271
q15	516	480	478	478
q16	447	498	431	431
q17	1218	1715	1268	1268
q18	7656	7442	7388	7388
q19	818	1155	1043	1043
q20	2018	2037	1921	1921
q21	5292	4950	4599	4599
q22	1104	1070	1013	1013
Total cold run time: 53024 ms
Total hot run time: 51628 ms

@doris-robot
Copy link

doris-robot commented Oct 14, 2025

TPC-DS: Total hot run time: 192599 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f531bce511a25da67414ca013f7ea01126f45fe, data reload: false

query1	932	410	395	395
query2	6123	1926	1860	1860
query3	8681	204	205	204
query4	33305	24108	23547	23547
query5	3608	615	464	464
query6	293	198	192	192
query7	4205	495	326	326
query8	301	231	236	231
query9	9533	2598	2598	2598
query10	446	323	288	288
query11	18113	15590	15204	15204
query12	160	110	112	110
query13	1551	550	427	427
query14	9123	7597	6706	6706
query15	248	187	182	182
query16	8056	646	515	515
query17	1577	776	589	589
query18	2128	424	342	342
query19	242	193	169	169
query20	126	133	121	121
query21	222	137	116	116
query22	4661	4712	4581	4581
query23	35010	33956	34324	33956
query24	7308	2694	2692	2692
query25	577	506	441	441
query26	1163	291	179	179
query27	2027	484	360	360
query28	5587	2237	2228	2228
query29	785	613	470	470
query30	234	195	169	169
query31	1031	912	837	837
query32	94	63	62	62
query33	529	371	345	345
query34	750	853	522	522
query35	810	810	717	717
query36	1041	1029	969	969
query37	117	89	68	68
query38	3998	4042	3965	3965
query39	1526	1471	1514	1471
query40	218	128	109	109
query41	49	50	50	50
query42	125	107	105	105
query43	512	524	493	493
query44	1376	839	827	827
query45	181	173	178	173
query46	893	1063	695	695
query47	2026	2028	1932	1932
query48	409	437	364	364
query49	789	500	411	411
query50	702	694	422	422
query51	7344	7420	7207	7207
query52	105	103	91	91
query53	235	259	193	193
query54	548	555	484	484
query55	79	79	82	79
query56	277	298	261	261
query57	1265	1250	1246	1246
query58	237	216	224	216
query59	3151	3152	3134	3134
query60	296	291	268	268
query61	133	123	116	116
query62	809	749	716	716
query63	238	203	208	203
query64	4498	1001	654	654
query65	3383	3359	3315	3315
query66	1024	455	306	306
query67	16157	15777	15599	15599
query68	7725	826	546	546
query69	485	314	270	270
query70	1136	1099	1114	1099
query71	360	297	273	273
query72	5703	3807	3777	3777
query73	628	750	356	356
query74	10484	9238	9145	9145
query75	3212	3118	2632	2632
query76	3118	1155	769	769
query77	480	366	277	277
query78	10362	10518	9604	9604
query79	3473	896	602	602
query80	692	539	453	453
query81	506	256	222	222
query82	590	116	91	91
query83	165	167	146	146
query84	255	104	76	76
query85	781	351	295	295
query86	395	317	311	311
query87	4367	4330	4280	4280
query88	5355	2435	2430	2430
query89	420	343	296	296
query90	1824	192	195	192
query91	140	142	111	111
query92	63	55	51	51
query93	2152	914	553	553
query94	699	407	310	310
query95	363	300	274	274
query96	500	618	284	284
query97	3155	3247	3103	3103
query98	237	209	202	202
query99	1469	1413	1282	1282
Total cold run time: 294574 ms
Total hot run time: 192599 ms

@doris-robot
Copy link

doris-robot commented Oct 14, 2025

ClickBench: Total hot run time: 28.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f531bce511a25da67414ca013f7ea01126f45fe, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.02	0.03
query3	0.23	0.07	0.06
query4	1.63	0.11	0.10
query5	0.53	0.53	0.51
query6	1.13	0.72	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.57	0.50	0.51
query10	0.55	0.55	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.59
query14	0.76	0.79	0.79
query15	0.84	0.85	0.83
query16	0.40	0.37	0.37
query17	1.07	1.05	1.08
query18	0.24	0.22	0.23
query19	1.91	1.84	1.92
query20	0.01	0.01	0.01
query21	15.38	0.94	0.55
query22	0.73	0.82	0.68
query23	15.04	1.44	0.55
query24	3.64	1.12	0.27
query25	0.19	0.09	0.05
query26	0.28	0.16	0.13
query27	0.05	0.06	0.05
query28	12.65	1.07	0.44
query29	12.64	3.97	3.26
query30	0.26	0.09	0.06
query31	2.82	0.60	0.39
query32	3.22	0.53	0.46
query33	2.99	3.01	3.04
query34	16.52	5.17	4.62
query35	4.53	4.54	4.63
query36	0.64	0.51	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.05 s
Total hot run time: 28.31 s

@hubgeter @morningman
[fix](hdfs)Fix be coredump in HDFS reader during profile collection. (#…
7e4c05e 
…56806)

### What problem does this PR solve?

Related PR: #54926
Issue Number: close #55747

Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read
failed, but HDFS reader still held such a dangling pointer, which led to
issue #55747
@morningman morningman force-pushed the auto-pick-56806-branch-3.1 branch from 3f531bc to 7e4c05e Compare October 16, 2025 02:55
@morningman
Copy link
Contributor

morningman commented Oct 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-H: Total hot run time: 32722 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7e4c05ea960cbd35ca9fe65e039546e39a716108, data reload: false

------ Round 1 ----------------------------------
q1	17602	5504	5443	5443
q2	2026	402	283	283
q3	11795	1260	741	741
q4	10557	874	453	453
q5	9454	2359	2148	2148
q6	181	164	134	134
q7	889	752	619	619
q8	9332	1358	1108	1108
q9	5177	4911	4943	4911
q10	6782	2275	1837	1837
q11	468	278	263	263
q12	334	361	211	211
q13	17774	3622	3032	3032
q14	223	237	223	223
q15	534	473	474	473
q16	425	441	380	380
q17	616	861	366	366
q18	6814	6539	6366	6366
q19	1222	950	535	535
q20	345	350	206	206
q21	2859	2227	1988	1988
q22	1028	1041	1002	1002
Total cold run time: 106437 ms
Total hot run time: 32722 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5518	5481	5516	5481
q2	225	333	242	242
q3	2286	2674	2358	2358
q4	1383	1796	1403	1403
q5	4417	5036	4936	4936
q6	167	165	131	131
q7	2064	1993	1844	1844
q8	2648	2844	2738	2738
q9	7271	7212	7164	7164
q10	3057	3322	2691	2691
q11	563	512	489	489
q12	689	773	608	608
q13	3369	3796	3131	3131
q14	281	307	268	268
q15	515	478	480	478
q16	450	488	443	443
q17	1265	1724	1254	1254
q18	7600	7435	7327	7327
q19	779	1059	1099	1059
q20	2050	2038	1900	1900
q21	5261	4834	4725	4725
q22	1062	1089	998	998
Total cold run time: 52920 ms
Total hot run time: 51668 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-DS: Total hot run time: 193633 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7e4c05ea960cbd35ca9fe65e039546e39a716108, data reload: false

query1	951	428	401	401
query2	6211	1992	1913	1913
query3	8680	206	209	206
query4	33908	23759	24221	23759
query5	5047	609	469	469
query6	327	205	191	191
query7	4207	491	324	324
query8	310	255	243	243
query9	9364	2621	2614	2614
query10	505	326	268	268
query11	18253	15392	15322	15322
query12	169	110	114	110
query13	1569	547	428	428
query14	9615	7449	7271	7271
query15	276	196	180	180
query16	8030	655	498	498
query17	1524	792	591	591
query18	2143	431	327	327
query19	245	194	173	173
query20	128	126	126	126
query21	209	132	112	112
query22	4578	4680	4469	4469
query23	35021	34463	33978	33978
query24	7406	2745	2729	2729
query25	525	474	414	414
query26	816	286	175	175
query27	2013	479	371	371
query28	5370	2248	2235	2235
query29	697	589	481	481
query30	252	197	163	163
query31	981	916	800	800
query32	95	61	58	58
query33	478	372	325	325
query34	734	856	535	535
query35	783	819	749	749
query36	1017	1056	958	958
query37	105	92	69	69
query38	4023	3989	4022	3989
query39	1572	1474	1466	1466
query40	216	120	106	106
query41	48	47	46	46
query42	119	101	103	101
query43	502	543	486	486
query44	1323	827	834	827
query45	190	181	172	172
query46	885	1053	679	679
query47	1974	1966	1952	1952
query48	411	434	348	348
query49	754	508	422	422
query50	707	690	434	434
query51	7350	7244	7252	7244
query52	107	103	92	92
query53	238	262	197	197
query54	547	566	472	472
query55	81	88	82	82
query56	270	273	269	269
query57	1256	1240	1226	1226
query58	242	234	222	222
query59	3108	3324	3125	3125
query60	292	285	284	284
query61	123	114	133	114
query62	802	758	691	691
query63	238	199	197	197
query64	3750	984	624	624
query65	3385	3355	3291	3291
query66	802	415	311	311
query67	16218	15744	15602	15602
query68	2490	840	610	610
query69	447	307	266	266
query70	1206	1129	1159	1129
query71	346	301	261	261
query72	5722	3845	3820	3820
query73	623	761	369	369
query74	10060	9160	9282	9160
query75	3188	3162	2692	2692
query76	2855	1194	787	787
query77	525	386	290	290
query78	10421	10487	9668	9668
query79	3019	890	619	619
query80	785	542	490	490
query81	485	261	222	222
query82	628	122	90	90
query83	158	165	147	147
query84	243	99	80	80
query85	778	347	293	293
query86	406	330	276	276
query87	4319	4367	4197	4197
query88	5118	2407	2393	2393
query89	407	336	300	300
query90	1889	194	195	194
query91	140	139	107	107
query92	63	58	52	52
query93	2725	904	561	561
query94	739	410	303	303
query95	353	277	269	269
query96	489	613	294	294
query97	3138	3277	3096	3096
query98	227	217	200	200
query99	1324	1427	1292	1292
Total cold run time: 289391 ms
Total hot run time: 193633 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

ClickBench: Total hot run time: 29.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7e4c05ea960cbd35ca9fe65e039546e39a716108, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.03	0.02
query3	0.23	0.07	0.06
query4	1.63	0.10	0.11
query5	0.52	0.50	0.50
query6	1.15	0.74	0.73
query7	0.03	0.01	0.02
query8	0.04	0.02	0.03
query9	0.58	0.50	0.50
query10	0.55	0.55	0.55
query11	0.14	0.10	0.10
query12	0.15	0.11	0.11
query13	0.61	0.61	0.59
query14	0.78	0.80	0.81
query15	0.84	0.84	0.82
query16	0.37	0.38	0.40
query17	1.07	1.01	1.08
query18	0.25	0.23	0.22
query19	1.87	1.81	1.83
query20	0.01	0.01	0.02
query21	15.41	0.97	0.60
query22	0.74	0.73	0.81
query23	15.03	1.39	0.57
query24	3.33	1.69	1.44
query25	0.24	0.12	0.19
query26	0.27	0.15	0.14
query27	0.06	0.08	0.04
query28	14.28	0.99	0.44
query29	12.58	3.96	3.23
query30	0.24	0.09	0.06
query31	2.82	0.60	0.38
query32	3.22	0.55	0.46
query33	3.01	3.05	3.04
query34	16.62	5.22	4.57
query35	4.62	4.60	4.61
query36	0.63	0.50	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.14	0.12
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.51 s
Total hot run time: 29.59 s

@morningman morningman merged commit 8b8425c into branch-3.1 Oct 16, 2025
22 of 25 checks passed
@github-actions github-actions bot deleted the auto-pick-56806-branch-3.1 branch October 16, 2025 15:29
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@hello-stephen @doris-robot @morningman @morrySnow @dataroaring @hubgeter