Skip to content

branch-3.1: [enhancement](be_metrics) update scan bytes metric in file_scanner. #53729 #54222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Aug 4, 2025
Merged

branch-3.1: [enhancement](be_metrics) update scan bytes metric in file_scanner. #53729 #54222

morrySnow merged 1 commit into from
Aug 4, 2025

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Aug 1, 2025 •
edited by morrySnow
Loading

Cherry-pick #53729

@kaka11chen kaka11chen requested a review from morrySnow as a code owner August 1, 2025 10:06
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 1, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen
Copy link
Contributor Author

kaka11chen commented Aug 1, 2025

run buildall

@kaka11chen kaka11chen force-pushed the cherry-pick-53729_branch-3.1 branch from 6a26ade to 2a4aa97 Compare August 1, 2025 11:05
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Aug 1, 2025

run buildall

@kaka11chen kaka11chen force-pushed the cherry-pick-53729_branch-3.1 branch from 2a4aa97 to 77af352 Compare August 1, 2025 11:58
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Aug 1, 2025

run buildall

@kaka11chen kaka11chen force-pushed the cherry-pick-53729_branch-3.1 branch from 77af352 to 16da847 Compare August 2, 2025 03:04
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Aug 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 2, 2025

TPC-H: Total hot run time: 32815 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 16da84730586410ce0e336299be81cfcca967a4b, data reload: false

------ Round 1 ----------------------------------
q1	17602	5516	5403	5403
q2	2079	283	162	162
q3	10583	1249	759	759
q4	10274	873	467	467
q5	8907	2394	2187	2187
q6	183	163	138	138
q7	935	750	626	626
q8	9316	1450	1266	1266
q9	5247	4906	4993	4906
q10	6758	2273	1818	1818
q11	477	276	267	267
q12	338	368	225	225
q13	17774	3592	3002	3002
q14	231	225	210	210
q15	521	481	461	461
q16	424	422	365	365
q17	630	880	382	382
q18	6911	6440	6334	6334
q19	1205	962	577	577
q20	339	351	211	211
q21	3165	2234	2065	2065
q22	1070	1042	984	984
Total cold run time: 104969 ms
Total hot run time: 32815 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5538	5535	5541	5535
q2	238	330	238	238
q3	2284	2634	2367	2367
q4	1414	1877	1411	1411
q5	4434	4971	5021	4971
q6	172	168	129	129
q7	2085	1958	1824	1824
q8	2652	2840	2721	2721
q9	7190	7288	7207	7207
q10	2989	3329	2816	2816
q11	569	504	483	483
q12	663	767	660	660
q13	3392	3827	3160	3160
q14	277	311	284	284
q15	512	473	469	469
q16	426	481	436	436
q17	1224	1740	1256	1256
q18	7710	7417	7484	7417
q19	838	1172	1092	1092
q20	2029	2057	1878	1878
q21	5272	5026	4692	4692
q22	1096	1039	1032	1032
Total cold run time: 53004 ms
Total hot run time: 52078 ms

@doris-robot
Copy link

doris-robot commented Aug 2, 2025

TPC-DS: Total hot run time: 196611 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 16da84730586410ce0e336299be81cfcca967a4b, data reload: false

query1	1296	967	922	922
query2	6230	1849	1838	1838
query3	10920	4385	4633	4385
query4	32813	23736	23358	23358
query5	3694	623	470	470
query6	272	197	180	180
query7	3991	494	323	323
query8	315	251	253	251
query9	9265	2611	2604	2604
query10	497	327	261	261
query11	17984	15769	15284	15284
query12	162	114	105	105
query13	1556	540	418	418
query14	9622	6591	7245	6591
query15	237	210	194	194
query16	7748	687	495	495
query17	1634	772	604	604
query18	2052	427	326	326
query19	229	194	179	179
query20	130	121	119	119
query21	206	142	113	113
query22	4696	4649	4764	4649
query23	35128	33886	34534	33886
query24	7267	2703	2728	2703
query25	506	484	428	428
query26	964	273	177	177
query27	2446	481	354	354
query28	5999	2229	2269	2229
query29	587	572	452	452
query30	240	189	171	171
query31	1030	919	858	858
query32	67	58	52	52
query33	494	363	303	303
query34	735	850	513	513
query35	791	812	729	729
query36	1015	1024	953	953
query37	104	90	66	66
query38	4035	4034	3987	3987
query39	1515	1484	1477	1477
query40	206	124	110	110
query41	48	53	47	47
query42	123	107	106	106
query43	509	520	489	489
query44	1375	870	839	839
query45	188	182	173	173
query46	895	1070	684	684
query47	2007	2013	1986	1986
query48	433	442	351	351
query49	750	493	400	400
query50	682	676	434	434
query51	7436	7371	7157	7157
query52	104	103	94	94
query53	230	253	185	185
query54	546	547	482	482
query55	83	87	81	81
query56	261	280	276	276
query57	1309	1285	1213	1213
query58	248	233	226	226
query59	2980	3160	3037	3037
query60	299	290	280	280
query61	119	121	114	114
query62	811	759	710	710
query63	231	199	194	194
query64	3503	1019	644	644
query65	3344	3268	3191	3191
query66	860	414	310	310
query67	16137	15872	15671	15671
query68	6447	820	536	536
query69	488	301	261	261
query70	1237	1091	1135	1091
query71	399	285	269	269
query72	5707	3758	3784	3758
query73	642	739	345	345
query74	9410	9381	8952	8952
query75	3204	3153	2694	2694
query76	3103	1181	743	743
query77	493	365	287	287
query78	10537	10513	9688	9688
query79	3771	910	596	596
query80	695	555	431	431
query81	500	270	224	224
query82	686	123	90	90
query83	168	158	147	147
query84	254	99	88	88
query85	807	370	299	299
query86	395	310	303	303
query87	4371	4308	4244	4244
query88	5089	2358	2367	2358
query89	414	332	297	297
query90	1825	185	186	185
query91	133	136	109	109
query92	69	53	51	51
query93	2621	886	546	546
query94	707	407	311	311
query95	344	274	269	269
query96	502	605	287	287
query97	3169	3341	3165	3165
query98	227	207	198	198
query99	1387	1384	1298	1298
Total cold run time: 293990 ms
Total hot run time: 196611 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 2, 2025

BE UT Coverage Report

Increment line coverage 18.40% (30/163) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.40% (12639/27838)
Line Coverage 36.27% (112694/310722)
Region Coverage 35.30% (58230/164943)
Branch Coverage 32.50% (31672/97452)

@doris-robot
Copy link

doris-robot commented Aug 2, 2025

ClickBench: Total hot run time: 28.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 16da84730586410ce0e336299be81cfcca967a4b, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.11	0.11
query5	0.53	0.51	0.51
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.57	0.51	0.50
query10	0.57	0.55	0.56
query11	0.14	0.10	0.11
query12	0.13	0.10	0.10
query13	0.61	0.60	0.59
query14	0.78	0.79	0.79
query15	0.84	0.82	0.82
query16	0.37	0.39	0.39
query17	0.99	1.01	1.06
query18	0.24	0.21	0.23
query19	1.90	1.90	1.87
query20	0.01	0.01	0.01
query21	15.38	0.90	0.59
query22	0.74	0.80	0.53
query23	15.23	1.36	0.56
query24	3.01	1.06	0.74
query25	0.14	0.17	0.12
query26	0.32	0.14	0.14
query27	0.05	0.04	0.05
query28	14.08	0.98	0.43
query29	12.56	3.93	3.27
query30	0.25	0.09	0.08
query31	2.82	0.59	0.38
query32	3.22	0.53	0.47
query33	2.97	2.97	3.01
query34	16.66	5.20	4.58
query35	4.56	4.56	4.59
query36	0.65	0.49	0.50
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.97 s
Total hot run time: 28.72 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 2, 2025

BE Regression && UT Coverage Report

Increment line coverage 82.61% (133/161) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.25% (20973/27506)
Line Coverage 69.63% (216201/310515)
Region Coverage 67.59% (129281/191281)
Branch Coverage 61.12% (67213/109976)

@morningman morningman changed the title [enhancement](be_metrics) update scan bytes metric in file_scanner. branch-3.1: [enhancement](be_metrics) update scan bytes metric in file_scanner. Aug 3, 2025
@morningman morningman force-pushed the cherry-pick-53729_branch-3.1 branch from 16da847 to 38f3d82 Compare August 3, 2025 22:08
@morningman
Copy link
Contributor

morningman commented Aug 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 3, 2025

TPC-H: Total hot run time: 32676 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 38f3d826d343009205c449387045895b20846794, data reload: false

------ Round 1 ----------------------------------
q1	17614	5469	5331	5331
q2	2065	282	166	166
q3	10434	1206	753	753
q4	10205	866	449	449
q5	8113	2350	2079	2079
q6	175	163	130	130
q7	898	734	621	621
q8	9313	1410	1152	1152
q9	5219	4954	4966	4954
q10	6800	2277	1830	1830
q11	481	278	262	262
q12	329	354	205	205
q13	18275	3554	3124	3124
q14	246	239	218	218
q15	519	475	470	470
q16	424	435	379	379
q17	621	868	395	395
q18	7071	6550	6487	6487
q19	1311	947	547	547
q20	308	337	202	202
q21	2950	2140	1932	1932
q22	1067	1003	990	990
Total cold run time: 104438 ms
Total hot run time: 32676 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5555	5453	5542	5453
q2	229	333	228	228
q3	2258	2556	2294	2294
q4	1352	1814	1345	1345
q5	4389	4805	4946	4805
q6	164	163	130	130
q7	2033	1953	1817	1817
q8	2617	2810	2686	2686
q9	7268	7185	7230	7185
q10	3001	3306	2830	2830
q11	561	494	482	482
q12	631	734	594	594
q13	3387	3776	3159	3159
q14	290	307	277	277
q15	514	483	468	468
q16	441	491	432	432
q17	1202	1724	1258	1258
q18	7507	7342	7291	7291
q19	775	1156	1090	1090
q20	1994	2037	1920	1920
q21	5249	4947	4575	4575
q22	1082	1055	985	985
Total cold run time: 52499 ms
Total hot run time: 51304 ms

@doris-robot
Copy link

doris-robot commented Aug 3, 2025

BE UT Coverage Report

Increment line coverage 18.40% (30/163) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.40% (12639/27838)
Line Coverage 36.27% (112696/310722)
Region Coverage 35.30% (58230/164943)
Branch Coverage 32.50% (31674/97452)

@doris-robot
Copy link

doris-robot commented Aug 3, 2025

TPC-DS: Total hot run time: 196143 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 38f3d826d343009205c449387045895b20846794, data reload: false

query1	1288	945	911	911
query2	6254	1869	1849	1849
query3	10874	4503	4559	4503
query4	33245	23708	23832	23708
query5	4162	614	446	446
query6	294	192	185	185
query7	3987	496	326	326
query8	300	249	236	236
query9	9339	2588	2569	2569
query10	485	326	252	252
query11	17983	15340	15238	15238
query12	163	107	105	105
query13	1566	545	427	427
query14	10047	6646	6713	6646
query15	219	197	178	178
query16	7854	652	486	486
query17	1564	750	621	621
query18	2066	426	320	320
query19	211	191	177	177
query20	137	127	119	119
query21	206	131	110	110
query22	4529	4597	4460	4460
query23	35104	34296	33733	33733
query24	7210	2682	2697	2682
query25	511	484	432	432
query26	837	295	172	172
query27	2051	479	346	346
query28	5300	2198	2187	2187
query29	605	613	440	440
query30	229	196	165	165
query31	1013	894	857	857
query32	66	60	58	58
query33	500	356	308	308
query34	748	848	530	530
query35	791	828	749	749
query36	1041	1050	951	951
query37	110	90	65	65
query38	3965	3994	3980	3980
query39	1531	1475	1475	1475
query40	203	118	104	104
query41	49	47	47	47
query42	120	112	100	100
query43	498	511	470	470
query44	1325	823	825	823
query45	180	176	174	174
query46	894	1068	679	679
query47	1966	1973	1921	1921
query48	404	425	350	350
query49	739	495	401	401
query50	680	715	435	435
query51	7399	7419	7150	7150
query52	99	98	92	92
query53	228	260	187	187
query54	576	540	480	480
query55	81	80	83	80
query56	289	267	256	256
query57	1272	1276	1224	1224
query58	247	228	224	224
query59	3037	3066	3054	3054
query60	297	272	260	260
query61	110	109	109	109
query62	813	785	711	711
query63	234	199	190	190
query64	3136	1004	633	633
query65	3374	3329	3291	3291
query66	828	425	316	316
query67	16501	15731	15432	15432
query68	6487	804	534	534
query69	501	303	282	282
query70	1183	1081	1074	1074
query71	394	291	261	261
query72	5750	3828	3824	3824
query73	630	734	354	354
query74	10374	9425	8916	8916
query75	3248	3141	2675	2675
query76	3103	1188	760	760
query77	486	344	274	274
query78	10257	10316	9575	9575
query79	3741	925	594	594
query80	797	519	436	436
query81	505	260	217	217
query82	1079	124	91	91
query83	168	164	155	155
query84	250	116	84	84
query85	774	354	290	290
query86	407	295	311	295
query87	4318	4434	4146	4146
query88	5036	2395	2377	2377
query89	426	341	299	299
query90	1737	192	189	189
query91	165	133	112	112
query92	68	56	50	50
query93	3003	927	542	542
query94	699	414	305	305
query95	350	291	280	280
query96	491	610	281	281
query97	3190	3268	3174	3174
query98	227	211	209	209
query99	1520	1408	1293	1293
Total cold run time: 295328 ms
Total hot run time: 196143 ms

@doris-robot
Copy link

doris-robot commented Aug 3, 2025

ClickBench: Total hot run time: 28.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 38f3d826d343009205c449387045895b20846794, data reload: false

query1	0.04	0.03	0.02
query2	0.08	0.02	0.03
query3	0.24	0.06	0.06
query4	1.63	0.10	0.10
query5	0.50	0.50	0.53
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.56	0.51	0.50
query10	0.56	0.56	0.54
query11	0.14	0.10	0.10
query12	0.13	0.11	0.12
query13	0.61	0.60	0.59
query14	0.78	0.80	0.78
query15	0.85	0.84	0.84
query16	0.38	0.39	0.37
query17	1.06	1.06	1.00
query18	0.24	0.22	0.22
query19	1.90	1.82	1.76
query20	0.01	0.01	0.01
query21	15.41	0.96	0.60
query22	0.74	1.09	0.58
query23	14.90	1.41	0.51
query24	3.01	1.27	0.64
query25	0.20	0.16	0.15
query26	0.25	0.14	0.14
query27	0.08	0.06	0.07
query28	13.61	1.05	0.42
query29	12.54	3.93	3.28
query30	0.25	0.10	0.07
query31	2.82	0.59	0.37
query32	3.22	0.54	0.46
query33	2.96	3.02	3.03
query34	16.83	5.23	4.53
query35	4.57	4.57	4.55
query36	0.63	0.51	0.48
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.03
query40	0.16	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.41 s
Total hot run time: 28.44 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

BE Regression && UT Coverage Report

Increment line coverage 82.61% (133/161) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.23% (20968/27506)
Line Coverage 69.63% (216203/310515)
Region Coverage 67.62% (129349/191281)
Branch Coverage 61.16% (67261/109976)

@morrySnow morrySnow changed the title branch-3.1: [enhancement](be_metrics) update scan bytes metric in file_scanner. branch-3.1: [enhancement](be_metrics) update scan bytes metric in file_scanner. #53729 Aug 4, 2025
@morrySnow morrySnow merged commit 03e8434 into apache:branch-3.1 Aug 4, 2025
23 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kaka11chen @hello-stephen @doris-robot @morningman @morrySnow