Skip to content

branch-3.1: [bugfix](be_metrics) update scan bytes metric correctly #52232 #52584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 1, 2025
Merged

branch-3.1: [bugfix](be_metrics) update scan bytes metric correctly #52232 #52584

morrySnow merged 1 commit into from
Jul 1, 2025

Conversation

@yiguolei
Copy link
Contributor

@yiguolei yiguolei commented Jul 1, 2025 •
edited by morrySnow
Loading

Cherry-picked from #52232

@yiguolei
[bugfix](be_metrics) update scan bytes metric correctly (apache#52232)
b0dc6f1 
1. unify scanbytes and scan rows definition in audit log，doris
metrics，profile。 scan bytes is the uncompressed data read from the file
+ from the page cache. scan rows is the raw rows read from the file.
2. add update realtime counter interface in scanner, so other scanner
type could implement this interface to get these counters realtime.
3. the uncompressed data read in page io is wrong.
4. cputimer counter is not updated correctly.

DO NOT merge this pr to 3.0 or 3.1, because there is some behavior
change for scan bytes. If user has config workload group policy or some
monitor on scan bytes metric, there will be something wrong.
@yiguolei yiguolei requested a review from morrySnow as a code owner July 1, 2025 07:50
@yiguolei
Copy link
Contributor Author

yiguolei commented Jul 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

TPC-H: Total hot run time: 39910 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0dc6f121cdf9a96fdaaa295fe72317efa36d88c, data reload: false

------ Round 1 ----------------------------------
q1	17577	6999	6629	6629
q2	2060	201	204	201
q3	10566	1130	1208	1130
q4	10305	813	766	766
q5	7775	2840	2847	2840
q6	214	132	134	132
q7	987	633	600	600
q8	9351	1981	2027	1981
q9	6725	6442	6439	6439
q10	6986	2282	2279	2279
q11	467	268	258	258
q12	404	212	204	204
q13	17789	2998	3002	2998
q14	251	211	209	209
q15	504	470	468	468
q16	483	369	369	369
q17	982	522	556	522
q18	7303	6634	6668	6634
q19	1319	1041	992	992
q20	473	202	202	202
q21	3911	3202	3077	3077
q22	1105	982	980	980
Total cold run time: 107537 ms
Total hot run time: 39910 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6663	6590	6568	6568
q2	338	241	238	238
q3	2897	2744	2756	2744
q4	1899	1712	1725	1712
q5	5431	5479	5437	5437
q6	206	127	123	123
q7	2111	1708	1734	1708
q8	3266	3426	3448	3426
q9	8589	8571	8500	8500
q10	3481	3399	3395	3395
q11	596	483	487	483
q12	776	565	599	565
q13	10526	2960	2938	2938
q14	294	262	266	262
q15	508	456	451	451
q16	476	421	414	414
q17	1784	1568	1557	1557
q18	7896	7328	7513	7328
q19	1667	1511	1537	1511
q20	2034	1815	1788	1788
q21	5016	4854	4751	4751
q22	1113	1004	994	994
Total cold run time: 67567 ms
Total hot run time: 56893 ms

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

TPC-DS: Total hot run time: 189469 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0dc6f121cdf9a96fdaaa295fe72317efa36d88c, data reload: false

query1	992	372	370	370
query2	6517	1928	1888	1888
query3	6703	218	217	217
query4	33865	23802	23473	23473
query5	4793	471	460	460
query6	287	176	175	175
query7	4621	305	309	305
query8	285	228	228	228
query9	9582	2599	2585	2585
query10	487	291	252	252
query11	18663	15289	15312	15289
query12	157	103	106	103
query13	1664	419	424	419
query14	10643	6790	7185	6790
query15	255	173	177	173
query16	8050	466	492	466
query17	1605	564	543	543
query18	2136	306	299	299
query19	375	148	146	146
query20	111	107	107	107
query21	203	102	102	102
query22	4495	4137	4060	4060
query23	34320	33124	33569	33124
query24	11187	2803	2867	2803
query25	669	381	390	381
query26	1649	166	164	164
query27	2911	345	338	338
query28	7869	2113	2130	2113
query29	966	435	428	428
query30	318	157	159	157
query31	1011	775	775	775
query32	90	56	63	56
query33	791	304	301	301
query34	930	507	525	507
query35	871	715	716	715
query36	1064	892	914	892
query37	130	69	71	69
query38	3966	3835	3854	3835
query39	1441	1402	1403	1402
query40	283	103	99	99
query41	55	50	53	50
query42	118	109	103	103
query43	517	486	469	469
query44	1262	818	805	805
query45	186	171	171	171
query46	1135	741	740	740
query47	1929	1772	1853	1772
query48	427	343	347	343
query49	1277	425	412	412
query50	827	424	421	421
query51	7339	7162	7157	7157
query52	109	96	98	96
query53	261	186	194	186
query54	1352	488	473	473
query55	86	80	82	80
query56	279	268	264	264
query57	1307	1127	1166	1127
query58	246	223	229	223
query59	2972	2862	2912	2862
query60	300	276	270	270
query61	142	138	136	136
query62	877	702	689	689
query63	221	194	193	193
query64	5352	751	735	735
query65	3283	3221	3196	3196
query66	1456	325	331	325
query67	15891	15422	15370	15370
query68	4515	593	576	576
query69	446	270	272	270
query70	1185	1147	1093	1093
query71	420	269	266	266
query72	6872	3962	4120	3962
query73	769	355	360	355
query74	10303	9302	8836	8836
query75	3424	2657	2638	2638
query76	3025	1044	1107	1044
query77	416	275	282	275
query78	10526	9652	9518	9518
query79	1255	598	595	595
query80	936	447	435	435
query81	516	222	219	219
query82	1127	95	92	92
query83	229	153	144	144
query84	233	76	74	74
query85	1291	305	294	294
query86	385	302	293	293
query87	4379	4229	4279	4229
query88	3537	2410	2372	2372
query89	415	295	290	290
query90	1894	192	189	189
query91	144	109	111	109
query92	59	53	53	53
query93	1092	565	557	557
query94	891	294	288	288
query95	360	259	261	259
query96	603	282	286	282
query97	3246	3097	3130	3097
query98	215	202	188	188
query99	1490	1301	1300	1300
Total cold run time: 302359 ms
Total hot run time: 189469 ms

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

BE UT Coverage Report

Increment line coverage 10.71% (3/28) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.07% (12383/27475)
Line Coverage 36.07% (109594/303837)
Region Coverage 35.14% (56810/161662)
Branch Coverage 32.27% (30818/95490)

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

ClickBench: Total hot run time: 29.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0dc6f121cdf9a96fdaaa295fe72317efa36d88c, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.63	0.10	0.10
query5	0.54	0.51	0.51
query6	1.12	0.73	0.73
query7	0.04	0.02	0.01
query8	0.04	0.04	0.03
query9	0.56	0.50	0.50
query10	0.55	0.56	0.55
query11	0.14	0.09	0.10
query12	0.14	0.10	0.11
query13	0.61	0.61	0.60
query14	0.78	0.81	0.82
query15	0.85	0.83	0.83
query16	0.37	0.38	0.37
query17	1.02	1.05	1.02
query18	0.23	0.23	0.21
query19	1.98	1.80	1.86
query20	0.01	0.01	0.01
query21	15.42	0.61	0.59
query22	2.50	2.14	2.14
query23	16.85	0.91	0.85
query24	3.08	1.49	0.39
query25	0.18	0.30	0.05
query26	0.48	0.14	0.14
query27	0.05	0.05	0.04
query28	10.63	0.49	0.44
query29	12.55	3.25	3.23
query30	0.25	0.06	0.06
query31	2.87	0.37	0.38
query32	3.25	0.48	0.46
query33	3.00	2.98	3.06
query34	17.17	4.50	4.48
query35	4.55	4.52	4.48
query36	0.67	0.48	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.11	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.91 s
Total hot run time: 29.86 s

@morrySnow morrySnow changed the title branch-3.1: [bugfix](be_metrics) update scan bytes metric correctly (#52232) branch-3.1: [bugfix](be_metrics) update scan bytes metric correctly #52232 Jul 1, 2025
@morrySnow morrySnow merged commit 28a8c76 into apache:branch-3.1 Jul 1, 2025
23 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yiguolei @doris-robot @morrySnow