Skip to content

Conversation

@yiguolei
Copy link
Contributor

@yiguolei yiguolei commented Jun 24, 2025

What problem does this PR solve?

  1. unify scanbytes and scan rows definition in audit log,doris metrics,profile。 scan bytes is the uncompressed data read from the file + from the page cache. scan rows is the raw rows read from the file.
  2. add update realtime counter interface in scanner, so other scanner type could implement this interface to get these counters realtime.
  3. the uncompressed data read in page io is wrong.
  4. cputimer counter is not updated correctly.

DO NOT merge this pr to 3.0 , because there is some behavior change for scan bytes. If user has config workload group policy or some monitor on scan bytes metric, there will be something wrong.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jun 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yiguolei
Copy link
Contributor Author

run buildall

@yiguolei yiguolei changed the title [bugfix&enhancement](be_metrics) update scan bytes metric correctly [bugfix](be_metrics) update scan bytes metric correctly Jun 24, 2025
@yiguolei
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 4.76% (1/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.93% (15338/26942)
Line Coverage 46.01% (139028/302184)
Region Coverage 45.36% (70422/155261)
Branch Coverage 40.15% (37208/92678)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (21/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 62.37% (16530/26504)
Line Coverage 52.19% (157613/301996)
Region Coverage 49.67% (90506/182227)
Branch Coverage 43.31% (45670/105450)

@yiguolei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33866 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 654d5b947ddccab0c629d4273fddb467e7819204, data reload: false

------ Round 1 ----------------------------------
q1	17612	5207	5069	5069
q2	1932	275	188	188
q3	10347	1360	698	698
q4	10241	1030	516	516
q5	7809	2413	2325	2325
q6	184	155	126	126
q7	892	750	586	586
q8	9294	1301	1152	1152
q9	6965	5096	5108	5096
q10	6884	2384	1945	1945
q11	509	296	278	278
q12	336	358	214	214
q13	17769	3705	3097	3097
q14	234	221	215	215
q15	559	486	483	483
q16	416	423	374	374
q17	616	878	356	356
q18	7515	7178	7081	7081
q19	1440	953	540	540
q20	332	332	215	215
q21	3764	3160	2338	2338
q22	1025	1020	974	974
Total cold run time: 106675 ms
Total hot run time: 33866 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5113	5072	5062	5062
q2	239	319	219	219
q3	2228	2719	2311	2311
q4	1374	1799	1314	1314
q5	4203	4364	4359	4359
q6	210	166	126	126
q7	1976	1915	1772	1772
q8	2627	2564	2521	2521
q9	7099	7121	7160	7121
q10	3052	3200	2786	2786
q11	586	515	486	486
q12	697	744	619	619
q13	3594	3901	3275	3275
q14	278	290	275	275
q15	522	480	480	480
q16	434	479	428	428
q17	1150	1547	1375	1375
q18	7358	7212	6909	6909
q19	764	799	858	799
q20	1919	1957	1812	1812
q21	4810	4354	4223	4223
q22	1048	1066	991	991
Total cold run time: 51281 ms
Total hot run time: 49263 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185065 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 654d5b947ddccab0c629d4273fddb467e7819204, data reload: false

query1	1012	403	406	403
query2	6548	1663	1659	1659
query3	6739	217	211	211
query4	26679	23845	23507	23507
query5	4350	601	450	450
query6	332	224	195	195
query7	4620	508	305	305
query8	262	225	211	211
query9	8614	2646	2644	2644
query10	460	329	267	267
query11	15941	15095	14935	14935
query12	159	112	101	101
query13	1636	518	409	409
query14	9291	5680	5649	5649
query15	206	187	172	172
query16	7418	644	469	469
query17	1197	797	556	556
query18	2006	407	295	295
query19	186	185	167	167
query20	114	125	113	113
query21	212	130	111	111
query22	4151	4236	3942	3942
query23	34221	33016	33120	33016
query24	8572	2357	2356	2356
query25	530	459	407	407
query26	1246	266	147	147
query27	2781	525	353	353
query28	4297	2129	2112	2112
query29	749	557	435	435
query30	289	225	199	199
query31	918	840	762	762
query32	80	64	60	60
query33	592	353	306	306
query34	784	837	523	523
query35	779	833	738	738
query36	939	971	869	869
query37	114	101	78	78
query38	4152	4192	4045	4045
query39	1506	1438	1417	1417
query40	205	119	102	102
query41	56	54	52	52
query42	125	105	102	102
query43	496	538	456	456
query44	1297	820	800	800
query45	176	172	167	167
query46	828	1027	625	625
query47	1805	1791	1685	1685
query48	390	402	313	313
query49	742	481	404	404
query50	632	684	419	419
query51	4294	4150	4113	4113
query52	106	106	104	104
query53	226	256	183	183
query54	578	573	505	505
query55	85	80	88	80
query56	316	307	272	272
query57	1205	1191	1109	1109
query58	264	252	261	252
query59	2520	2695	2457	2457
query60	351	303	316	303
query61	126	126	133	126
query62	793	720	666	666
query63	217	188	185	185
query64	4302	993	655	655
query65	4364	4231	4239	4231
query66	1098	403	313	313
query67	15796	15558	15280	15280
query68	5990	897	535	535
query69	499	315	284	284
query70	1167	1114	1031	1031
query71	400	319	286	286
query72	5852	4791	4871	4791
query73	654	642	347	347
query74	8869	9255	8651	8651
query75	3238	3204	2732	2732
query76	3139	1179	718	718
query77	489	395	281	281
query78	10442	10461	9634	9634
query79	1936	814	585	585
query80	653	514	506	506
query81	504	265	237	237
query82	211	128	99	99
query83	267	269	262	262
query84	255	112	87	87
query85	823	434	398	398
query86	327	283	284	283
query87	4463	4365	4261	4261
query88	2895	2247	2268	2247
query89	481	313	279	279
query90	1756	205	202	202
query91	134	136	112	112
query92	63	59	61	59
query93	1018	940	597	597
query94	626	407	291	291
query95	378	285	285	285
query96	493	573	281	281
query97	2688	2745	2672	2672
query98	236	214	223	214
query99	1338	1420	1265	1265
Total cold run time: 270359 ms
Total hot run time: 185065 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.06 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 654d5b947ddccab0c629d4273fddb467e7819204, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.24	0.08	0.09
query4	1.62	0.11	0.11
query5	0.42	0.44	0.41
query6	1.18	0.65	0.65
query7	0.03	0.02	0.01
query8	0.05	0.03	0.03
query9	0.60	0.51	0.52
query10	0.57	0.58	0.57
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.62	0.62
query14	0.80	0.82	0.81
query15	0.92	0.87	0.86
query16	0.37	0.39	0.40
query17	1.08	1.06	1.07
query18	0.23	0.22	0.21
query19	1.92	1.87	1.87
query20	0.02	0.01	0.01
query21	15.39	0.91	0.55
query22	0.76	1.25	0.65
query23	14.90	1.40	0.62
query24	7.38	1.11	0.37
query25	0.38	0.24	0.09
query26	0.66	0.17	0.15
query27	0.06	0.06	0.05
query28	8.85	0.96	0.44
query29	12.57	4.04	3.33
query30	0.25	0.09	0.06
query31	2.85	0.59	0.38
query32	3.25	0.56	0.46
query33	3.10	3.10	3.05
query34	16.10	5.36	4.77
query35	4.80	4.88	4.77
query36	0.69	0.51	0.49
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.59 s
Total hot run time: 29.06 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 9.38% (3/32) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.02% (15364/26946)
Line Coverage 46.11% (139420/302384)
Region Coverage 45.44% (70649/155471)
Branch Coverage 40.20% (37315/92816)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 1, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jul 1, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 1, 2025

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit af77883 into apache:master Jul 1, 2025
25 of 28 checks passed
yiguolei added a commit to yiguolei/incubator-doris that referenced this pull request Jul 1, 2025
1. unify scanbytes and scan rows definition in audit log,doris
metrics,profile。 scan bytes is the uncompressed data read from the file
+ from the page cache. scan rows is the raw rows read from the file.
2. add update realtime counter interface in scanner, so other scanner
type could implement this interface to get these counters realtime.
3. the uncompressed data read in page io is wrong.
4. cputimer counter is not updated correctly.

DO NOT merge this pr to 3.0 or 3.1, because there is some behavior
change for scan bytes. If user has config workload group policy or some
monitor on scan bytes metric, there will be something wrong.
morrySnow pushed a commit that referenced this pull request Jul 1, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jul 4, 2025
### What problem does this PR solve?
1. unify scanbytes and scan rows definition in audit log,doris
metrics,profile。 scan bytes is the uncompressed data read from the file
+ from the page cache. scan rows is the raw rows read from the file.
2. add update realtime counter interface in scanner, so other scanner
type could implement this interface to get these counters realtime.
3. the uncompressed data read in page io is wrong.
4. cputimer counter is not updated correctly.

DO NOT merge this pr to 3.0 or 3.1, because there is some behavior
change for scan bytes. If user has config workload group policy or some
monitor on scan bytes metric, there will be something wrong.
koarz pushed a commit to koarz/doris that referenced this pull request Jul 4, 2025
### What problem does this PR solve?
1. unify scanbytes and scan rows definition in audit log,doris
metrics,profile。 scan bytes is the uncompressed data read from the file
+ from the page cache. scan rows is the raw rows read from the file.
2. add update realtime counter interface in scanner, so other scanner
type could implement this interface to get these counters realtime.
3. the uncompressed data read in page io is wrong.
4. cputimer counter is not updated correctly.

DO NOT merge this pr to 3.0 or 3.1, because there is some behavior
change for scan bytes. If user has config workload group policy or some
monitor on scan bytes metric, there will be something wrong.
koarz pushed a commit to koarz/doris that referenced this pull request Jul 4, 2025
### What problem does this PR solve?
1. unify scanbytes and scan rows definition in audit log,doris
metrics,profile。 scan bytes is the uncompressed data read from the file
+ from the page cache. scan rows is the raw rows read from the file.
2. add update realtime counter interface in scanner, so other scanner
type could implement this interface to get these counters realtime.
3. the uncompressed data read in page io is wrong.
4. cputimer counter is not updated correctly.

DO NOT merge this pr to 3.0 or 3.1, because there is some behavior
change for scan bytes. If user has config workload group policy or some
monitor on scan bytes metric, there will be something wrong.
morningman pushed a commit that referenced this pull request Jul 28, 2025
…53729)

### What problem does this PR solve?

Problem Summary:

### Release note

The external part is implemented according to the framework defined by
the unified audit log, Doris metrics, and scanbytes and scan rows in
Profile in #52232.

However,
- Scan bytes in the external table currently represents the bytes
counted by **the top-level File Reader** called by the scan reader
layer.
- Scan rows represents the number of scanned rows of the underlying
storage. The number of scanned rows of parquet/orc does not include the
number of rows of skipped page/rowgroup. **Note: However, there is still
a problem that `jni_reader` has not yet implemented the number of rows
that only contain the storage to be read.**
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Aug 1, 2025
…pache#53729)

Problem Summary:

The external part is implemented according to the framework defined by
the unified audit log, Doris metrics, and scanbytes and scan rows in
Profile in apache#52232.

However,
- Scan bytes in the external table currently represents the bytes
counted by **the top-level File Reader** called by the scan reader
layer.
- Scan rows represents the number of scanned rows of the underlying
storage. The number of scanned rows of parquet/orc does not include the
number of rows of skipped page/rowgroup. **Note: However, there is still
a problem that `jni_reader` has not yet implemented the number of rows
that only contain the storage to be read.**
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Aug 1, 2025
…pache#53729)

Problem Summary:

The external part is implemented according to the framework defined by
the unified audit log, Doris metrics, and scanbytes and scan rows in
Profile in apache#52232.

However,
- Scan bytes in the external table currently represents the bytes
counted by **the top-level File Reader** called by the scan reader
layer.
- Scan rows represents the number of scanned rows of the underlying
storage. The number of scanned rows of parquet/orc does not include the
number of rows of skipped page/rowgroup. **Note: However, there is still
a problem that `jni_reader` has not yet implemented the number of rows
that only contain the storage to be read.**
zhiqiang-hhhh pushed a commit to zhiqiang-hhhh/doris that referenced this pull request Aug 4, 2025
…pache#53729)

### What problem does this PR solve?

Problem Summary:

### Release note

The external part is implemented according to the framework defined by
the unified audit log, Doris metrics, and scanbytes and scan rows in
Profile in apache#52232.

However,
- Scan bytes in the external table currently represents the bytes
counted by **the top-level File Reader** called by the scan reader
layer.
- Scan rows represents the number of scanned rows of the underlying
storage. The number of scanned rows of parquet/orc does not include the
number of rows of skipped page/rowgroup. **Note: However, there is still
a problem that `jni_reader` has not yet implemented the number of rows
that only contain the storage to be read.**
freemandealer added a commit to freemandealer/doris that referenced this pull request Sep 11, 2025
forget update profile counters in update_realtime_counters
bug introduced in apache#52232

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
freemandealer added a commit to freemandealer/doris that referenced this pull request Sep 11, 2025
forget update profile counters in update_realtime_counters
bug introduced in apache#52232

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
dataroaring pushed a commit that referenced this pull request Sep 14, 2025
forget update profile counters in update_realtime_counters. bug
introduced in #52232
freemandealer added a commit to freemandealer/doris that referenced this pull request Oct 30, 2025
…55929)

forget update profile counters in update_realtime_counters. bug
introduced in apache#52232
freemandealer added a commit to freemandealer/doris that referenced this pull request Oct 30, 2025
…55929)

forget update profile counters in update_realtime_counters. bug
introduced in apache#52232
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants