Skip to content

branch-3.1: [Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet orc profiles. #51248 #52615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 2, 2025
Merged

branch-3.1: [Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet orc profiles. #51248 #52615

morrySnow merged 1 commit into from
Jul 2, 2025

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Jul 1, 2025 •
edited by morrySnow
Loading

Cherry-pick #51248

@kaka11chen
[Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewri…
f184a27 
…teTime, LazyReadFilteredRows profile metrics in parquet orc profiles. (apache#51248)

[Enhancement] (multi-catalog) Add PredicateFilterTime,
DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet/orc profiles.
@kaka11chen kaka11chen requested a review from morrySnow as a code owner July 1, 2025 14:19
@Thearas
Copy link
Contributor

Thearas commented Jul 1, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen
Copy link
Contributor Author

kaka11chen commented Jul 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

TPC-H: Total hot run time: 40184 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f184a2753fc1f57f984eeaf4b6c8196362aaf35e, data reload: false

------ Round 1 ----------------------------------
q1	17573	7052	6718	6718
q2	2065	214	176	176
q3	10571	1182	1200	1182
q4	11145	790	804	790
q5	7740	2825	2862	2825
q6	210	133	134	133
q7	1002	628	608	608
q8	9351	2011	2018	2011
q9	6747	6470	6487	6470
q10	6988	2304	2305	2304
q11	458	263	267	263
q12	394	220	208	208
q13	17798	2991	2980	2980
q14	247	207	195	195
q15	507	469	469	469
q16	463	376	375	375
q17	1001	600	654	600
q18	7234	6650	6593	6593
q19	1337	1060	948	948
q20	508	206	204	204
q21	3899	3172	3152	3152
q22	1096	980	1001	980
Total cold run time: 108334 ms
Total hot run time: 40184 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6640	6631	6629	6629
q2	328	238	234	234
q3	3038	2971	2966	2966
q4	2087	1849	1862	1849
q5	5708	5738	5796	5738
q6	212	124	127	124
q7	2183	1830	1790	1790
q8	3390	3509	3523	3509
q9	8887	8947	9011	8947
q10	3585	3553	3541	3541
q11	606	507	480	480
q12	832	598	612	598
q13	4863	3100	3159	3100
q14	290	266	264	264
q15	513	462	452	452
q16	476	422	421	421
q17	1838	1646	1601	1601
q18	8199	7788	7655	7655
q19	1711	1513	1628	1513
q20	2087	1835	1873	1835
q21	5026	4953	4843	4843
q22	1102	995	991	991
Total cold run time: 63601 ms
Total hot run time: 59080 ms

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

TPC-DS: Total hot run time: 189699 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f184a2753fc1f57f984eeaf4b6c8196362aaf35e, data reload: false

query1	975	401	357	357
query2	6518	1935	1874	1874
query3	6705	215	214	214
query4	34158	23621	23414	23414
query5	4364	456	464	456
query6	285	187	194	187
query7	4626	312	311	311
query8	294	233	228	228
query9	9369	2568	2569	2568
query10	504	275	256	256
query11	18184	15540	15295	15295
query12	163	104	102	102
query13	1656	422	424	422
query14	9699	7071	7109	7071
query15	251	176	182	176
query16	8132	464	497	464
query17	1652	585	573	573
query18	2149	303	315	303
query19	363	155	164	155
query20	118	109	111	109
query21	216	105	110	105
query22	4495	4189	4284	4189
query23	34261	33988	33783	33783
query24	11152	2863	2903	2863
query25	655	391	377	377
query26	1621	165	170	165
query27	2862	339	347	339
query28	7742	2103	2103	2103
query29	988	427	430	427
query30	323	157	164	157
query31	993	804	816	804
query32	97	60	60	60
query33	793	296	309	296
query34	939	491	510	491
query35	859	711	712	711
query36	1059	922	953	922
query37	133	70	70	70
query38	4014	3861	3860	3860
query39	1480	1432	1425	1425
query40	289	97	98	97
query41	52	60	49	49
query42	120	99	102	99
query43	510	448	460	448
query44	1230	785	790	785
query45	184	170	167	167
query46	1171	733	738	733
query47	1945	1894	1905	1894
query48	436	332	327	327
query49	1286	385	396	385
query50	824	423	419	419
query51	7204	7013	7094	7013
query52	101	89	91	89
query53	256	182	180	180
query54	1349	464	481	464
query55	77	77	81	77
query56	266	264	237	237
query57	1283	1150	1168	1150
query58	249	216	212	212
query59	3059	3015	2777	2777
query60	292	257	260	257
query61	114	110	125	110
query62	896	666	681	666
query63	218	185	187	185
query64	5196	650	660	650
query65	3304	3211	3198	3198
query66	1453	307	323	307
query67	15888	15606	15614	15606
query68	3186	603	597	597
query69	406	272	268	268
query70	1166	1102	1085	1085
query71	335	257	255	255
query72	6106	2622	3919	2622
query73	752	360	362	360
query74	9729	9296	9379	9296
query75	3365	2654	2671	2654
query76	1931	1068	1094	1068
query77	427	271	272	271
query78	10321	9603	9538	9538
query79	1095	604	639	604
query80	634	431	445	431
query81	490	224	222	222
query82	183	98	88	88
query83	164	144	155	144
query84	238	75	85	75
query85	994	309	294	294
query86	308	299	294	294
query87	4391	4222	4217	4217
query88	3579	2386	2355	2355
query89	399	305	298	298
query90	2089	186	182	182
query91	146	118	109	109
query92	60	51	53	51
query93	1045	569	566	566
query94	837	287	293	287
query95	366	253	264	253
query96	607	292	278	278
query97	3312	3174	3157	3157
query98	208	201	203	201
query99	1527	1316	1334	1316
Total cold run time: 294495 ms
Total hot run time: 189699 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 1, 2025

BE UT Coverage Report

Increment line coverage 9.46% (14/148) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.06% (12384/27481)
Line Coverage 36.05% (109649/304140)
Region Coverage 35.12% (56819/161765)
Branch Coverage 32.25% (30820/95568)

@morrySnow morrySnow changed the title [Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet orc profiles. branch-3.1: [Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet orc profiles. #51248 Jul 2, 2025
@morrySnow morrySnow merged commit efba6cb into apache:branch-3.1 Jul 2, 2025
21 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kaka11chen @Thearas @doris-robot @hello-stephen @morrySnow