Skip to content

[Opt](multi-catalog)Disable dict filter in parquet/orc reader if have non-single conjuncts. #52536

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jul 8, 2025
Merged

[Opt](multi-catalog)Disable dict filter in parquet/orc reader if have non-single conjuncts. #52536

dataroaring merged 1 commit into from
Jul 8, 2025

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Jun 30, 2025

What problem does this PR solve?

Problem Summary:

Release note

Cherry-pick #44777

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@kaka11chen kaka11chen requested a review from dataroaring as a code owner June 30, 2025 09:31
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Jun 30, 2025

run buildall

@Thearas
Copy link
Contributor

Thearas commented Jun 30, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

doris-robot commented Jun 30, 2025

TPC-H: Total hot run time: 39368 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f9467137afa7cab0caa26ce2687d9b6beaeb634f, data reload: false

------ Round 1 ----------------------------------
q1	11602	6670	6589	6589
q2	1773	164	161	161
q3	2690	1201	1039	1039
q4	1605	673	663	663
q5	2957	2888	2836	2836
q6	216	133	131	131
q7	984	628	608	608
q8	2005	1961	1976	1961
q9	6594	6378	6408	6378
q10	7004	2244	2257	2244
q11	457	256	263	256
q12	382	208	201	201
q13	4466	2943	2982	2943
q14	237	198	206	198
q15	516	477	455	455
q16	468	374	369	369
q17	941	611	501	501
q18	7336	6698	6551	6551
q19	1223	1042	1045	1042
q20	477	193	201	193
q21	3875	3111	3176	3111
q22	1075	979	938	938
Total cold run time: 58883 ms
Total hot run time: 39368 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6548	6547	6564	6547
q2	330	234	233	233
q3	2868	2748	2762	2748
q4	1925	1704	1704	1704
q5	5404	5440	5443	5440
q6	205	127	123	123
q7	2119	1742	1697	1697
q8	3233	3364	3714	3364
q9	8522	8595	8843	8595
q10	3592	3559	3537	3537
q11	601	494	504	494
q12	788	618	602	602
q13	3896	3149	3129	3129
q14	303	274	279	274
q15	501	479	470	470
q16	484	426	439	426
q17	1816	1609	1634	1609
q18	8145	7814	7523	7523
q19	1713	1548	1691	1548
q20	2112	1896	1856	1856
q21	5340	5029	5095	5029
q22	1108	1053	1015	1015
Total cold run time: 61553 ms
Total hot run time: 57963 ms

@doris-robot
Copy link

doris-robot commented Jun 30, 2025

TPC-DS: Total hot run time: 190867 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f9467137afa7cab0caa26ce2687d9b6beaeb634f, data reload: false

query1	806	373	366	366
query2	6297	1818	1869	1818
query3	6461	224	213	213
query4	33170	23840	23829	23829
query5	4397	455	426	426
query6	259	180	182	180
query7	4443	314	310	310
query8	275	228	215	215
query9	9346	2578	2563	2563
query10	476	255	249	249
query11	18245	15171	15129	15129
query12	156	103	100	100
query13	1711	447	417	417
query14	10320	6429	6675	6429
query15	256	166	170	166
query16	8065	454	453	453
query17	1544	558	539	539
query18	2123	326	336	326
query19	239	168	162	162
query20	120	112	108	108
query21	210	101	103	101
query22	4855	4595	4621	4595
query23	35190	34234	34034	34034
query24	11831	3010	2853	2853
query25	726	416	435	416
query26	1931	178	176	176
query27	3269	361	353	353
query28	8085	2189	2171	2171
query29	1129	454	463	454
query30	318	156	168	156
query31	1053	810	811	810
query32	98	59	54	54
query33	802	301	308	301
query34	1254	504	520	504
query35	893	752	706	706
query36	1115	915	950	915
query37	193	67	67	67
query38	4108	3858	3956	3858
query39	1586	1524	1648	1524
query40	289	102	107	102
query41	55	50	53	50
query42	121	100	100	100
query43	515	469	475	469
query44	1280	854	838	838
query45	186	165	170	165
query46	1230	698	729	698
query47	2021	1881	1927	1881
query48	477	388	388	388
query49	1274	401	373	373
query50	838	419	408	408
query51	7674	7309	7246	7246
query52	97	85	86	85
query53	252	180	183	180
query54	1156	448	448	448
query55	74	76	73	73
query56	267	255	231	231
query57	1296	1152	1164	1152
query58	223	199	205	199
query59	3055	2795	2950	2795
query60	289	260	249	249
query61	106	106	109	106
query62	867	659	675	659
query63	216	189	182	182
query64	5165	663	627	627
query65	3267	3180	3187	3180
query66	1364	285	308	285
query67	15780	15581	15487	15487
query68	5606	579	578	578
query69	421	263	253	253
query70	1173	1080	1099	1080
query71	338	261	252	252
query72	6350	4031	4113	4031
query73	752	344	360	344
query74	10318	8941	8960	8941
query75	3379	2601	2655	2601
query76	3377	1119	1100	1100
query77	402	272	272	272
query78	10517	9649	9478	9478
query79	2447	608	607	607
query80	1119	420	410	410
query81	530	219	219	219
query82	856	89	86	86
query83	245	137	144	137
query84	253	82	72	72
query85	1516	305	303	303
query86	417	298	305	298
query87	4345	4220	4196	4196
query88	3954	2361	2321	2321
query89	392	288	283	283
query90	2296	184	184	184
query91	185	147	146	146
query92	67	48	49	48
query93	1666	564	545	545
query94	1100	272	283	272
query95	355	246	250	246
query96	602	276	277	276
query97	3264	3098	3139	3098
query98	215	200	198	198
query99	1504	1290	1366	1290
Total cold run time: 306750 ms
Total hot run time: 190867 ms

@doris-robot
Copy link

doris-robot commented Jun 30, 2025

ClickBench: Total hot run time: 29.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f9467137afa7cab0caa26ce2687d9b6beaeb634f, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.02	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.54	0.51	0.54
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.05	0.02
query9	0.56	0.51	0.52
query10	0.56	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.58
query14	0.77	0.80	0.79
query15	0.84	0.81	0.82
query16	0.38	0.38	0.39
query17	1.03	1.03	1.01
query18	0.23	0.22	0.21
query19	1.91	1.78	1.81
query20	0.01	0.02	0.01
query21	15.39	0.57	0.56
query22	2.35	1.77	2.60
query23	16.98	0.90	0.87
query24	3.05	0.97	0.75
query25	0.26	0.30	0.09
query26	0.19	0.13	0.13
query27	0.04	0.06	0.04
query28	10.84	0.49	0.42
query29	12.58	3.21	3.18
query30	0.26	0.06	0.06
query31	2.86	0.38	0.37
query32	3.26	0.46	0.46
query33	2.97	3.04	3.02
query34	16.97	4.46	4.48
query35	4.52	4.52	4.54
query36	0.65	0.48	0.48
query37	0.09	0.06	0.07
query38	0.04	0.03	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.04
Total cold run time: 104.51 s
Total hot run time: 29.79 s

@kaka11chen @morningman
[Opt](multi-catalog)Disable dict filter in parquet/orc reader if have…
90b851c 
… non-single conjuncts. (apache#44777)

Related PR: apache#26386

Problem Summary:

Because of apache#26386, we split the conjunctions into
`single_slot_filter_conjuncts` and `non_single_slot_filter_conjuncts`,
where you can use the final dictionary filtering and delayed
materialization of the `single_slot_filter_conjuncts` part to perform
`non_single_slot_filter_conjuncts`. However, this results in fewer
conditions for late materialization, so the effect will be poor.
@morningman morningman force-pushed the cherry-pick-44777_3.0 branch from f946713 to 90b851c Compare July 3, 2025 06:56
@morningman
Copy link
Contributor

morningman commented Jul 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 3, 2025

TPC-H: Total hot run time: 39707 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 90b851cea96798eae2cacffb73e21c32bd9bd7e0, data reload: false

------ Round 1 ----------------------------------
q1	17615	6748	6602	6602
q2	2049	166	164	164
q3	10859	1073	1228	1073
q4	10412	718	683	683
q5	7736	2882	2835	2835
q6	209	134	138	134
q7	1003	612	598	598
q8	9354	1930	2010	1930
q9	6577	6381	6410	6381
q10	6979	2239	2273	2239
q11	476	263	262	262
q12	392	213	216	213
q13	17810	2996	2993	2993
q14	237	199	206	199
q15	513	462	485	462
q16	495	375	374	374
q17	990	524	603	524
q18	7160	6699	6730	6699
q19	1398	1086	1038	1038
q20	454	198	197	197
q21	4272	3325	3117	3117
q22	1090	990	1017	990
Total cold run time: 108080 ms
Total hot run time: 39707 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6585	6602	6600	6600
q2	319	242	232	232
q3	2942	2764	2882	2764
q4	2026	1803	1779	1779
q5	5798	5747	5733	5733
q6	211	134	134	134
q7	2205	1800	1782	1782
q8	3409	3582	3554	3554
q9	8930	8806	8991	8806
q10	3563	3522	3525	3522
q11	585	500	496	496
q12	803	584	575	575
q13	10247	3141	3276	3141
q14	305	262	262	262
q15	512	469	465	465
q16	487	460	429	429
q17	1854	1640	1603	1603
q18	8250	7694	7655	7655
q19	1670	1592	1667	1592
q20	2141	1859	1820	1820
q21	5039	5131	4944	4944
q22	1146	1065	1076	1065
Total cold run time: 69027 ms
Total hot run time: 58953 ms

@doris-robot
Copy link

doris-robot commented Jul 3, 2025

BE UT Coverage Report

Increment line coverage 5.56% (3/54) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.24% (10945/26538)
Line Coverage 32.02% (93685/292599)
Region Coverage 31.13% (48327/155223)
Branch Coverage 27.58% (24752/89760)

@doris-robot
Copy link

doris-robot commented Jul 3, 2025

TPC-DS: Total hot run time: 196953 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 90b851cea96798eae2cacffb73e21c32bd9bd7e0, data reload: false

query1	1276	897	919	897
query2	6390	1901	1847	1847
query3	10961	4424	4351	4351
query4	62501	29519	23972	23972
query5	5137	477	464	464
query6	431	186	176	176
query7	5443	317	309	309
query8	333	240	225	225
query9	8391	2565	2536	2536
query10	436	266	263	263
query11	17795	15517	16241	15517
query12	162	105	114	105
query13	1445	434	430	430
query14	10067	6904	6793	6793
query15	200	187	176	176
query16	7087	531	481	481
query17	1122	594	620	594
query18	1848	335	329	329
query19	210	167	161	161
query20	120	113	112	112
query21	204	111	108	108
query22	4596	4413	4545	4413
query23	34539	34221	34725	34221
query24	7097	2979	2941	2941
query25	533	421	430	421
query26	652	169	172	169
query27	2343	356	356	356
query28	4442	2115	2140	2115
query29	710	453	440	440
query30	249	155	154	154
query31	1017	813	787	787
query32	68	57	60	57
query33	456	307	300	300
query34	923	508	527	508
query35	839	725	756	725
query36	1072	982	999	982
query37	114	71	66	66
query38	4099	4016	3943	3943
query39	1528	1502	1492	1492
query40	195	98	101	98
query41	54	47	48	47
query42	117	106	101	101
query43	499	484	485	484
query44	1189	800	812	800
query45	189	170	170	170
query46	1164	742	725	725
query47	2009	1875	1906	1875
query48	469	381	376	376
query49	741	400	389	389
query50	847	430	421	421
query51	7442	7081	7257	7081
query52	100	92	95	92
query53	267	199	189	189
query54	587	468	468	468
query55	77	80	77	77
query56	269	275	253	253
query57	1310	1188	1224	1188
query58	210	207	229	207
query59	3258	2996	2988	2988
query60	285	250	260	250
query61	110	118	111	111
query62	797	702	687	687
query63	228	192	196	192
query64	1429	699	631	631
query65	3330	3176	3261	3176
query66	705	292	288	288
query67	15838	15505	15338	15338
query68	4208	577	589	577
query69	425	264	266	264
query70	1123	1143	1125	1125
query71	363	270	250	250
query72	6488	4154	3856	3856
query73	747	344	364	344
query74	9810	9031	9290	9031
query75	3330	2662	2670	2662
query76	2224	1173	1032	1032
query77	488	293	281	281
query78	10610	9503	9594	9503
query79	1190	613	619	613
query80	826	433	463	433
query81	498	221	224	221
query82	1307	88	88	88
query83	240	147	142	142
query84	286	83	79	79
query85	875	316	301	301
query86	312	303	278	278
query87	4334	4234	4241	4234
query88	3625	2368	2334	2334
query89	421	296	292	292
query90	1996	187	191	187
query91	184	165	150	150
query92	66	48	53	48
query93	1260	554	547	547
query94	780	297	290	290
query95	354	268	258	258
query96	602	275	285	275
query97	3315	3124	3150	3124
query98	214	197	208	197
query99	1640	1299	1317	1299
Total cold run time: 314692 ms
Total hot run time: 196953 ms

dataroaring
dataroaring approved these changes Jul 8, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 4fffbeb into apache:branch-3.0 Jul 8, 2025
20 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kaka11chen @Thearas @doris-robot @morningman @dataroaring