Skip to content

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Nov 29, 2024

What problem does this PR solve?

Related PR: #26386

Problem Summary:

Because of #26386, we split the conjunctions into single_slot_filter_conjuncts and non_single_slot_filter_conjuncts, where you can use the final dictionary filtering and delayed materialization of the single_slot_filter_conjuncts part to perform non_single_slot_filter_conjuncts. However, this results in fewer conditions for late materialization, so the effect will be poor.

Release note

At present, it is a matter of how many conditions there are for late materialization and whether to perform dictionary filtering on multiple columns. Because late materialization is more important, we use when there are non_single_slot_filter_conjuncts in the filter, directly without dictionary filtering, and use the conjuncts to do late materialization.

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen
Copy link
Contributor Author

run buildall

@kaka11chen kaka11chen force-pushed the disable_dict_filter_when_not_single_conjuncts branch from 6b0d1b8 to 1f8ccb3 Compare November 29, 2024 09:29
@kaka11chen
Copy link
Contributor Author

run buildall

@kaka11chen kaka11chen force-pushed the disable_dict_filter_when_not_single_conjuncts branch from 1f8ccb3 to 3c6b546 Compare November 29, 2024 09:44
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.33% (9976/26028)
Line Coverage: 29.42% (83544/284001)
Region Coverage: 28.53% (42965/150574)
Branch Coverage: 25.15% (21823/86760)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3c6b546b8735b1c074b62e2c756ba6b0aa483ae0_3c6b546b8735b1c074b62e2c756ba6b0aa483ae0/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40268 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c6b546b8735b1c074b62e2c756ba6b0aa483ae0, data reload: false

------ Round 1 ----------------------------------
q1	17777	7581	7322	7322
q2	2050	185	175	175
q3	10639	1057	1186	1057
q4	10356	780	760	760
q5	7600	2720	2736	2720
q6	244	150	145	145
q7	979	634	596	596
q8	9264	1885	1896	1885
q9	6685	6560	6531	6531
q10	6985	2302	2324	2302
q11	480	254	255	254
q12	414	225	225	225
q13	17773	3034	3064	3034
q14	242	210	217	210
q15	579	544	507	507
q16	662	587	583	583
q17	997	580	555	555
q18	7521	6745	6726	6726
q19	1326	1027	963	963
q20	467	180	182	180
q21	4052	3291	3222	3222
q22	385	319	316	316
Total cold run time: 107477 ms
Total hot run time: 40268 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7309	7307	7290	7290
q2	324	236	231	231
q3	2915	2877	2975	2877
q4	2095	1851	1891	1851
q5	5647	5692	5675	5675
q6	225	141	141	141
q7	2257	1854	1788	1788
q8	3405	3486	3564	3486
q9	8983	9086	9113	9086
q10	3601	3613	3549	3549
q11	597	495	498	495
q12	830	618	605	605
q13	12269	3329	3281	3281
q14	300	267	270	267
q15	601	530	535	530
q16	691	655	649	649
q17	1878	1600	1602	1600
q18	8415	7947	7663	7663
q19	1721	1609	1531	1531
q20	2150	1875	1865	1865
q21	5712	5609	5482	5482
q22	638	588	604	588
Total cold run time: 72563 ms
Total hot run time: 60530 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197585 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c6b546b8735b1c074b62e2c756ba6b0aa483ae0, data reload: false

query1	1243	922	926	922
query2	6248	2067	2114	2067
query3	11115	4380	4373	4373
query4	66903	28992	23545	23545
query5	4997	460	466	460
query6	417	189	183	183
query7	5620	300	293	293
query8	324	226	239	226
query9	8814	2722	2733	2722
query10	454	254	245	245
query11	17319	15346	15918	15346
query12	155	108	109	108
query13	1633	433	433	433
query14	11119	7533	7543	7533
query15	214	203	194	194
query16	7359	466	476	466
query17	1045	567	568	567
query18	1850	292	297	292
query19	190	146	178	146
query20	118	119	111	111
query21	198	100	107	100
query22	4774	4550	4502	4502
query23	35405	34374	34828	34374
query24	5594	2542	2506	2506
query25	488	397	387	387
query26	632	154	150	150
query27	1824	284	283	283
query28	4474	2489	2510	2489
query29	688	444	404	404
query30	210	157	162	157
query31	973	834	841	834
query32	64	56	55	55
query33	414	315	290	290
query34	938	516	532	516
query35	895	778	756	756
query36	1111	955	948	948
query37	118	80	71	71
query38	4481	4435	4389	4389
query39	1521	1472	1466	1466
query40	202	101	102	101
query41	45	43	47	43
query42	124	96	100	96
query43	538	516	490	490
query44	1161	826	830	826
query45	188	173	173	173
query46	1144	728	751	728
query47	2016	1945	1948	1945
query48	415	325	327	325
query49	719	403	398	398
query50	839	397	397	397
query51	7445	7143	7101	7101
query52	99	87	85	85
query53	256	183	173	173
query54	505	398	389	389
query55	78	74	77	74
query56	254	230	229	229
query57	1262	1138	1125	1125
query58	214	209	208	208
query59	3380	3210	3063	3063
query60	270	244	243	243
query61	112	105	103	103
query62	784	652	685	652
query63	207	191	186	186
query64	1364	700	643	643
query65	3279	3266	3169	3169
query66	624	306	297	297
query67	15866	15709	15520	15520
query68	4141	546	547	546
query69	409	248	246	246
query70	1159	1129	1060	1060
query71	318	245	241	241
query72	6383	4033	4001	4001
query73	766	435	357	357
query74	10096	9173	9050	9050
query75	3375	2646	2687	2646
query76	1855	1064	1053	1053
query77	456	270	273	270
query78	10334	9406	9398	9398
query79	2284	601	617	601
query80	1426	415	435	415
query81	510	235	224	224
query82	1280	117	111	111
query83	219	139	142	139
query84	279	68	71	68
query85	1005	296	311	296
query86	429	311	312	311
query87	4702	4569	4585	4569
query88	3984	2231	2179	2179
query89	413	287	289	287
query90	1866	183	182	182
query91	147	108	104	104
query92	68	50	50	50
query93	3239	541	530	530
query94	765	294	302	294
query95	341	245	240	240
query96	630	275	280	275
query97	2851	2632	2654	2632
query98	215	191	191	191
query99	1601	1348	1328	1328
Total cold run time: 322875 ms
Total hot run time: 197585 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.7 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3c6b546b8735b1c074b62e2c756ba6b0aa483ae0, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.10
query5	0.43	0.43	0.40
query6	1.18	0.66	0.66
query7	0.02	0.02	0.01
query8	0.03	0.03	0.03
query9	0.58	0.51	0.51
query10	0.55	0.55	0.57
query11	0.15	0.10	0.12
query12	0.13	0.12	0.11
query13	0.61	0.60	0.59
query14	2.83	2.74	2.84
query15	0.91	0.82	0.82
query16	0.39	0.39	0.38
query17	1.04	1.06	1.02
query18	0.23	0.22	0.21
query19	1.89	1.85	2.06
query20	0.02	0.01	0.01
query21	15.36	0.60	0.59
query22	2.73	2.25	1.40
query23	16.95	1.06	0.89
query24	2.99	2.09	1.48
query25	0.26	0.22	0.06
query26	0.56	0.14	0.14
query27	0.05	0.04	0.05
query28	9.50	1.10	1.07
query29	12.56	3.20	3.17
query30	0.26	0.06	0.06
query31	2.89	0.37	0.38
query32	3.28	0.47	0.47
query33	2.98	2.97	3.12
query34	17.02	4.47	4.49
query35	4.52	4.58	4.52
query36	0.68	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.04 s
Total hot run time: 32.7 s

@kaka11chen kaka11chen marked this pull request as ready for review December 19, 2024 08:29
@kaka11chen kaka11chen marked this pull request as draft December 19, 2024 08:32
@kaka11chen kaka11chen marked this pull request as ready for review December 30, 2024 09:08
@kaka11chen
Copy link
Contributor Author

run cloud_p0

@kaka11chen kaka11chen force-pushed the disable_dict_filter_when_not_single_conjuncts branch from 3c6b546 to 8640b08 Compare April 28, 2025 12:36
@kaka11chen
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 5.66% (3/53) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 54.66% (14738/26961)
Line Coverage 43.74% (129065/295061)
Region Coverage 42.48% (65891/155112)
Branch Coverage 37.04% (33206/89642)

@hello-stephen
Copy link
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 47.17% (25/53) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.05% (20656/26465)
Line Coverage 71.61% (210953/294577)
Region Coverage 69.76% (126624/181502)
Branch Coverage 63.05% (64431/102192)

@morningman morningman force-pushed the disable_dict_filter_when_not_single_conjuncts branch from 8640b08 to 1c37449 Compare May 16, 2025 06:50
@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33738 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1c3744973ba0e29bafab4e6bfad081e68e4eaf49, data reload: false

------ Round 1 ----------------------------------
q1	26124	5060	4992	4992
q2	2066	275	181	181
q3	10407	1236	696	696
q4	10231	1005	505	505
q5	7515	2646	2318	2318
q6	177	162	130	130
q7	912	737	638	638
q8	9348	1241	1008	1008
q9	6901	5088	5083	5083
q10	6862	2320	1886	1886
q11	486	290	279	279
q12	342	349	210	210
q13	17771	3653	3128	3128
q14	225	227	215	215
q15	552	492	486	486
q16	422	428	392	392
q17	619	880	403	403
q18	7762	7387	7130	7130
q19	1586	946	545	545
q20	345	339	220	220
q21	3765	3160	2317	2317
q22	1037	1011	976	976
Total cold run time: 115455 ms
Total hot run time: 33738 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5413	5056	5053	5053
q2	232	326	234	234
q3	2145	2629	2295	2295
q4	1304	1741	1359	1359
q5	4398	4379	4383	4379
q6	207	164	128	128
q7	2020	1923	1768	1768
q8	2566	2600	2466	2466
q9	7235	7234	6950	6950
q10	2991	3165	2760	2760
q11	584	510	500	500
q12	712	777	621	621
q13	3474	3895	3244	3244
q14	282	291	274	274
q15	531	474	479	474
q16	458	485	461	461
q17	1124	1510	1388	1388
q18	7787	7521	7405	7405
q19	782	831	959	831
q20	1973	2075	1838	1838
q21	4769	4403	4311	4311
q22	1085	1082	1027	1027
Total cold run time: 52072 ms
Total hot run time: 49766 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193042 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1c3744973ba0e29bafab4e6bfad081e68e4eaf49, data reload: false

query1	1396	1099	1081	1081
query2	6394	1822	1789	1789
query3	11042	4523	4512	4512
query4	55021	25391	23293	23293
query5	4954	502	440	440
query6	322	209	199	199
query7	4878	508	297	297
query8	315	253	242	242
query9	5367	2669	2675	2669
query10	434	334	287	287
query11	15029	15059	14816	14816
query12	154	111	107	107
query13	1031	534	412	412
query14	10149	6373	6270	6270
query15	208	204	179	179
query16	7083	670	517	517
query17	1059	753	618	618
query18	1562	431	322	322
query19	206	189	173	173
query20	132	124	130	124
query21	217	135	122	122
query22	4391	4434	4346	4346
query23	34474	33598	33548	33548
query24	6817	2458	2447	2447
query25	474	512	415	415
query26	706	277	159	159
query27	2305	520	346	346
query28	3047	2187	2189	2187
query29	614	582	443	443
query30	274	224	192	192
query31	864	839	791	791
query32	73	64	63	63
query33	464	359	300	300
query34	797	867	568	568
query35	811	833	744	744
query36	956	1001	910	910
query37	120	103	84	84
query38	4315	4287	4190	4190
query39	1509	1472	1656	1472
query40	206	120	107	107
query41	56	55	51	51
query42	127	116	115	115
query43	520	520	489	489
query44	1344	849	849	849
query45	184	176	165	165
query46	872	1087	660	660
query47	1854	1859	1810	1810
query48	403	444	345	345
query49	704	519	464	464
query50	677	704	424	424
query51	4212	4277	4268	4268
query52	112	106	101	101
query53	222	258	192	192
query54	597	593	544	544
query55	86	85	82	82
query56	305	321	315	315
query57	1206	1178	1143	1143
query58	266	269	259	259
query59	2740	2830	2702	2702
query60	333	345	297	297
query61	134	128	122	122
query62	753	747	680	680
query63	234	193	189	189
query64	1897	1031	668	668
query65	4293	4236	4247	4236
query66	712	399	304	304
query67	15809	15390	15549	15390
query68	7232	882	519	519
query69	546	320	266	266
query70	1189	1143	1064	1064
query71	495	344	301	301
query72	5862	4972	5055	4972
query73	1527	738	356	356
query74	9176	9049	8671	8671
query75	3834	3185	2712	2712
query76	4157	1192	751	751
query77	618	370	269	269
query78	10191	10073	9383	9383
query79	3043	814	582	582
query80	665	533	449	449
query81	486	268	221	221
query82	486	124	92	92
query83	349	247	233	233
query84	291	100	81	81
query85	775	363	307	307
query86	378	332	293	293
query87	4415	4422	4257	4257
query88	3814	2344	2360	2344
query89	411	311	295	295
query90	1856	213	220	213
query91	143	138	111	111
query92	75	61	55	55
query93	1977	928	576	576
query94	667	401	315	315
query95	377	288	295	288
query96	503	590	293	293
query97	2737	2751	2654	2654
query98	237	209	218	209
query99	1418	1413	1265	1265
Total cold run time: 300752 ms
Total hot run time: 193042 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.79 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c3744973ba0e29bafab4e6bfad081e68e4eaf49, data reload: false

query1	0.04	0.04	0.02
query2	0.12	0.10	0.10
query3	0.25	0.19	0.19
query4	1.58	0.20	0.19
query5	0.42	0.42	0.44
query6	1.18	0.67	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.59	0.51	0.52
query10	0.58	0.57	0.57
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.62	0.60
query14	0.78	0.81	0.84
query15	0.89	0.86	0.87
query16	0.39	0.40	0.37
query17	1.05	1.06	1.05
query18	0.23	0.22	0.21
query19	1.99	1.80	1.81
query20	0.01	0.02	0.01
query21	15.39	0.94	0.57
query22	0.76	1.20	0.73
query23	14.88	1.38	0.62
query24	7.36	1.47	0.40
query25	0.37	0.08	0.18
query26	0.68	0.16	0.14
query27	0.05	0.04	0.04
query28	8.89	0.86	0.44
query29	12.56	3.94	3.37
query30	0.25	0.09	0.06
query31	2.82	0.60	0.39
query32	3.23	0.55	0.47
query33	3.07	3.05	3.13
query34	15.84	5.12	4.47
query35	4.49	4.54	4.53
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 102.85 s
Total hot run time: 28.79 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 5.66% (3/53) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.87% (14885/26641)
Line Coverage 44.66% (131907/295332)
Region Coverage 43.78% (66415/151685)
Branch Coverage 38.37% (34028/88680)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 47.17% (25/53) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.40% (20826/26228)
Line Coverage 72.63% (214502/295343)
Region Coverage 70.83% (126226/178210)
Branch Coverage 64.54% (65375/101300)

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 19, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 51f89f1 into apache:master May 23, 2025
25 of 29 checks passed
kaka11chen added a commit to kaka11chen/doris that referenced this pull request May 29, 2025
… non-single conjuncts. (apache#44777)

Related PR: apache#26386

Problem Summary:

Because of apache#26386, we split the conjunctions into
`single_slot_filter_conjuncts` and `non_single_slot_filter_conjuncts`,
where you can use the final dictionary filtering and delayed
materialization of the `single_slot_filter_conjuncts` part to perform
`non_single_slot_filter_conjuncts`. However, this results in fewer
conditions for late materialization, so the effect will be poor.
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
… non-single conjuncts. (apache#44777)

### What problem does this PR solve?

Related PR: apache#26386

Problem Summary:

Because of apache#26386, we split the conjunctions into
`single_slot_filter_conjuncts` and `non_single_slot_filter_conjuncts`,
where you can use the final dictionary filtering and delayed
materialization of the `single_slot_filter_conjuncts` part to perform
`non_single_slot_filter_conjuncts`. However, this results in fewer
conditions for late materialization, so the effect will be poor.
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Jun 30, 2025
… non-single conjuncts. (apache#44777)

Related PR: apache#26386

Problem Summary:

Because of apache#26386, we split the conjunctions into
`single_slot_filter_conjuncts` and `non_single_slot_filter_conjuncts`,
where you can use the final dictionary filtering and delayed
materialization of the `single_slot_filter_conjuncts` part to perform
`non_single_slot_filter_conjuncts`. However, this results in fewer
conditions for late materialization, so the effect will be poor.
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Jun 30, 2025
… non-single conjuncts. (apache#44777)

Related PR: apache#26386

Problem Summary:

Because of apache#26386, we split the conjunctions into
`single_slot_filter_conjuncts` and `non_single_slot_filter_conjuncts`,
where you can use the final dictionary filtering and delayed
materialization of the `single_slot_filter_conjuncts` part to perform
`non_single_slot_filter_conjuncts`. However, this results in fewer
conditions for late materialization, so the effect will be poor.
morrySnow pushed a commit that referenced this pull request Jul 1, 2025
…ader if have non-single conjuncts #44777 (#52559)

Cherry-picked from #44777
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Jul 1, 2025
morningman pushed a commit to kaka11chen/doris that referenced this pull request Jul 3, 2025
… non-single conjuncts. (apache#44777)

Related PR: apache#26386

Problem Summary:

Because of apache#26386, we split the conjunctions into
`single_slot_filter_conjuncts` and `non_single_slot_filter_conjuncts`,
where you can use the final dictionary filtering and delayed
materialization of the `single_slot_filter_conjuncts` part to perform
`non_single_slot_filter_conjuncts`. However, this results in fewer
conditions for late materialization, so the effect will be poor.
yiguolei pushed a commit that referenced this pull request Jul 4, 2025
… non-single conjuncts. (#52617)

### What problem does this PR solve?

Problem Summary:

Cherry-pick #44777
dataroaring pushed a commit that referenced this pull request Jul 8, 2025
… non-single conjuncts. (#52536)

### What problem does this PR solve?

Problem Summary:

### Release note

Cherry-pick #44777
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants