Skip to content

branch-3.1: [feature](inverted index) Support the IS NOT NULL condition in expr via inverted indexes #50748 #52644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Jul 3, 2025
Merged

branch-3.1: [feature](inverted index) Support the IS NOT NULL condition in expr via inverted indexes #50748 #52644

morrySnow merged 2 commits into from
Jul 3, 2025

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Jul 2, 2025

pick from master #50748

@csun5285
[feature](inverted index) Support the IS NOT NULL condition in expr v…
67e8fc8 
…ia inverted indexes (apache#50748)

Support the IS NOT NULL condition in expr via inverted indexes
@csun5285 csun5285 requested a review from morrySnow as a code owner July 2, 2025 04:42
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

TPC-H: Total hot run time: 39658 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 67e8fc8b258c6235e15fe62f96e05096481ccd51, data reload: false

------ Round 1 ----------------------------------
q1	17575	6936	6669	6669
q2	2057	227	191	191
q3	10593	1150	1170	1150
q4	10539	764	719	719
q5	7716	2860	2786	2786
q6	218	132	136	132
q7	1021	650	629	629
q8	9437	1930	2031	1930
q9	6746	6420	6428	6420
q10	7041	2249	2324	2249
q11	470	269	270	269
q12	394	221	211	211
q13	17801	2987	2987	2987
q14	243	198	219	198
q15	517	469	480	469
q16	487	376	379	376
q17	975	544	554	544
q18	7522	6634	6592	6592
q19	1333	977	941	941
q20	493	205	204	204
q21	3908	3002	3124	3002
q22	1088	1000	990	990
Total cold run time: 108174 ms
Total hot run time: 39658 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6660	6656	6650	6650
q2	324	224	230	224
q3	2920	2932	3117	2932
q4	2100	1818	1803	1803
q5	5752	5742	5715	5715
q6	213	124	129	124
q7	2203	1792	1783	1783
q8	3353	3492	3525	3492
q9	8894	8865	8861	8861
q10	3560	3528	3492	3492
q11	590	479	508	479
q12	823	608	619	608
q13	6945	3141	3224	3141
q14	295	274	281	274
q15	513	460	462	460
q16	497	428	435	428
q17	1834	1596	1589	1589
q18	8074	7686	7745	7686
q19	1692	1610	1557	1557
q20	2168	1870	1890	1870
q21	5308	5112	4986	4986
q22	1130	991	984	984
Total cold run time: 65848 ms
Total hot run time: 59138 ms

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

TPC-DS: Total hot run time: 189392 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 67e8fc8b258c6235e15fe62f96e05096481ccd51, data reload: false

query1	967	370	360	360
query2	6524	2000	1916	1916
query3	6707	214	225	214
query4	33762	23980	23244	23244
query5	4305	462	487	462
query6	272	171	175	171
query7	4628	305	312	305
query8	287	222	244	222
query9	9686	2570	2561	2561
query10	496	269	257	257
query11	18539	15293	15139	15139
query12	163	101	102	101
query13	1643	432	425	425
query14	9667	6713	7321	6713
query15	272	173	179	173
query16	8149	463	465	463
query17	1625	601	572	572
query18	2151	314	316	314
query19	334	158	152	152
query20	118	108	107	107
query21	208	106	106	106
query22	4393	4168	4152	4152
query23	34228	33275	33304	33275
query24	11315	2931	2867	2867
query25	689	418	425	418
query26	1408	173	173	173
query27	2864	349	345	345
query28	7770	2107	2134	2107
query29	929	443	441	441
query30	320	165	160	160
query31	1024	786	803	786
query32	100	59	60	59
query33	808	313	309	309
query34	921	499	517	499
query35	854	720	729	720
query36	1084	914	944	914
query37	138	75	74	74
query38	4059	3781	3875	3781
query39	1469	1421	1433	1421
query40	294	103	103	103
query41	52	50	52	50
query42	117	103	99	99
query43	505	487	468	468
query44	1317	800	824	800
query45	186	173	175	173
query46	1162	732	734	732
query47	1932	1807	1826	1807
query48	420	338	349	338
query49	1206	408	412	408
query50	826	424	415	415
query51	7254	7115	7106	7106
query52	101	91	95	91
query53	262	186	191	186
query54	1275	473	477	473
query55	80	85	79	79
query56	281	257	251	251
query57	1281	1197	1164	1164
query58	255	217	217	217
query59	3243	2954	2859	2859
query60	280	246	252	246
query61	113	113	176	113
query62	894	676	688	676
query63	221	194	195	194
query64	5368	645	643	643
query65	3296	3188	3202	3188
query66	1450	322	337	322
query67	15848	15480	15449	15449
query68	4889	579	575	575
query69	441	267	264	264
query70	1203	1085	1103	1085
query71	342	263	256	256
query72	6429	4096	4010	4010
query73	745	348	366	348
query74	10265	8992	8952	8952
query75	3381	2655	2666	2655
query76	2830	988	1132	988
query77	390	284	272	272
query78	10387	9711	9529	9529
query79	1465	593	604	593
query80	1153	433	430	430
query81	525	221	227	221
query82	921	85	87	85
query83	224	159	143	143
query84	234	79	77	77
query85	1295	305	297	297
query86	359	292	294	292
query87	4326	4249	4239	4239
query88	3627	2376	2330	2330
query89	422	294	298	294
query90	1992	186	186	186
query91	143	121	109	109
query92	59	53	56	53
query93	1113	569	549	549
query94	946	297	291	291
query95	358	262	260	260
query96	617	293	285	285
query97	3291	3204	3123	3123
query98	206	204	201	201
query99	1526	1319	1272	1272
Total cold run time: 300499 ms
Total hot run time: 189392 ms

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

ClickBench: Total hot run time: 29.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 67e8fc8b258c6235e15fe62f96e05096481ccd51, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.52	0.50	0.51
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.55	0.50	0.50
query10	0.55	0.54	0.54
query11	0.14	0.10	0.11
query12	0.13	0.11	0.11
query13	0.61	0.60	0.60
query14	0.80	0.78	0.79
query15	0.84	0.83	0.83
query16	0.37	0.38	0.39
query17	1.03	1.03	1.04
query18	0.24	0.23	0.22
query19	1.89	1.85	1.80
query20	0.01	0.02	0.01
query21	15.44	0.57	0.58
query22	2.53	1.54	1.79
query23	16.85	0.99	0.80
query24	3.59	1.91	0.87
query25	0.25	0.12	0.05
query26	0.59	0.14	0.14
query27	0.05	0.04	0.05
query28	9.50	0.53	0.46
query29	12.60	3.23	3.25
query30	0.25	0.06	0.05
query31	2.85	0.37	0.39
query32	3.26	0.46	0.45
query33	2.98	2.97	3.01
query34	16.92	4.44	4.47
query35	4.54	4.55	4.55
query36	0.67	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 104.15 s
Total hot run time: 29.69 s

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 2, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 2, 2025

BE UT Coverage Report

Increment line coverage 77.63% (59/76) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.13% (12407/27491)
Line Coverage 36.06% (109729/304268)
Region Coverage 35.22% (56990/161823)
Branch Coverage 32.31% (30887/95606)

@morrySnow morrySnow changed the title [feature](inverted index) Support the IS NOT NULL condition in expr via inverted indexes (#50748) branch-3.1: [feature](inverted index) Support the IS NOT NULL condition in expr via inverted indexes #50748 Jul 3, 2025
@morrySnow morrySnow merged commit 16014b8 into apache:branch-3.1 Jul 3, 2025
18 of 21 checks passed
etah000 pushed a commit to etah000/doris that referenced this pull request Jul 7, 2025
@csun5285 @etah000
branch-3.1: [feature](inverted index) Support the IS NOT NULL conditi…
a81e38d 
…on in expr via inverted indexes apache#50748 (apache#52644)

pick from master apache#50748
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@csun5285 @hello-stephen @doris-robot @morrySnow