Skip to content

[fix](inverted index) fix query error #50860

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 1 commit into from
May 14, 2025
Merged

[fix](inverted index) fix query error #50860

airborne12 merged 1 commit into from
May 14, 2025

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented May 13, 2025

What problem does this PR solve?

‌When the IN predicate in an expression contains parameters generated through CAST operations, utilizing inverted indexes may produce erroneous query results.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented May 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
Copy link
Contributor Author

csun5285 commented May 13, 2025

run buildall

zzzxl1993
zzzxl1993 approved these changes May 13, 2025
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented May 13, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented May 13, 2025

TPC-H: Total hot run time: 33645 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 202528ba4979e1924177f59c0e1b338e16bf6283, data reload: false

------ Round 1 ----------------------------------
q1	26354	4995	5026	4995
q2	2084	264	189	189
q3	10414	1210	688	688
q4	10219	997	532	532
q5	7517	2387	2304	2304
q6	180	161	135	135
q7	912	744	618	618
q8	9338	1285	1065	1065
q9	6655	5022	5108	5022
q10	6818	2314	1917	1917
q11	480	292	277	277
q12	353	366	222	222
q13	17778	3685	3102	3102
q14	231	241	212	212
q15	521	472	492	472
q16	420	434	380	380
q17	604	858	373	373
q18	7637	7226	7029	7029
q19	1662	960	552	552
q20	330	335	218	218
q21	3905	3361	2392	2392
q22	1039	1023	951	951
Total cold run time: 115451 ms
Total hot run time: 33645 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5122	5106	5064	5064
q2	241	329	231	231
q3	2144	2651	2290	2290
q4	1328	1759	1355	1355
q5	4543	4405	4427	4405
q6	225	162	134	134
q7	2009	1908	1761	1761
q8	2562	2532	2503	2503
q9	7199	7197	6995	6995
q10	3026	3196	2739	2739
q11	576	504	505	504
q12	693	774	621	621
q13	3408	3845	3290	3290
q14	294	286	272	272
q15	517	484	486	484
q16	450	483	435	435
q17	1141	1532	1366	1366
q18	7700	7604	7475	7475
q19	785	802	931	802
q20	1955	2029	1928	1928
q21	5315	4750	4510	4510
q22	1060	1014	1009	1009
Total cold run time: 52293 ms
Total hot run time: 50173 ms

@doris-robot
Copy link

doris-robot commented May 13, 2025

TPC-DS: Total hot run time: 186791 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 202528ba4979e1924177f59c0e1b338e16bf6283, data reload: false

query1	993	464	504	464
query2	6556	1862	1886	1862
query3	6742	219	223	219
query4	26500	23380	23628	23380
query5	4314	624	468	468
query6	306	198	198	198
query7	4621	502	297	297
query8	304	265	248	248
query9	8570	2625	2662	2625
query10	456	324	277	277
query11	15265	15064	14849	14849
query12	156	113	113	113
query13	1646	520	397	397
query14	8716	6235	6174	6174
query15	205	192	177	177
query16	7095	657	454	454
query17	946	761	573	573
query18	1954	393	295	295
query19	190	179	157	157
query20	116	118	117	117
query21	217	119	103	103
query22	4045	4145	3995	3995
query23	33995	33103	33227	33103
query24	8517	2466	2401	2401
query25	537	451	411	411
query26	1246	267	153	153
query27	2779	498	343	343
query28	4334	2128	2112	2112
query29	758	573	446	446
query30	283	214	190	190
query31	924	878	743	743
query32	74	65	66	65
query33	559	398	319	319
query34	832	867	522	522
query35	793	797	720	720
query36	966	994	875	875
query37	116	102	77	77
query38	4157	4206	4125	4125
query39	1474	1408	1435	1408
query40	209	114	104	104
query41	59	57	56	56
query42	130	111	118	111
query43	508	514	477	477
query44	1324	839	822	822
query45	174	180	172	172
query46	865	1036	655	655
query47	1776	1781	1704	1704
query48	389	429	311	311
query49	780	527	444	444
query50	654	675	408	408
query51	4120	4092	4027	4027
query52	113	110	104	104
query53	234	256	184	184
query54	597	585	516	516
query55	89	80	86	80
query56	298	317	293	293
query57	1158	1142	1078	1078
query58	274	260	275	260
query59	2600	2708	2569	2569
query60	378	329	324	324
query61	157	149	146	146
query62	818	726	705	705
query63	237	195	220	195
query64	4372	1008	661	661
query65	4363	4210	4226	4210
query66	1147	419	311	311
query67	15860	15363	15317	15317
query68	7845	885	528	528
query69	499	312	271	271
query70	1173	1071	1135	1071
query71	436	328	353	328
query72	5601	4789	4817	4789
query73	679	660	360	360
query74	8887	9164	8872	8872
query75	3576	3202	2735	2735
query76	3347	1184	752	752
query77	776	381	292	292
query78	10075	10239	9339	9339
query79	2002	836	587	587
query80	584	528	499	499
query81	472	258	216	216
query82	432	130	104	104
query83	256	245	244	244
query84	292	115	84	84
query85	772	357	310	310
query86	329	301	310	301
query87	4555	4618	4340	4340
query88	3549	2300	2303	2300
query89	382	311	290	290
query90	1939	213	217	213
query91	140	144	110	110
query92	79	57	58	57
query93	1616	942	596	596
query94	707	423	313	313
query95	367	292	285	285
query96	500	580	288	288
query97	3128	3237	3129	3129
query98	245	215	207	207
query99	1330	1372	1282	1282
Total cold run time: 272510 ms
Total hot run time: 186791 ms

@doris-robot
Copy link

doris-robot commented May 13, 2025

ClickBench: Total hot run time: 28.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 202528ba4979e1924177f59c0e1b338e16bf6283, data reload: false

query1	0.04	0.05	0.03
query2	0.12	0.10	0.10
query3	0.25	0.20	0.20
query4	1.59	0.20	0.11
query5	0.58	0.54	0.55
query6	1.20	0.72	0.72
query7	0.03	0.01	0.02
query8	0.04	0.04	0.03
query9	0.58	0.52	0.52
query10	0.58	0.57	0.56
query11	0.15	0.11	0.10
query12	0.14	0.12	0.11
query13	0.62	0.61	0.60
query14	0.78	0.80	0.81
query15	0.88	0.85	0.88
query16	0.37	0.38	0.40
query17	1.02	1.04	1.04
query18	0.22	0.21	0.22
query19	1.95	1.87	1.76
query20	0.02	0.01	0.01
query21	15.43	0.91	0.56
query22	0.76	1.18	0.98
query23	14.69	1.35	0.64
query24	6.89	1.83	0.34
query25	0.33	0.16	0.11
query26	0.66	0.15	0.12
query27	0.05	0.05	0.04
query28	9.36	0.92	0.45
query29	12.59	3.95	3.28
query30	0.25	0.10	0.06
query31	2.83	0.60	0.37
query32	3.23	0.54	0.47
query33	3.08	3.07	3.02
query34	15.88	5.12	4.54
query35	4.50	4.50	4.56
query36	0.65	0.50	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 102.83 s
Total hot run time: 28.96 s

airborne12
airborne12 approved these changes May 13, 2025
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 13, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 13, 2025

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

doris-robot commented May 13, 2025

BE UT Coverage Report

Increment line coverage 50.00% (2/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.80% (14897/26698)
Line Coverage 44.59% (131814/295581)
Region Coverage 43.64% (66272/151851)
Branch Coverage 38.26% (33961/88758)

@hello-stephen
Copy link
Contributor

hello-stephen commented May 13, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (4/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.37% (20846/26264)
Line Coverage 72.56% (214431/295517)
Region Coverage 70.70% (126101/178356)
Branch Coverage 64.48% (65367/101368)

@airborne12 airborne12 merged commit f245077 into apache:master May 14, 2025
26 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request May 14, 2025
@csun5285
[fix](inverted index) fix query error (#50860)
7c58ba3 
‌When the IN predicate in an expression contains parameters generated
through CAST operations, utilizing inverted indexes may produce
erroneous query results.
@github-actions github-actions bot mentioned this pull request May 14, 2025
Merged
github-actions bot pushed a commit that referenced this pull request May 14, 2025
@csun5285
[fix](inverted index) fix query error (#50860)
56a70fb 
‌When the IN predicate in an expression contains parameters generated
through CAST operations, utilizing inverted indexes may produce
erroneous query results.
@github-actions github-actions bot mentioned this pull request May 14, 2025
Closed
dataroaring pushed a commit that referenced this pull request May 14, 2025
@github-actions @csun5285
branch-3.0: [fix](inverted index) fix query error #50860 (#50868)
a9abe89 
Cherry-picked from #50860

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
csun5285 added a commit to csun5285/doris that referenced this pull request May 14, 2025
@csun5285
[fix](inverted index) fix query error (apache#50860)
f21eb79 
‌When the IN predicate in an expression contains parameters generated
through CAST operations, utilizing inverted indexes may produce
erroneous query results.
@csun5285 csun5285 mentioned this pull request May 14, 2025
Merged
yiguolei pushed a commit that referenced this pull request May 17, 2025
@csun5285
[fix](inverted index) fix query error (#50860) (#50909)
505c9af 
pick from master #50860
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@csun5285 @koarz
[fix](inverted index) fix query error (apache#50860)
2d795ec 
‌When the IN predicate in an expression contains parameters generated
through CAST operations, utilizing inverted indexes may produce
erroneous query results.
@yiguolei yiguolei mentioned this pull request Aug 11, 2025
Open
csun5285 added a commit to csun5285/doris that referenced this pull request Sep 23, 2025
@csun5285
[fix](inverted index) fix query error (apache#50860) (apache#4006)
b7b90d5 
‌When the IN predicate in an expression contains parameters generated
through CAST operations, utilizing inverted indexes may produce
erroneous query results.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.6-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@csun5285 @Thearas @doris-robot @hello-stephen @airborne12 @zzzxl1993 @yiguolei @dataroaring @morrySnow