Skip to content

branch-3.1: [fix](count) fix wrong count push down logic (#56182) #56482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Sep 28, 2025
Merged

branch-3.1: [fix](count) fix wrong count push down logic (#56182) #56482

morningman merged 1 commit into from
Sep 28, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Sep 25, 2025

bp #56182

@morningman
[fix](count) fix wrong count push down logic (apache#56182)
2d60482 
Introduced from topn optimization.

When executing query like `select count(*) from tbl`, it will trigger
"count push down optimization".
which means it will send some "dummy" split to BE, each with a part of
row count number.
But due to the bug, BE will use the range offset info in these dummy
split to do the row group filter logic,
which is incorrect and will result in empty result because all row group
will be filtered.

This PR fix it, to not filter the row group if it is a dummy split.

How to reproduce:

1. find an iceberg table with file size at least 16MB
2. set file_split_size=4MB
3. select count(*) from table, it will return empty result
@morningman
Copy link
Contributor Author

morningman commented Sep 25, 2025

run buildall

@Thearas
Copy link
Contributor

Thearas commented Sep 25, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 25, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.33% (1235/1500)
Line Coverage 66.23% (22262/33611)
Region Coverage 67.69% (11164/16492)
Branch Coverage 57.24% (5896/10300)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 25, 2025

FE UT Coverage Report

Increment line coverage 0.00% (0/4) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Sep 25, 2025

TPC-H: Total hot run time: 32853 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2d6048253e13292b4cb4837ad12673c9522023e9, data reload: false

------ Round 1 ----------------------------------
q1	17608	5516	5503	5503
q2	2028	401	289	289
q3	12330	1262	749	749
q4	10243	889	453	453
q5	8496	2436	2177	2177
q6	185	169	131	131
q7	948	770	627	627
q8	9355	1485	1125	1125
q9	5312	4988	4938	4938
q10	6799	2292	1841	1841
q11	469	270	255	255
q12	335	355	210	210
q13	17772	3663	2997	2997
q14	238	224	211	211
q15	521	466	468	466
q16	430	427	369	369
q17	595	888	361	361
q18	7168	6436	6361	6361
q19	1480	978	548	548
q20	337	346	209	209
q21	2794	2168	2021	2021
q22	1055	1019	1012	1012
Total cold run time: 106498 ms
Total hot run time: 32853 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5581	5554	5588	5554
q2	241	327	232	232
q3	2299	2655	2302	2302
q4	1301	1772	1405	1405
q5	4456	5050	5017	5017
q6	180	169	136	136
q7	2088	1964	1806	1806
q8	2700	2916	2821	2821
q9	7290	7317	7258	7258
q10	3035	3265	2714	2714
q11	573	519	489	489
q12	650	773	617	617
q13	3445	3817	3195	3195
q14	295	300	266	266
q15	515	475	465	465
q16	435	480	461	461
q17	1246	1735	1282	1282
q18	7721	7568	7381	7381
q19	788	1143	1131	1131
q20	2068	2052	1882	1882
q21	5448	4950	4547	4547
q22	1091	1047	992	992
Total cold run time: 53446 ms
Total hot run time: 51953 ms

@doris-robot
Copy link

doris-robot commented Sep 25, 2025

TPC-DS: Total hot run time: 192469 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2d6048253e13292b4cb4837ad12673c9522023e9, data reload: false

query1	953	410	412	410
query2	6156	1938	1894	1894
query3	8686	199	200	199
query4	33579	23816	23620	23620
query5	3564	626	460	460
query6	291	214	189	189
query7	4201	494	309	309
query8	307	245	238	238
query9	9482	2603	2588	2588
query10	474	330	272	272
query11	18572	15554	15288	15288
query12	158	108	107	107
query13	1568	561	433	433
query14	10183	6622	7144	6622
query15	244	192	176	176
query16	8071	675	547	547
query17	1554	777	599	599
query18	2133	426	327	327
query19	230	195	184	184
query20	127	126	132	126
query21	211	132	117	117
query22	4509	4644	4315	4315
query23	34940	34270	33900	33900
query24	7657	2719	2728	2719
query25	561	504	445	445
query26	812	287	174	174
query27	2294	487	357	357
query28	5220	2275	2233	2233
query29	718	611	481	481
query30	264	194	162	162
query31	1000	959	858	858
query32	97	61	68	61
query33	499	386	362	362
query34	769	856	533	533
query35	811	816	770	770
query36	1015	1057	952	952
query37	108	93	70	70
query38	4022	4030	4043	4030
query39	1551	1476	1524	1476
query40	208	115	110	110
query41	48	51	49	49
query42	123	104	109	104
query43	506	539	492	492
query44	1415	841	843	841
query45	189	178	175	175
query46	898	1063	702	702
query47	2013	2016	1911	1911
query48	409	445	350	350
query49	744	511	411	411
query50	662	695	462	462
query51	7386	7349	7157	7157
query52	103	107	97	97
query53	238	265	204	204
query54	550	534	484	484
query55	82	84	78	78
query56	268	279	268	268
query57	1261	1263	1216	1216
query58	248	225	219	219
query59	3054	3167	3097	3097
query60	302	289	288	288
query61	119	130	116	116
query62	787	760	692	692
query63	235	195	192	192
query64	3811	1029	631	631
query65	3452	3360	3316	3316
query66	849	442	316	316
query67	16520	15719	15503	15503
query68	7860	828	550	550
query69	498	306	268	268
query70	1219	1117	1094	1094
query71	440	304	280	280
query72	5180	3789	3844	3789
query73	651	757	354	354
query74	10159	9142	9090	9090
query75	3786	3183	2669	2669
query76	3556	1176	778	778
query77	750	370	281	281
query78	10309	10366	9579	9579
query79	2547	888	604	604
query80	756	539	483	483
query81	485	259	218	218
query82	588	120	90	90
query83	171	160	144	144
query84	288	94	89	89
query85	746	364	306	306
query86	343	321	292	292
query87	4344	4347	4284	4284
query88	3476	2439	2406	2406
query89	404	338	306	306
query90	1949	194	192	192
query91	134	155	114	114
query92	64	56	52	52
query93	1090	869	543	543
query94	647	418	320	320
query95	355	293	277	277
query96	489	602	295	295
query97	3224	3259	3152	3152
query98	236	208	195	195
query99	1562	1400	1291	1291
Total cold run time: 292377 ms
Total hot run time: 192469 ms

@doris-robot
Copy link

doris-robot commented Sep 25, 2025

ClickBench: Total hot run time: 29.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2d6048253e13292b4cb4837ad12673c9522023e9, data reload: false

query1	0.04	0.03	0.02
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.65	0.09	0.09
query5	0.50	0.52	0.50
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.59	0.51	0.50
query10	0.56	0.56	0.56
query11	0.16	0.11	0.12
query12	0.14	0.13	0.13
query13	0.62	0.59	0.60
query14	0.77	0.80	0.82
query15	0.87	0.84	0.84
query16	0.38	0.37	0.39
query17	1.06	1.07	1.02
query18	0.20	0.19	0.20
query19	1.98	1.92	1.87
query20	0.02	0.02	0.02
query21	15.40	0.96	0.66
query22	0.75	0.77	0.68
query23	14.78	1.45	0.68
query24	2.26	0.36	0.23
query25	0.15	0.08	0.09
query26	0.28	0.19	0.18
query27	0.08	0.08	0.08
query28	13.39	1.31	0.56
query29	12.68	4.13	3.43
query30	0.26	0.08	0.06
query31	2.82	0.63	0.41
query32	3.22	0.57	0.48
query33	3.04	3.05	3.05
query34	16.35	5.25	4.53
query35	4.59	4.59	4.65
query36	0.63	0.50	0.48
query37	0.21	0.16	0.17
query38	0.17	0.16	0.16
query39	0.06	0.04	0.04
query40	0.17	0.14	0.13
query41	0.10	0.06	0.06
query42	0.07	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 102.56 s
Total hot run time: 29.3 s

@morningman morningman merged commit f19e845 into apache:branch-3.1 Sep 28, 2025
21 of 22 checks passed
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morningman @Thearas @hello-stephen @doris-robot @morrySnow