Skip to content

[External](parquet) pass non predicates column's offset index to RowGroupReader #55795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Oct 23, 2025
Merged

[External](parquet) pass non predicates column's offset index to RowGroupReader #55795

morningman merged 1 commit into from
Oct 23, 2025

Conversation

@liutang123
Copy link
Contributor

@liutang123 liutang123 commented Sep 9, 2025 •
edited by morningman
Loading

What problem does this PR solve?

Now, Parquet reader don't pass non predicate coloumn's offset index to RowGroupReader.
create_page_reader will create PageReader rather than PageReaderWithOffsetIndex.
The PageReader will parse each page header when skip_page and next_page_header. In the case of merging io, read and enlarge severely.

So, in this PR, we pass the offset index to RowGroupReader.

select * from a.b.c t1 where t1.dt>='2023-07-01' and t1.dt<='2024-07-01' and t1.user_id = 2

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liutang123
Copy link
Contributor Author

liutang123 commented Sep 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 9, 2025

TPC-H: Total hot run time: 34644 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5317515b16d555e5192e08cb9e5662a454ca1355, data reload: false

------ Round 1 ----------------------------------
q1	17591	5221	5049	5049
q2	2016	317	212	212
q3	10248	1272	697	697
q4	10226	1036	529	529
q5	7553	2479	2332	2332
q6	186	170	135	135
q7	976	755	626	626
q8	9354	1334	1114	1114
q9	7038	5097	5082	5082
q10	6946	2418	1983	1983
q11	524	303	286	286
q12	358	361	224	224
q13	17794	3663	3049	3049
q14	257	252	229	229
q15	573	491	489	489
q16	1020	1006	965	965
q17	587	871	379	379
q18	7361	7135	7107	7107
q19	1399	935	537	537
q20	343	349	232	232
q21	3895	2621	2420	2420
q22	1056	1017	968	968
Total cold run time: 107301 ms
Total hot run time: 34644 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5237	5302	5135	5135
q2	254	334	228	228
q3	2208	2690	2292	2292
q4	1359	1806	1294	1294
q5	4227	4564	4550	4550
q6	224	171	134	134
q7	2065	2069	1864	1864
q8	2647	2632	2789	2632
q9	7457	7258	7399	7258
q10	3081	3394	2843	2843
q11	554	534	490	490
q12	681	775	669	669
q13	3726	3853	3398	3398
q14	289	308	274	274
q15	548	479	472	472
q16	1070	1099	1087	1087
q17	1191	1598	1420	1420
q18	8032	7578	7613	7578
q19	824	723	774	723
q20	1935	1950	1805	1805
q21	4692	4407	4330	4330
q22	1084	1067	1015	1015
Total cold run time: 53385 ms
Total hot run time: 51491 ms

@doris-robot
Copy link

doris-robot commented Sep 9, 2025

TPC-DS: Total hot run time: 189549 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5317515b16d555e5192e08cb9e5662a454ca1355, data reload: false

query1	1048	439	425	425
query2	6566	1689	1697	1689
query3	6749	225	242	225
query4	26419	23541	22988	22988
query5	4432	657	522	522
query6	347	252	238	238
query7	4673	535	309	309
query8	306	260	246	246
query9	8643	2898	2904	2898
query10	487	348	299	299
query11	15991	15300	15005	15005
query12	184	125	129	125
query13	1690	555	443	443
query14	11127	9305	9247	9247
query15	218	202	178	178
query16	7708	659	492	492
query17	1471	805	639	639
query18	2083	448	338	338
query19	267	200	177	177
query20	133	128	124	124
query21	213	137	117	117
query22	4123	4105	3968	3968
query23	33949	32962	33089	32962
query24	8112	2365	2399	2365
query25	574	523	442	442
query26	1235	280	170	170
query27	2687	503	368	368
query28	4364	2240	2233	2233
query29	811	617	536	536
query30	296	220	200	200
query31	917	830	730	730
query32	93	84	85	84
query33	577	396	355	355
query34	795	835	531	531
query35	818	819	785	785
query36	1023	1026	896	896
query37	133	114	95	95
query38	3656	3530	3476	3476
query39	1496	1435	1429	1429
query40	229	136	138	136
query41	69	64	60	60
query42	127	119	118	118
query43	491	501	461	461
query44	1340	848	858	848
query45	185	186	171	171
query46	848	1014	650	650
query47	1777	1788	1763	1763
query48	405	431	328	328
query49	739	514	405	405
query50	675	683	408	408
query51	3908	4040	3927	3927
query52	116	118	110	110
query53	244	267	200	200
query54	611	613	553	553
query55	97	96	93	93
query56	348	352	352	352
query57	1194	1195	1135	1135
query58	290	280	281	280
query59	2519	2564	2518	2518
query60	364	351	371	351
query61	164	159	161	159
query62	839	715	683	683
query63	236	210	198	198
query64	4440	1140	839	839
query65	4059	3989	4016	3989
query66	1097	495	365	365
query67	15360	15303	15100	15100
query68	8762	942	588	588
query69	498	341	301	301
query70	1368	1265	1274	1265
query71	582	353	377	353
query72	5989	5023	5053	5023
query73	767	652	366	366
query74	8920	9090	9020	9020
query75	4262	3239	2834	2834
query76	3696	1222	758	758
query77	800	412	346	346
query78	9549	9599	8852	8852
query79	2655	887	612	612
query80	708	604	537	537
query81	478	264	223	223
query82	457	174	140	140
query83	305	266	264	264
query84	309	114	102	102
query85	897	472	497	472
query86	345	324	308	308
query87	3727	3736	3673	3673
query88	3020	2251	2222	2222
query89	413	336	298	298
query90	1953	239	233	233
query91	162	169	155	155
query92	94	79	74	74
query93	1249	995	655	655
query94	700	426	321	321
query95	419	332	328	328
query96	489	581	284	284
query97	2960	3012	2854	2854
query98	250	228	222	222
query99	1433	1408	1317	1317
Total cold run time: 277401 ms
Total hot run time: 189549 ms

@doris-robot
Copy link

doris-robot commented Sep 9, 2025

ClickBench: Total hot run time: 29.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5317515b16d555e5192e08cb9e5662a454ca1355, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.06
query3	0.25	0.09	0.08
query4	1.60	0.11	0.12
query5	0.28	0.27	0.26
query6	1.16	0.66	0.63
query7	0.03	0.02	0.02
query8	0.05	0.05	0.05
query9	0.60	0.53	0.52
query10	0.59	0.58	0.57
query11	0.17	0.11	0.11
query12	0.16	0.12	0.13
query13	0.64	0.63	0.62
query14	1.02	1.02	1.03
query15	0.86	0.88	0.86
query16	0.41	0.39	0.40
query17	1.06	1.05	1.06
query18	0.22	0.22	0.20
query19	1.94	1.75	1.77
query20	0.01	0.01	0.01
query21	15.39	0.95	0.59
query22	0.76	1.11	0.68
query23	15.01	1.41	0.61
query24	6.52	2.48	0.63
query25	0.43	0.21	0.15
query26	0.63	0.16	0.14
query27	0.07	0.05	0.06
query28	9.80	0.93	0.45
query29	12.56	3.98	3.28
query30	0.30	0.14	0.12
query31	2.83	0.61	0.38
query32	3.25	0.56	0.48
query33	3.10	3.11	3.09
query34	15.99	5.47	4.84
query35	4.86	4.90	4.96
query36	0.70	0.50	0.49
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.03	0.03	0.04
query40	0.18	0.15	0.13
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 103.94 s
Total hot run time: 29.71 s

@liutang123
Copy link
Contributor Author

liutang123 commented Sep 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 9, 2025

TPC-H: Total hot run time: 34512 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 960e2ca8b0b79dc13c257785f41ced9d8b32262f, data reload: false

------ Round 1 ----------------------------------
q1	17595	5176	5042	5042
q2	1974	325	205	205
q3	10279	1270	703	703
q4	10226	1015	529	529
q5	7533	2451	2274	2274
q6	183	166	138	138
q7	928	774	660	660
q8	9364	1335	1202	1202
q9	6918	5116	5095	5095
q10	6949	2408	1945	1945
q11	501	305	275	275
q12	353	356	230	230
q13	17792	3639	3032	3032
q14	225	242	217	217
q15	579	516	497	497
q16	993	1006	956	956
q17	605	847	352	352
q18	7483	7119	7079	7079
q19	1522	944	569	569
q20	338	334	229	229
q21	3739	3209	2315	2315
q22	1047	1058	968	968
Total cold run time: 107126 ms
Total hot run time: 34512 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5209	5060	5154	5060
q2	244	332	227	227
q3	2163	2640	2270	2270
q4	1350	1748	1369	1369
q5	4188	4497	4401	4401
q6	229	176	141	141
q7	2063	1970	1897	1897
q8	2688	2582	2559	2559
q9	7470	7297	7288	7288
q10	3109	3372	2923	2923
q11	575	522	501	501
q12	746	795	874	795
q13	3426	3852	3329	3329
q14	305	300	269	269
q15	526	506	485	485
q16	1066	1111	1055	1055
q17	1185	1401	1384	1384
q18	7803	7867	7585	7585
q19	825	853	975	853
q20	2040	2104	1912	1912
q21	4612	4414	4162	4162
q22	1134	1019	994	994
Total cold run time: 52956 ms
Total hot run time: 51459 ms

@doris-robot
Copy link

doris-robot commented Sep 9, 2025

TPC-DS: Total hot run time: 188475 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 960e2ca8b0b79dc13c257785f41ced9d8b32262f, data reload: false

query1	1083	436	420	420
query2	6549	1700	1665	1665
query3	6760	227	227	227
query4	26334	23843	23049	23049
query5	4622	665	531	531
query6	354	261	233	233
query7	4682	528	302	302
query8	327	267	255	255
query9	8651	2913	2880	2880
query10	501	372	309	309
query11	15907	15271	14656	14656
query12	195	127	118	118
query13	1700	588	454	454
query14	11741	9192	9196	9192
query15	227	193	177	177
query16	7705	640	477	477
query17	1357	738	634	634
query18	2048	420	334	334
query19	197	192	186	186
query20	135	128	125	125
query21	213	133	114	114
query22	4155	4355	3955	3955
query23	33963	33097	33017	33017
query24	7941	2356	2422	2356
query25	548	505	448	448
query26	734	280	164	164
query27	2661	509	350	350
query28	4293	2249	2210	2210
query29	735	601	492	492
query30	288	219	201	201
query31	891	800	730	730
query32	88	80	82	80
query33	563	395	347	347
query34	797	866	524	524
query35	779	862	733	733
query36	967	1017	893	893
query37	125	113	91	91
query38	3515	3491	3537	3491
query39	1481	1437	1462	1437
query40	231	143	126	126
query41	64	61	59	59
query42	136	120	121	120
query43	495	500	480	480
query44	1370	864	856	856
query45	180	177	171	171
query46	859	1027	660	660
query47	1797	1855	1788	1788
query48	388	432	331	331
query49	744	511	411	411
query50	645	687	401	401
query51	3853	3852	3882	3852
query52	121	116	108	108
query53	242	267	203	203
query54	617	602	538	538
query55	100	89	91	89
query56	345	330	305	305
query57	1189	1201	1136	1136
query58	298	289	278	278
query59	2597	2652	2551	2551
query60	357	363	339	339
query61	175	157	170	157
query62	822	733	666	666
query63	226	196	200	196
query64	3195	1157	815	815
query65	4050	3938	3968	3938
query66	885	432	361	361
query67	15337	15380	15152	15152
query68	7985	935	583	583
query69	508	343	298	298
query70	1361	1270	1290	1270
query71	581	345	361	345
query72	5876	5078	5250	5078
query73	755	698	362	362
query74	9001	8821	8823	8821
query75	4066	3208	2786	2786
query76	3569	1194	731	731
query77	799	402	333	333
query78	9768	9625	8899	8899
query79	2603	824	593	593
query80	716	573	519	519
query81	504	261	232	232
query82	497	163	133	133
query83	268	260	238	238
query84	264	108	93	93
query85	915	471	425	425
query86	398	316	315	315
query87	3695	3774	3638	3638
query88	3547	2177	2265	2177
query89	424	330	302	302
query90	1903	226	223	223
query91	166	169	135	135
query92	96	77	70	70
query93	1956	1036	638	638
query94	711	411	339	339
query95	418	333	327	327
query96	481	575	277	277
query97	2912	2983	2845	2845
query98	251	223	221	221
query99	1648	1396	1285	1285
Total cold run time: 275772 ms
Total hot run time: 188475 ms

@liutang123
Copy link
Contributor Author

liutang123 commented Sep 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 9, 2025

ClickBench: Total hot run time: 30.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 960e2ca8b0b79dc13c257785f41ced9d8b32262f, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.06	0.05
query3	0.25	0.08	0.08
query4	1.60	0.12	0.12
query5	0.28	0.27	0.27
query6	1.18	0.65	0.64
query7	0.04	0.03	0.03
query8	0.05	0.04	0.04
query9	0.61	0.53	0.53
query10	0.58	0.59	0.58
query11	0.17	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.62	0.63
query14	1.04	1.01	1.01
query15	0.86	0.86	0.87
query16	0.41	0.39	0.39
query17	1.05	1.08	1.08
query18	0.22	0.19	0.20
query19	1.88	1.86	1.87
query20	0.02	0.01	0.02
query21	15.42	1.03	0.59
query22	0.78	1.15	0.72
query23	14.87	1.40	0.59
query24	6.60	1.67	1.36
query25	0.48	0.13	0.15
query26	0.52	0.16	0.14
query27	0.07	0.06	0.05
query28	10.36	0.95	0.43
query29	12.56	3.88	3.28
query30	0.28	0.13	0.11
query31	2.84	0.57	0.39
query32	3.24	0.57	0.48
query33	3.10	3.06	3.04
query34	16.32	5.52	4.82
query35	4.91	4.91	4.95
query36	0.70	0.51	0.50
query37	0.10	0.08	0.07
query38	0.07	0.05	0.04
query39	0.04	0.02	0.02
query40	0.18	0.15	0.16
query41	0.08	0.04	0.03
query42	0.04	0.04	0.02
query43	0.05	0.04	0.03
Total cold run time: 104.77 s
Total hot run time: 30.54 s

@doris-robot
Copy link

doris-robot commented Sep 9, 2025

TPC-H: Total hot run time: 34669 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2d47b96bbe2363c1b21fe62947e5924a29b6ef71, data reload: false

------ Round 1 ----------------------------------
q1	17630	5198	5070	5070
q2	2023	308	226	226
q3	10238	1295	719	719
q4	10224	1032	521	521
q5	7545	2459	2312	2312
q6	185	172	142	142
q7	947	771	667	667
q8	9356	1401	1062	1062
q9	6892	5093	5179	5093
q10	6899	2396	1975	1975
q11	499	308	281	281
q12	354	382	231	231
q13	17786	3631	3034	3034
q14	242	237	215	215
q15	549	507	497	497
q16	1035	1008	932	932
q17	611	867	361	361
q18	7597	7192	7185	7185
q19	1227	935	590	590
q20	367	344	235	235
q21	3943	2555	2328	2328
q22	1079	1040	993	993
Total cold run time: 107228 ms
Total hot run time: 34669 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5121	5088	5098	5088
q2	254	330	233	233
q3	2148	2706	2274	2274
q4	1343	1773	1327	1327
q5	4229	4119	4153	4119
q6	206	166	128	128
q7	1929	1821	1686	1686
q8	2509	2431	2348	2348
q9	6840	6776	6724	6724
q10	2959	3178	2709	2709
q11	577	509	496	496
q12	654	751	596	596
q13	3298	3719	3013	3013
q14	269	294	271	271
q15	519	476	489	476
q16	1046	1064	1005	1005
q17	1108	1404	1356	1356
q18	7335	7031	6995	6995
q19	817	816	837	816
q20	1931	1951	1826	1826
q21	4851	4310	4411	4310
q22	1112	1063	1000	1000
Total cold run time: 51055 ms
Total hot run time: 48796 ms

@doris-robot
Copy link

doris-robot commented Sep 9, 2025

TPC-DS: Total hot run time: 188775 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2d47b96bbe2363c1b21fe62947e5924a29b6ef71, data reload: false

query1	1076	439	439	439
query2	6550	1688	1677	1677
query3	6749	230	233	230
query4	26465	23038	23170	23038
query5	4377	661	526	526
query6	346	251	238	238
query7	4660	541	310	310
query8	331	270	265	265
query9	8676	2900	2935	2900
query10	479	360	327	327
query11	15935	14942	14660	14660
query12	168	119	118	118
query13	1666	567	433	433
query14	10599	9144	9172	9144
query15	252	200	177	177
query16	7279	637	487	487
query17	1216	740	648	648
query18	1993	428	335	335
query19	206	202	178	178
query20	135	132	126	126
query21	220	131	115	115
query22	4049	4208	3984	3984
query23	33829	32977	32982	32977
query24	8173	2415	2441	2415
query25	586	530	460	460
query26	1241	280	202	202
query27	2705	516	363	363
query28	4424	2236	2222	2222
query29	851	611	506	506
query30	297	221	200	200
query31	883	821	728	728
query32	94	79	93	79
query33	572	392	350	350
query34	820	855	532	532
query35	839	812	753	753
query36	961	1035	934	934
query37	130	115	95	95
query38	3534	3521	3473	3473
query39	1503	1433	1410	1410
query40	228	139	126	126
query41	67	62	66	62
query42	133	120	120	120
query43	516	473	468	468
query44	1411	884	961	884
query45	185	185	173	173
query46	852	1016	662	662
query47	1789	1817	1707	1707
query48	407	435	343	343
query49	739	508	403	403
query50	681	690	408	408
query51	3937	3959	3834	3834
query52	122	124	113	113
query53	258	279	203	203
query54	619	616	551	551
query55	98	95	89	89
query56	356	339	330	330
query57	1199	1175	1108	1108
query58	299	282	303	282
query59	2612	2628	2604	2604
query60	371	360	345	345
query61	170	152	163	152
query62	843	762	685	685
query63	236	197	206	197
query64	4525	1136	822	822
query65	4018	3934	3967	3934
query66	1189	436	360	360
query67	15637	15294	15074	15074
query68	4727	952	663	663
query69	523	338	293	293
query70	1366	1294	1337	1294
query71	535	352	334	334
query72	5889	5085	5483	5085
query73	700	679	376	376
query74	8911	8990	8755	8755
query75	3281	3267	2681	2681
query76	3170	1164	749	749
query77	488	413	340	340
query78	9508	9740	8860	8860
query79	1009	938	602	602
query80	723	594	530	530
query81	492	254	227	227
query82	228	169	143	143
query83	276	276	275	275
query84	266	122	100	100
query85	906	467	430	430
query86	342	312	325	312
query87	3748	3741	3626	3626
query88	2978	2317	2318	2317
query89	397	337	288	288
query90	1767	233	226	226
query91	158	165	140	140
query92	85	83	72	72
query93	1156	1019	654	654
query94	634	433	320	320
query95	409	347	334	334
query96	501	603	292	292
query97	2880	3023	2849	2849
query98	270	222	220	220
query99	1344	1410	1295	1295
Total cold run time: 268033 ms
Total hot run time: 188775 ms

@doris-robot
Copy link

doris-robot commented Sep 9, 2025

ClickBench: Total hot run time: 30.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2d47b96bbe2363c1b21fe62947e5924a29b6ef71, data reload: false

query1	0.05	0.05	0.05
query2	0.11	0.06	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.12
query5	0.28	0.27	0.25
query6	1.17	0.65	0.63
query7	0.03	0.02	0.02
query8	0.05	0.05	0.05
query9	0.64	0.54	0.54
query10	0.57	0.57	0.57
query11	0.17	0.12	0.11
query12	0.15	0.11	0.12
query13	0.64	0.64	0.61
query14	1.00	1.05	1.06
query15	0.86	0.84	0.86
query16	0.41	0.40	0.39
query17	1.06	1.03	1.06
query18	0.21	0.20	0.20
query19	1.97	1.85	1.88
query20	0.02	0.01	0.02
query21	15.40	0.96	0.58
query22	0.75	1.15	0.96
query23	14.75	1.40	0.61
query24	5.91	0.99	2.12
query25	0.51	0.27	0.11
query26	0.60	0.16	0.14
query27	0.07	0.06	0.06
query28	10.18	0.98	0.42
query29	12.55	3.92	3.23
query30	0.28	0.14	0.11
query31	2.83	0.60	0.39
query32	3.24	0.56	0.49
query33	3.10	3.16	3.10
query34	16.22	5.47	4.83
query35	4.89	4.96	4.92
query36	0.71	0.52	0.50
query37	0.11	0.08	0.07
query38	0.06	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 103.8 s
Total hot run time: 30.38 s

@liutang123
Copy link
Contributor Author

liutang123 commented Sep 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 10, 2025

TPC-H: Total hot run time: 34756 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e39ffd94b72ac8a78964296826c427807b1dccd, data reload: false

------ Round 1 ----------------------------------
q1	17600	5217	4983	4983
q2	1983	321	242	242
q3	10225	1305	723	723
q4	10217	1028	530	530
q5	7575	2464	2325	2325
q6	185	169	136	136
q7	915	759	638	638
q8	9350	1326	1077	1077
q9	7037	5121	5205	5121
q10	6907	2383	1996	1996
q11	499	295	280	280
q12	352	363	235	235
q13	17764	3639	3074	3074
q14	252	253	220	220
q15	569	498	492	492
q16	1010	1020	944	944
q17	598	867	363	363
q18	7563	7211	7316	7211
q19	1102	963	561	561
q20	335	344	231	231
q21	3753	2553	2364	2364
q22	1082	1046	1010	1010
Total cold run time: 106873 ms
Total hot run time: 34756 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5090	5100	5126	5100
q2	242	325	223	223
q3	2154	2702	2307	2307
q4	1352	1789	1351	1351
q5	4253	4564	4581	4564
q6	231	171	138	138
q7	2045	2009	1888	1888
q8	2636	2576	2642	2576
q9	7566	7313	7495	7313
q10	3137	3231	2912	2912
q11	570	516	497	497
q12	715	788	627	627
q13	3530	3958	3306	3306
q14	293	322	315	315
q15	556	496	494	494
q16	1070	1143	1068	1068
q17	1193	1517	1420	1420
q18	7995	7790	7457	7457
q19	858	866	871	866
q20	2052	2083	1939	1939
q21	5008	4452	4317	4317
q22	1105	1070	1004	1004
Total cold run time: 53651 ms
Total hot run time: 51682 ms

@doris-robot
Copy link

doris-robot commented Sep 10, 2025

TPC-DS: Total hot run time: 189533 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2e39ffd94b72ac8a78964296826c427807b1dccd, data reload: false

query1	1053	453	402	402
query2	6569	1714	1709	1709
query3	6762	226	246	226
query4	26730	23303	22937	22937
query5	4366	630	500	500
query6	338	261	241	241
query7	4685	501	307	307
query8	312	281	278	278
query9	8648	2972	2906	2906
query10	496	337	313	313
query11	15905	15065	14937	14937
query12	178	125	122	122
query13	1664	547	431	431
query14	11282	9343	9244	9244
query15	209	198	172	172
query16	7680	673	481	481
query17	1223	751	631	631
query18	2036	442	373	373
query19	200	194	176	176
query20	137	132	124	124
query21	212	141	118	118
query22	4055	4240	4116	4116
query23	34218	33045	32913	32913
query24	8203	2389	2374	2374
query25	582	526	444	444
query26	1244	287	169	169
query27	2737	510	357	357
query28	4395	2282	2255	2255
query29	789	616	498	498
query30	298	234	202	202
query31	919	818	719	719
query32	92	81	81	81
query33	580	389	352	352
query34	812	865	547	547
query35	864	830	760	760
query36	972	1033	949	949
query37	135	120	101	101
query38	3543	3573	3601	3573
query39	1516	1413	1438	1413
query40	232	150	136	136
query41	73	69	70	69
query42	134	125	120	120
query43	511	502	483	483
query44	1356	872	912	872
query45	181	184	184	184
query46	854	1020	667	667
query47	1745	1802	1758	1758
query48	394	445	320	320
query49	740	517	422	422
query50	683	698	414	414
query51	3895	3937	3916	3916
query52	122	121	109	109
query53	258	277	202	202
query54	606	614	541	541
query55	99	89	90	89
query56	365	340	328	328
query57	1233	1208	1134	1134
query58	302	293	285	285
query59	2608	2745	2592	2592
query60	365	348	358	348
query61	166	160	182	160
query62	822	720	690	690
query63	229	193	201	193
query64	4462	1140	821	821
query65	4051	3975	3973	3973
query66	1106	474	378	378
query67	15738	15383	15276	15276
query68	8016	964	594	594
query69	508	346	307	307
query70	1414	1207	1293	1207
query71	648	357	328	328
query72	6028	5038	5272	5038
query73	696	683	364	364
query74	9191	8869	8891	8869
query75	3974	3259	2814	2814
query76	3501	1151	744	744
query77	814	400	337	337
query78	9575	9773	8831	8831
query79	2885	862	592	592
query80	778	579	518	518
query81	507	262	226	226
query82	469	170	141	141
query83	263	260	257	257
query84	264	113	90	90
query85	892	465	427	427
query86	395	329	313	313
query87	3748	3719	3631	3631
query88	3838	2228	2230	2228
query89	403	333	301	301
query90	1866	229	226	226
query91	168	167	138	138
query92	91	79	77	77
query93	2433	1009	648	648
query94	696	422	327	327
query95	412	341	334	334
query96	479	589	281	281
query97	2987	2991	2889	2889
query98	261	221	270	221
query99	1653	1416	1304	1304
Total cold run time: 279822 ms
Total hot run time: 189533 ms

@doris-robot
Copy link

doris-robot commented Sep 10, 2025

ClickBench: Total hot run time: 30.5 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2e39ffd94b72ac8a78964296826c427807b1dccd, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.06	0.06
query3	0.25	0.08	0.08
query4	1.62	0.12	0.11
query5	0.27	0.27	0.26
query6	1.18	0.65	0.66
query7	0.04	0.03	0.03
query8	0.05	0.05	0.04
query9	0.64	0.53	0.54
query10	0.58	0.57	0.58
query11	0.16	0.12	0.13
query12	0.15	0.12	0.12
query13	0.63	0.63	0.62
query14	1.04	1.03	1.04
query15	0.89	0.87	0.87
query16	0.41	0.40	0.39
query17	1.05	1.08	1.06
query18	0.22	0.20	0.21
query19	1.93	1.86	1.86
query20	0.01	0.01	0.01
query21	15.41	0.98	0.57
query22	0.78	1.18	0.92
query23	14.74	1.42	0.65
query24	6.86	1.13	0.96
query25	0.57	0.29	0.07
query26	0.50	0.15	0.13
query27	0.06	0.05	0.05
query28	9.89	0.94	0.45
query29	12.56	4.03	3.36
query30	0.29	0.13	0.11
query31	2.83	0.61	0.39
query32	3.26	0.59	0.50
query33	3.12	3.08	3.08
query34	15.95	5.51	4.80
query35	4.93	4.95	4.92
query36	0.71	0.52	0.51
query37	0.11	0.07	0.07
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.16	0.15
query41	0.08	0.02	0.03
query42	0.04	0.04	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.28 s
Total hot run time: 30.5 s

@doris-robot
Copy link

doris-robot commented Sep 10, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 51.97% (17302/33295)
Line Coverage 37.30% (157642/422599)
Region Coverage 31.95% (120265/376415)
Branch Coverage 33.32% (52818/158505)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 10, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.77% (23136/32694)
Line Coverage 57.16% (241404/422345)
Region Coverage 52.51% (200504/381844)
Branch Coverage 54.18% (86371/159408)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 11, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 11, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 11, 2025

PR approved by anyone and no changes requested.

@morningman
Copy link
Contributor

morningman commented Oct 21, 2025

run buildall

kaka11chen
kaka11chen approved these changes Oct 21, 2025
View reviewed changes
Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

morningman
morningman approved these changes Oct 21, 2025
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

doris-robot commented Oct 21, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.60% (17898/34026)
Line Coverage 37.84% (162362/429024)
Region Coverage 32.26% (123793/383766)
Branch Coverage 33.67% (54247/161127)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 21, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.45% (23835/33359)
Line Coverage 57.82% (247904/428776)
Region Coverage 52.68% (204797/388724)
Branch Coverage 54.54% (88359/162022)

@morningman morningman merged commit 3c8cc78 into apache:master Oct 23, 2025
27 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 23, 2025
@liutang123
[External](parquet) pass non predicates column's offset index to RowG…
6ef3e99 
…roupReader (#55795)

### What problem does this PR solve?

Now, Parquet reader don't pass non predicate coloumn's offset index to
`RowGroupReader`.
`create_page_reader` will create `PageReader` rather than
`PageReaderWithOffsetIndex`.
The `PageReader` will parse each page header when `skip_page` and
`next_page_header`. In the case of merging io, read and enlarge
severely.

So, in this PR, we pass the offset index to `RowGroupReader`.

Co-authored-by: liutang123 <liulijia@gmail.com>
@github-actions github-actions bot mentioned this pull request Oct 23, 2025
Merged
yiguolei pushed a commit that referenced this pull request Oct 24, 2025
@github-actions @liutang123
branch-4.0: [External](parquet) pass non predicates column's offset i…
21a9e98 
…ndex to RowGroupReader #55795 (#57270)

Cherry-picked from #55795

Co-authored-by: Lijia Liu <liutang123@yeah.net>
Co-authored-by: liutang123 <liulijia@gmail.com>
dwdwqfwe pushed a commit to dwdwqfwe/doris that referenced this pull request Oct 24, 2025
@liutang123 @dwdwqfwe
[External](parquet) pass non predicates column's offset index to RowG…
4ac2b03 
…roupReader (apache#55795)

### What problem does this PR solve?

Now, Parquet reader don't pass non predicate coloumn's offset index to
`RowGroupReader`.
`create_page_reader` will create `PageReader` rather than
`PageReaderWithOffsetIndex`.
The `PageReader` will parse each page header when `skip_page` and
`next_page_header`. In the case of merging io, read and enlarge
severely.

So, in this PR, we pass the offset index to `RowGroupReader`.

Co-authored-by: liutang123 <liulijia@gmail.com>
hubgeter added a commit to hubgeter/doris that referenced this pull request Oct 28, 2025
@hubgeter
branch-3.1:[External](parquet) pass non predicates column's offset in…
bfd7c1d 
…dex to RowGroupReader (apache#55795)
@hubgeter hubgeter mentioned this pull request Oct 28, 2025
Merged
morrySnow pushed a commit that referenced this pull request Oct 30, 2025
@hubgeter
branch-3.1:[External](parquet) pass non predicates column's offset in…
92203ea 
…dex to RowGroupReader #55795 (#57409)

bp #55795
@yiguolei yiguolei mentioned this pull request Nov 5, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kaka11chen kaka11chen kaka11chen approved these changes

@morningman morningman morningman approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.x dev/3.1.3-merged dev/4.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@liutang123 @hello-stephen @doris-robot @morningman @kaka11chen @yiguolei