Skip to content

branch-3.1:[External](parquet) pass non predicates column's offset index to RowGroupReader #55795 #57409

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Oct 30, 2025
Merged

branch-3.1:[External](parquet) pass non predicates column's offset index to RowGroupReader #55795 #57409

morrySnow merged 1 commit into from
Oct 30, 2025

Conversation

@hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Oct 28, 2025 •
edited by morrySnow
Loading

bp #55795

@hubgeter hubgeter requested a review from morrySnow as a code owner October 28, 2025 07:11
@Thearas
Copy link
Contributor

Thearas commented Oct 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

hubgeter commented Oct 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 30, 2025

TPC-H: Total hot run time: 32802 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bfd7c1d23e4cbeab8a97c17fb72c93aa1c9f554a, data reload: false

------ Round 1 ----------------------------------
q1	17571	5519	5543	5519
q2	2057	290	160	160
q3	10435	1502	735	735
q4	10216	891	460	460
q5	7673	2405	2136	2136
q6	185	170	138	138
q7	909	751	616	616
q8	9339	1454	1202	1202
q9	5347	4980	4929	4929
q10	6790	2281	1812	1812
q11	453	283	257	257
q12	335	366	212	212
q13	17780	3642	3021	3021
q14	227	227	206	206
q15	508	457	455	455
q16	411	426	375	375
q17	584	872	373	373
q18	7217	6465	6453	6453
q19	1197	970	545	545
q20	329	348	206	206
q21	2938	2226	1968	1968
q22	1003	1039	1024	1024
Total cold run time: 103504 ms
Total hot run time: 32802 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5529	5529	5509	5509
q2	235	343	233	233
q3	2256	2641	2264	2264
q4	1342	1836	1421	1421
q5	4444	4886	5161	4886
q6	169	163	130	130
q7	2138	1948	1807	1807
q8	2628	2849	2725	2725
q9	7214	7220	7188	7188
q10	3017	3290	2806	2806
q11	589	503	497	497
q12	657	718	622	622
q13	3416	3738	3193	3193
q14	290	294	282	282
q15	526	473	466	466
q16	446	494	430	430
q17	1231	1763	1264	1264
q18	7691	7529	7323	7323
q19	784	1153	1103	1103
q20	1973	2019	1900	1900
q21	5328	4915	4670	4670
q22	1150	1098	1041	1041
Total cold run time: 53053 ms
Total hot run time: 51760 ms

@doris-robot
Copy link

doris-robot commented Oct 30, 2025

TPC-DS: Total hot run time: 191792 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bfd7c1d23e4cbeab8a97c17fb72c93aa1c9f554a, data reload: false

query1	1005	388	391	388
query2	6530	2057	2056	2056
query3	6707	216	221	216
query4	33720	23768	23477	23477
query5	4395	626	450	450
query6	279	192	186	186
query7	4632	496	318	318
query8	289	252	235	235
query9	9458	2620	2624	2620
query10	486	331	251	251
query11	18109	15468	15290	15290
query12	163	110	106	106
query13	1656	590	425	425
query14	9326	6785	7696	6785
query15	256	195	197	195
query16	8237	653	504	504
query17	1893	781	617	617
query18	2161	415	330	330
query19	221	206	170	170
query20	128	126	119	119
query21	212	135	117	117
query22	4385	4663	4525	4525
query23	35284	33724	33540	33540
query24	7246	2665	2663	2663
query25	534	490	434	434
query26	1203	282	169	169
query27	2000	468	339	339
query28	5201	2210	2158	2158
query29	773	587	475	475
query30	262	188	160	160
query31	960	886	803	803
query32	98	63	64	63
query33	540	375	315	315
query34	760	842	529	529
query35	762	803	724	724
query36	1001	1045	931	931
query37	106	93	71	71
query38	3949	3976	3845	3845
query39	1516	1449	1433	1433
query40	208	120	104	104
query41	52	52	48	48
query42	123	108	110	108
query43	542	518	504	504
query44	1379	818	837	818
query45	187	179	174	174
query46	865	1048	672	672
query47	1901	1927	1857	1857
query48	422	438	350	350
query49	811	483	399	399
query50	679	683	436	436
query51	7253	7240	7096	7096
query52	103	103	95	95
query53	237	252	184	184
query54	545	558	471	471
query55	80	80	87	80
query56	267	283	265	265
query57	1268	1253	1146	1146
query58	239	228	223	223
query59	3021	3190	3010	3010
query60	300	285	265	265
query61	119	119	116	116
query62	783	723	704	704
query63	236	205	189	189
query64	4582	995	646	646
query65	3298	3231	3219	3219
query66	1073	413	313	313
query67	16046	15791	15533	15533
query68	7930	826	542	542
query69	478	324	282	282
query70	1204	1113	1163	1113
query71	392	299	264	264
query72	5682	3991	3801	3801
query73	645	749	361	361
query74	10428	9473	9308	9308
query75	3486	3172	2718	2718
query76	3264	1184	797	797
query77	576	375	298	298
query78	10371	10482	9613	9613
query79	3449	906	610	610
query80	639	522	478	478
query81	525	254	226	226
query82	681	124	90	90
query83	165	161	145	145
query84	240	106	84	84
query85	809	371	316	316
query86	391	321	283	283
query87	4345	4317	4285	4285
query88	5139	2463	2430	2430
query89	406	332	294	294
query90	1940	191	192	191
query91	139	143	111	111
query92	68	58	53	53
query93	1936	899	550	550
query94	696	394	317	317
query95	337	292	260	260
query96	500	611	290	290
query97	3170	3273	3119	3119
query98	253	215	199	199
query99	1480	1448	1307	1307
Total cold run time: 294826 ms
Total hot run time: 191792 ms

@doris-robot
Copy link

doris-robot commented Oct 30, 2025

ClickBench: Total hot run time: 29.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bfd7c1d23e4cbeab8a97c17fb72c93aa1c9f554a, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.22	0.07	0.07
query4	1.64	0.11	0.10
query5	0.50	0.49	0.51
query6	1.15	0.74	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.50
query10	0.55	0.55	0.56
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	0.77	0.80	0.80
query15	0.84	0.84	0.83
query16	0.39	0.38	0.40
query17	1.05	1.04	1.04
query18	0.24	0.22	0.22
query19	2.00	1.86	1.82
query20	0.02	0.02	0.02
query21	15.37	0.93	0.55
query22	0.75	0.81	0.73
query23	15.04	1.41	0.56
query24	3.02	1.45	1.58
query25	0.16	0.09	0.09
query26	0.36	0.15	0.14
query27	0.04	0.05	0.04
query28	13.88	1.01	0.43
query29	12.59	3.99	3.32
query30	0.25	0.08	0.06
query31	2.81	0.58	0.39
query32	3.22	0.55	0.47
query33	3.00	3.06	3.05
query34	16.72	5.18	4.56
query35	4.60	4.56	4.61
query36	0.65	0.51	0.48
query37	0.08	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.94 s
Total hot run time: 29.68 s

@morrySnow morrySnow changed the title branch-3.1:[External](parquet) pass non predicates column's offset index to RowGroupReader (#55795) branch-3.1:[External](parquet) pass non predicates column's offset index to RowGroupReader #55795 Oct 30, 2025
@morrySnow morrySnow merged commit 92203ea into apache:branch-3.1 Oct 30, 2025
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hubgeter @Thearas @doris-robot @morningman @morrySnow