Skip to content

[Optimize](orc-reader) Optimize stripe footer multiple reads in orc reader. #51277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 11, 2025
Merged

[Optimize](orc-reader) Optimize stripe footer multiple reads in orc reader. #51277

dataroaring merged 1 commit into from
Jun 11, 2025

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented May 27, 2025

What problem does this PR solve?

Problem Summary:

Cherry-pick #51117

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@kaka11chen kaka11chen requested a review from dataroaring as a code owner May 27, 2025 03:48
@Thearas
Copy link
Contributor

Thearas commented May 27, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen
Copy link
Contributor Author

kaka11chen commented May 27, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 27, 2025

TPC-H: Total hot run time: 39747 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4bbd2cc6932705d6be3fb7d79c0f10fd1884dc96, data reload: false

------ Round 1 ----------------------------------
q1	17585	6738	6599	6599
q2	2061	170	160	160
q3	10652	1076	1181	1076
q4	10573	747	842	747
q5	7738	2809	2781	2781
q6	213	132	129	129
q7	988	612	599	599
q8	9343	1913	2012	1913
q9	6676	6351	6364	6351
q10	6995	2206	2265	2206
q11	466	256	254	254
q12	388	203	207	203
q13	17786	3043	2943	2943
q14	237	199	213	199
q15	493	462	458	458
q16	489	389	400	389
q17	1026	617	563	563
q18	7524	6980	6871	6871
q19	1377	1101	1038	1038
q20	480	202	197	197
q21	3939	3119	3145	3119
q22	1117	982	952	952
Total cold run time: 108146 ms
Total hot run time: 39747 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7028	6557	6552	6552
q2	326	241	225	225
q3	2868	2737	2847	2737
q4	2005	1811	1755	1755
q5	5764	5729	5735	5729
q6	211	126	127	126
q7	2213	1765	1846	1765
q8	3351	3569	3489	3489
q9	8906	8799	8906	8799
q10	3585	3530	3535	3530
q11	581	494	495	494
q12	815	587	634	587
q13	11125	3114	3227	3114
q14	294	263	271	263
q15	522	460	477	460
q16	473	467	444	444
q17	1853	1619	1604	1604
q18	8160	7944	7889	7889
q19	1707	1630	1537	1537
q20	2060	1821	1793	1793
q21	5574	5171	5340	5171
q22	1144	1058	1031	1031
Total cold run time: 70565 ms
Total hot run time: 59094 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented May 27, 2025

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.07% (10861/26443)
Line Coverage 31.87% (92705/290891)
Region Coverage 30.96% (47832/154486)
Branch Coverage 27.43% (24485/89268)

@doris-robot
Copy link

doris-robot commented May 27, 2025

TPC-DS: Total hot run time: 196787 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4bbd2cc6932705d6be3fb7d79c0f10fd1884dc96, data reload: false

query1	1271	905	921	905
query2	6312	1889	1860	1860
query3	10830	4347	4385	4347
query4	61754	29306	23537	23537
query5	5104	474	478	474
query6	415	180	190	180
query7	5450	313	304	304
query8	315	222	222	222
query9	8398	2570	2568	2568
query10	465	266	263	263
query11	17696	15092	15706	15092
query12	166	105	101	101
query13	1399	445	440	440
query14	11037	7579	6976	6976
query15	205	183	168	168
query16	7208	447	492	447
query17	1210	599	586	586
query18	1984	330	332	330
query19	218	170	171	170
query20	119	110	110	110
query21	220	105	99	99
query22	4598	4601	4579	4579
query23	34961	34304	34637	34304
query24	6291	2890	2890	2890
query25	560	442	441	441
query26	682	164	160	160
query27	2039	348	354	348
query28	4190	2170	2125	2125
query29	687	437	430	430
query30	238	161	157	157
query31	1011	817	802	802
query32	68	56	55	55
query33	418	292	271	271
query34	927	518	501	501
query35	861	744	751	744
query36	1062	940	960	940
query37	108	65	73	65
query38	4089	3917	3990	3917
query39	1577	1495	1480	1480
query40	201	100	99	99
query41	52	50	50	50
query42	118	102	105	102
query43	541	490	470	470
query44	1186	819	823	819
query45	198	172	166	166
query46	1168	738	730	730
query47	2008	1910	1918	1910
query48	475	398	386	386
query49	770	432	383	383
query50	851	418	414	414
query51	7312	7195	7248	7195
query52	104	92	89	89
query53	266	184	189	184
query54	568	462	466	462
query55	79	76	82	76
query56	259	229	253	229
query57	1294	1183	1181	1181
query58	222	206	212	206
query59	3200	2989	2896	2896
query60	286	248	271	248
query61	113	108	112	108
query62	789	681	670	670
query63	225	194	194	194
query64	1746	689	650	650
query65	3262	3192	3178	3178
query66	639	286	292	286
query67	15962	15528	15697	15528
query68	4196	590	569	569
query69	419	269	272	269
query70	1167	1100	1097	1097
query71	328	265	255	255
query72	6403	4017	3984	3984
query73	750	342	348	342
query74	10219	9179	8996	8996
query75	3378	2625	2688	2625
query76	1987	1078	1084	1078
query77	473	274	279	274
query78	10658	9652	9609	9609
query79	1238	585	592	585
query80	853	423	449	423
query81	512	220	220	220
query82	1289	89	88	88
query83	240	153	155	153
query84	293	85	77	77
query85	902	308	294	294
query86	323	281	306	281
query87	4502	4235	4273	4235
query88	3732	2380	2359	2359
query89	420	291	285	285
query90	1952	185	183	183
query91	181	173	152	152
query92	62	49	53	49
query93	1728	562	555	555
query94	707	297	285	285
query95	350	254	256	254
query96	605	277	283	277
query97	3288	3119	3172	3119
query98	214	196	210	196
query99	1832	1322	1326	1322
Total cold run time: 315238 ms
Total hot run time: 196787 ms

@doris-robot
Copy link

doris-robot commented May 27, 2025

ClickBench: Total hot run time: 29.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4bbd2cc6932705d6be3fb7d79c0f10fd1884dc96, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.10
query5	0.51	0.53	0.50
query6	1.12	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.51	0.50
query10	0.54	0.55	0.57
query11	0.14	0.11	0.13
query12	0.14	0.11	0.12
query13	0.60	0.59	0.59
query14	0.77	0.80	0.78
query15	0.84	0.82	0.81
query16	0.41	0.41	0.38
query17	1.07	1.07	1.06
query18	0.23	0.21	0.21
query19	1.85	1.85	1.87
query20	0.01	0.01	0.02
query21	15.41	0.60	0.59
query22	2.43	2.20	2.05
query23	17.07	0.98	0.87
query24	3.54	1.64	0.21
query25	0.19	0.06	0.12
query26	0.54	0.13	0.14
query27	0.05	0.04	0.04
query28	10.28	0.51	0.44
query29	12.62	3.23	3.22
query30	0.26	0.06	0.06
query31	2.86	0.38	0.39
query32	3.24	0.47	0.46
query33	2.97	2.97	3.02
query34	17.10	4.51	4.56
query35	4.54	4.55	4.46
query36	0.68	0.47	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.08 s
Total hot run time: 29.71 s

dataroaring
dataroaring approved these changes Jun 11, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 9e15686 into apache:branch-3.0 Jun 11, 2025
22 of 24 checks passed
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@kaka11chen @Thearas @doris-robot @hello-stephen @dataroaring @morningman @morrySnow