Skip to content

[fix & opt](orc) ORC-1525: Fix bad read in RleDecoderV2::readByte and Decompress zlib by libdeflate. #51775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Jun 18, 2025
Merged

[fix & opt](orc) ORC-1525: Fix bad read in RleDecoderV2::readByte and Decompress zlib by libdeflate. #51775

morningman merged 1 commit into from
Jun 18, 2025

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Jun 17, 2025 •
edited
Loading

What problem does this PR solve?

Problem Summary:

Release note

  1. Cherry-pick ORC-1525 to fix bad read in RleDecoderV2::readByte.

  2. Decompress zlib by libdeflate.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen
Copy link
Contributor Author

kaka11chen commented Jun 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 17, 2025

TPC-H: Total hot run time: 33823 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32c1459f0d03d8f119593dc4883803a25d922e90, data reload: false

------ Round 1 ----------------------------------
q1	17579	5136	4975	4975
q2	1939	270	169	169
q3	10346	1226	740	740
q4	10217	1003	511	511
q5	7634	2332	2296	2296
q6	177	159	127	127
q7	904	744	618	618
q8	9313	1280	1144	1144
q9	6845	5075	5060	5060
q10	6935	2370	1979	1979
q11	489	288	279	279
q12	360	347	211	211
q13	17778	3687	3064	3064
q14	236	231	219	219
q15	554	484	472	472
q16	427	428	366	366
q17	599	843	354	354
q18	7704	7204	7138	7138
q19	1762	982	545	545
q20	320	341	218	218
q21	3676	3194	2378	2378
q22	1023	1012	960	960
Total cold run time: 106817 ms
Total hot run time: 33823 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5175	5035	5270	5035
q2	240	323	223	223
q3	2110	2626	2261	2261
q4	1377	1750	1412	1412
q5	4194	4074	4340	4074
q6	213	171	124	124
q7	1997	1906	1769	1769
q8	2626	2589	2521	2521
q9	7217	7217	7293	7217
q10	3079	3224	2742	2742
q11	567	507	507	507
q12	672	764	606	606
q13	3505	3902	3353	3353
q14	288	305	277	277
q15	515	483	467	467
q16	461	476	444	444
q17	1133	1529	1389	1389
q18	7725	7537	7561	7537
q19	815	802	871	802
q20	1985	2105	1888	1888
q21	4939	4290	4377	4290
q22	1079	1042	1019	1019
Total cold run time: 51912 ms
Total hot run time: 49957 ms

@doris-robot
Copy link

doris-robot commented Jun 17, 2025

TPC-DS: Total hot run time: 192256 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32c1459f0d03d8f119593dc4883803a25d922e90, data reload: false

query1	1395	1026	983	983
query2	6277	1863	1872	1863
query3	10986	4466	4520	4466
query4	52973	25570	23033	23033
query5	5056	617	471	471
query6	354	208	193	193
query7	4866	519	292	292
query8	285	237	210	210
query9	5459	2634	2647	2634
query10	396	352	279	279
query11	15156	14979	14772	14772
query12	162	105	105	105
query13	1048	517	411	411
query14	10133	6331	6118	6118
query15	213	206	184	184
query16	7079	660	504	504
query17	1066	735	580	580
query18	1535	419	313	313
query19	202	196	166	166
query20	127	125	118	118
query21	206	123	106	106
query22	4494	4586	4303	4303
query23	34460	33572	33574	33572
query24	6585	2428	2435	2428
query25	458	465	405	405
query26	681	282	151	151
query27	2404	525	349	349
query28	3207	2160	2164	2160
query29	580	555	437	437
query30	276	236	190	190
query31	861	861	766	766
query32	72	66	60	60
query33	467	375	320	320
query34	780	860	537	537
query35	797	848	756	756
query36	938	996	897	897
query37	112	97	78	78
query38	4250	4284	4286	4284
query39	1511	1461	1470	1461
query40	221	128	146	128
query41	69	58	59	58
query42	130	120	106	106
query43	493	537	514	514
query44	1352	842	835	835
query45	180	177	178	177
query46	864	1044	655	655
query47	1860	1876	1738	1738
query48	402	435	354	354
query49	648	500	402	402
query50	645	737	422	422
query51	4216	4423	4285	4285
query52	109	116	95	95
query53	226	261	188	188
query54	589	572	499	499
query55	86	85	86	85
query56	313	316	297	297
query57	1213	1207	1158	1158
query58	273	279	262	262
query59	2752	2836	2755	2755
query60	324	327	310	310
query61	123	119	120	119
query62	733	702	661	661
query63	221	180	182	180
query64	1445	1064	681	681
query65	4288	4176	4179	4176
query66	710	395	300	300
query67	15881	15592	15529	15529
query68	5734	899	525	525
query69	534	304	266	266
query70	1131	1098	1117	1098
query71	466	324	311	311
query72	5946	4748	4717	4717
query73	1357	627	348	348
query74	9409	9062	8972	8972
query75	3207	3220	2713	2713
query76	3867	1189	756	756
query77	521	356	286	286
query78	10186	9989	9381	9381
query79	2814	843	579	579
query80	657	506	446	446
query81	494	265	223	223
query82	614	132	102	102
query83	321	245	244	244
query84	296	106	81	81
query85	767	342	318	318
query86	359	316	294	294
query87	4410	4378	4243	4243
query88	3410	2265	2250	2250
query89	404	321	277	277
query90	1632	210	207	207
query91	139	149	112	112
query92	69	61	55	55
query93	2424	923	573	573
query94	740	399	274	274
query95	370	295	291	291
query96	552	561	280	280
query97	2716	2750	2626	2626
query98	231	203	212	203
query99	1561	1392	1281	1281
Total cold run time: 295611 ms
Total hot run time: 192256 ms

@doris-robot
Copy link

doris-robot commented Jun 17, 2025

ClickBench: Total hot run time: 29.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 32c1459f0d03d8f119593dc4883803a25d922e90, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.07	0.06
query4	1.61	0.10	0.10
query5	0.42	0.44	0.42
query6	1.18	0.65	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.58	0.51	0.50
query10	0.58	0.57	0.58
query11	0.16	0.10	0.11
query12	0.15	0.11	0.11
query13	0.63	0.60	0.61
query14	0.80	0.82	0.81
query15	0.89	0.87	0.87
query16	0.39	0.39	0.42
query17	1.07	1.06	1.10
query18	0.23	0.21	0.20
query19	1.95	1.94	1.89
query20	0.02	0.01	0.01
query21	15.42	0.93	0.58
query22	0.76	1.12	0.89
query23	14.76	1.37	0.64
query24	6.79	2.19	0.51
query25	0.51	0.12	0.15
query26	0.59	0.17	0.15
query27	0.06	0.05	0.05
query28	10.26	0.89	0.44
query29	12.55	4.01	3.39
query30	0.25	0.10	0.07
query31	2.82	0.60	0.39
query32	3.24	0.55	0.46
query33	3.07	3.11	3.12
query34	16.02	5.38	4.79
query35	4.81	4.86	4.78
query36	0.70	0.50	0.48
query37	0.08	0.07	0.07
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.16 s
Total hot run time: 29.66 s

@kaka11chen kaka11chen force-pushed the cherry-pick-ORC-1525 branch from 32c1459 to 7c5c232 Compare June 17, 2025 08:00
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Jun 17, 2025

run buildall

@kaka11chen kaka11chen changed the title [fix](orc) ORC-1525: Fix bad read in RleDecoderV2::readByte. [fix & opt](orc) ORC-1525: Fix bad read in RleDecoderV2::readByte and Decompress zlib by libdeflate. Jun 17, 2025
@kaka11chen
[fix & opt](orc) ORC-1525: Fix bad read in RleDecoderV2::readByte and…
00dfbc1 
… Decompress zlib by libdeflate.

1.ORC-1525: Fix bad read in RleDecoderV2::readByte.
2.Decompress zlib by libdeflate.
@kaka11chen kaka11chen force-pushed the cherry-pick-ORC-1525 branch from 7c5c232 to 00dfbc1 Compare June 17, 2025 13:06
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Jun 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 17, 2025

TPC-H: Total hot run time: 33785 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 00dfbc1981a93102ad63551f88c9d73acbd2f644, data reload: false

------ Round 1 ----------------------------------
q1	17583	5107	4975	4975
q2	1946	273	169	169
q3	10313	1241	769	769
q4	10200	1010	506	506
q5	7541	2322	2362	2322
q6	181	165	134	134
q7	896	748	613	613
q8	9303	1292	1131	1131
q9	6718	5073	5003	5003
q10	6923	2387	1958	1958
q11	493	282	276	276
q12	340	357	221	221
q13	17748	3632	3032	3032
q14	223	236	215	215
q15	573	491	474	474
q16	429	430	375	375
q17	578	845	358	358
q18	7466	7258	7182	7182
q19	1614	963	566	566
q20	330	335	224	224
q21	3746	3134	2330	2330
q22	1018	1000	952	952
Total cold run time: 106162 ms
Total hot run time: 33785 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5192	5010	4980	4980
q2	235	316	230	230
q3	2178	2644	2293	2293
q4	1347	1774	1311	1311
q5	4160	4107	4280	4107
q6	211	170	134	134
q7	1992	1933	1766	1766
q8	2624	2662	2603	2603
q9	7164	7185	7235	7185
q10	3043	3251	2756	2756
q11	581	517	502	502
q12	657	765	628	628
q13	3541	3945	3325	3325
q14	295	292	275	275
q15	527	473	469	469
q16	459	510	466	466
q17	1144	1486	1408	1408
q18	7726	7543	7396	7396
q19	790	739	860	739
q20	1980	2051	1887	1887
q21	5057	4462	4523	4462
q22	1099	1024	1012	1012
Total cold run time: 52002 ms
Total hot run time: 49934 ms

@doris-robot
Copy link

doris-robot commented Jun 17, 2025

TPC-DS: Total hot run time: 192938 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 00dfbc1981a93102ad63551f88c9d73acbd2f644, data reload: false

query1	1391	1016	987	987
query2	6308	1874	1840	1840
query3	11017	4554	4503	4503
query4	53579	25763	23642	23642
query5	5345	519	474	474
query6	399	215	202	202
query7	5236	531	297	297
query8	315	230	220	220
query9	6948	2628	2653	2628
query10	450	332	256	256
query11	15232	15024	14868	14868
query12	164	115	115	115
query13	1253	525	429	429
query14	10206	6452	6585	6452
query15	198	187	187	187
query16	7020	646	449	449
query17	1052	792	573	573
query18	1582	422	335	335
query19	211	206	169	169
query20	135	118	118	118
query21	217	131	107	107
query22	4320	4335	4318	4318
query23	34379	33598	33538	33538
query24	6898	2417	2460	2417
query25	460	490	434	434
query26	653	270	154	154
query27	2269	502	354	354
query28	2990	2140	2137	2137
query29	611	576	441	441
query30	285	230	196	196
query31	893	852	804	804
query32	73	66	61	61
query33	462	375	359	359
query34	787	865	539	539
query35	799	828	754	754
query36	935	979	907	907
query37	126	108	84	84
query38	4287	4222	4153	4153
query39	1542	1474	1477	1474
query40	218	119	106	106
query41	61	57	56	56
query42	130	121	112	112
query43	510	509	485	485
query44	1355	879	841	841
query45	182	181	172	172
query46	858	1044	645	645
query47	1824	1857	1783	1783
query48	389	428	338	338
query49	667	490	399	399
query50	660	694	416	416
query51	4234	4280	4134	4134
query52	112	118	102	102
query53	224	261	182	182
query54	581	575	513	513
query55	85	92	82	82
query56	309	319	282	282
query57	1235	1236	1141	1141
query58	275	262	263	262
query59	2727	2750	2724	2724
query60	332	327	326	326
query61	130	125	133	125
query62	729	733	666	666
query63	235	193	193	193
query64	1509	1104	798	798
query65	4256	4189	4183	4183
query66	732	474	297	297
query67	15778	15594	15270	15270
query68	7046	890	518	518
query69	550	299	272	272
query70	1185	1082	1075	1075
query71	497	330	296	296
query72	5933	4875	5002	4875
query73	1219	697	349	349
query74	9260	9111	8885	8885
query75	3873	3174	2691	2691
query76	4294	1196	753	753
query77	619	384	293	293
query78	10058	10136	9265	9265
query79	3303	832	568	568
query80	671	519	460	460
query81	509	257	226	226
query82	469	128	95	95
query83	366	246	232	232
query84	289	100	84	84
query85	818	348	324	324
query86	392	291	291	291
query87	4375	4413	4213	4213
query88	3365	2352	2298	2298
query89	425	327	285	285
query90	1968	211	209	209
query91	135	140	111	111
query92	74	60	59	59
query93	2109	961	573	573
query94	692	425	331	331
query95	368	295	298	295
query96	487	574	283	283
query97	2743	2779	2649	2649
query98	235	205	200	200
query99	1465	1415	1272	1272
Total cold run time: 301156 ms
Total hot run time: 192938 ms

@doris-robot
Copy link

doris-robot commented Jun 17, 2025

ClickBench: Total hot run time: 29.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 00dfbc1981a93102ad63551f88c9d73acbd2f644, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.07
query4	1.61	0.11	0.10
query5	0.42	0.41	0.41
query6	1.17	0.65	0.66
query7	0.03	0.02	0.02
query8	0.05	0.03	0.04
query9	0.57	0.51	0.51
query10	0.58	0.55	0.58
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.64	0.61	0.61
query14	0.81	0.85	0.80
query15	0.90	0.86	0.89
query16	0.39	0.39	0.39
query17	1.05	1.06	1.06
query18	0.23	0.21	0.22
query19	1.97	1.82	1.88
query20	0.01	0.01	0.01
query21	15.40	0.92	0.55
query22	0.75	1.22	0.70
query23	14.89	1.37	0.64
query24	6.97	1.13	1.13
query25	0.46	0.27	0.14
query26	0.70	0.16	0.14
query27	0.06	0.05	0.05
query28	9.93	0.93	0.45
query29	12.58	4.00	3.26
query30	0.26	0.10	0.06
query31	2.83	0.59	0.40
query32	3.26	0.55	0.46
query33	3.03	3.03	3.19
query34	16.10	5.46	4.82
query35	4.86	4.83	4.83
query36	0.68	0.51	0.48
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.15	0.14
query41	0.09	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.36 s
Total hot run time: 29.86 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 17, 2025

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.33% (15048/26713)
Line Coverage 45.10% (134580/298374)
Region Coverage 44.24% (67682/152984)
Branch Coverage 38.82% (34722/89440)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 17, 2025

BE Regression && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 61.24% (16100/26290)
Line Coverage 50.81% (151521/298216)
Region Coverage 48.13% (86618/179958)
Branch Coverage 41.63% (42568/102246)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 18, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 18, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 18, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit e190f1f into apache:master Jun 18, 2025
23 of 26 checks passed
@kaka11chen kaka11chen mentioned this pull request Jul 2, 2025
Merged
16 tasks
morrySnow pushed a commit that referenced this pull request Jul 3, 2025
@kaka11chen
branch-3.1: [fix & opt](orc) ORC-1525: Fix bad read in RleDecoderV2::…
9699556 
…readByte and Decompress zlib by libdeflate. #51775 (#52660)

Cherry-pick #51775
etah000 pushed a commit to etah000/doris that referenced this pull request Jul 7, 2025
@kaka11chen @etah000
branch-3.1: [fix & opt](orc) ORC-1525: Fix bad read in RleDecoderV2::…
dbf3216 
…readByte and Decompress zlib by libdeflate. apache#51775 (apache#52660)

Cherry-pick apache#51775
@kaka11chen kaka11chen mentioned this pull request Jul 10, 2025
Merged
16 tasks
yiguolei pushed a commit that referenced this pull request Jul 16, 2025
@kaka11chen
[fix & opt](orc) ORC-1525: Fix bad read in RleDecoderV2::readByte and…
d543713 
… Decompress zlib by libdeflate. (#53078)

### What problem does this PR solve?

Problem Summary:

### Release note

Cherry-pick #51775 

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Jul 21, 2025
@kaka11chen
[fix & opt](orc) ORC-1525: Fix bad read in RleDecoderV2::readByte and…
04d3c26 
… Decompress zlib by libdeflate.

Cherry-pick apache#51775
@kaka11chen kaka11chen mentioned this pull request Jul 21, 2025
Merged
@morningman morningman added the usercase Important user case type label label Jul 21, 2025
dataroaring pushed a commit that referenced this pull request Jul 21, 2025
@kaka11chen
[fix & opt](orc) ORC-1525: Fix bad read in RleDecoderV2::readByte and…
cc045b3 
… Decompress zlib by libdeflate. (#53610)

Cherry-pick #51775
@yiguolei yiguolei mentioned this pull request Aug 11, 2025
Open
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.7-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@kaka11chen @hello-stephen @doris-robot @morningman @CalvinKirs @yiguolei @dataroaring @morrySnow