Skip to content

branch-3.1: [fix](outfile) fix small file output with bz2 compression #56368 #57041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Oct 16, 2025
Merged

branch-3.1: [fix](outfile) fix small file output with bz2 compression #56368 #57041

morningman merged 1 commit into from
Oct 16, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Oct 16, 2025

bp #56368

@morningman
[fix](outfile) fix small file output with bz2 compression (apache#56368)
40a4453 
When export small data with bz2 compression, it may return
BZ_OUTBUFF_FULL,
because the pre allocated output buffer is too small.
This PR fix this by adding a fix size to avoid this problem
@morningman morningman requested a review from morrySnow as a code owner October 16, 2025 06:46
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 16, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Oct 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-H: Total hot run time: 33406 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40a4453cade1cabf36fb60a841a3abbeb79a6750, data reload: false

------ Round 1 ----------------------------------
q1	17606	6018	5788	5788
q2	2028	389	281	281
q3	12057	1243	747	747
q4	10531	880	477	477
q5	9522	2447	2198	2198
q6	190	165	131	131
q7	913	761	609	609
q8	9325	1489	1259	1259
q9	5333	5025	4925	4925
q10	6775	2292	1800	1800
q11	492	285	255	255
q12	350	368	214	214
q13	17828	3612	3065	3065
q14	225	234	212	212
q15	536	481	467	467
q16	426	428	372	372
q17	624	882	383	383
q18	6989	6508	6435	6435
q19	1218	974	553	553
q20	350	345	206	206
q21	2964	2232	2061	2061
q22	1031	1002	968	968
Total cold run time: 107313 ms
Total hot run time: 33406 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5598	5568	5688	5568
q2	239	327	234	234
q3	2263	2633	2321	2321
q4	1366	1784	1373	1373
q5	4447	5175	5092	5092
q6	174	167	127	127
q7	2071	2030	1835	1835
q8	2693	2844	2709	2709
q9	7316	7268	7311	7268
q10	3023	3261	2839	2839
q11	602	519	498	498
q12	678	789	626	626
q13	3445	3785	3156	3156
q14	279	298	280	280
q15	517	476	471	471
q16	458	496	442	442
q17	1252	1727	1307	1307
q18	7681	7498	7337	7337
q19	851	1172	1086	1086
q20	2042	2079	1940	1940
q21	5500	4974	4602	4602
q22	1094	1112	1028	1028
Total cold run time: 53589 ms
Total hot run time: 52139 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-DS: Total hot run time: 193629 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40a4453cade1cabf36fb60a841a3abbeb79a6750, data reload: false

query1	950	399	391	391
query2	6247	2011	1859	1859
query3	8690	210	216	210
query4	33481	24564	23742	23742
query5	3609	601	463	463
query6	286	193	181	181
query7	4216	523	331	331
query8	318	272	246	246
query9	9337	2620	2600	2600
query10	459	318	257	257
query11	18291	15450	15296	15296
query12	163	119	110	110
query13	1570	560	430	430
query14	9078	6899	7032	6899
query15	220	196	185	185
query16	7940	694	520	520
query17	1551	778	615	615
query18	2077	440	347	347
query19	225	217	171	171
query20	131	122	115	115
query21	206	132	113	113
query22	4584	4551	4530	4530
query23	35187	34736	34161	34161
query24	8506	2771	2751	2751
query25	540	511	437	437
query26	1170	289	175	175
query27	2833	498	361	361
query28	5696	2254	2226	2226
query29	784	641	481	481
query30	236	191	173	173
query31	1011	943	814	814
query32	84	63	58	58
query33	503	410	343	343
query34	788	876	520	520
query35	812	818	725	725
query36	1022	1084	959	959
query37	115	100	68	68
query38	4130	3992	4027	3992
query39	1555	1519	1556	1519
query40	207	126	115	115
query41	50	49	52	49
query42	131	114	109	109
query43	524	535	502	502
query44	1418	839	839	839
query45	184	179	174	174
query46	914	1056	691	691
query47	1977	1978	1938	1938
query48	422	442	338	338
query49	780	516	427	427
query50	703	707	440	440
query51	7344	7397	7245	7245
query52	110	105	93	93
query53	248	280	205	205
query54	570	577	496	496
query55	81	84	83	83
query56	288	281	258	258
query57	1311	1239	1218	1218
query58	240	227	219	219
query59	3084	3247	2983	2983
query60	316	299	271	271
query61	120	110	114	110
query62	803	771	699	699
query63	265	208	214	208
query64	4252	1029	666	666
query65	3402	3387	3319	3319
query66	1071	415	318	318
query67	16436	15964	15531	15531
query68	7454	842	553	553
query69	488	324	273	273
query70	1243	1156	1087	1087
query71	381	312	275	275
query72	5744	3822	3913	3822
query73	645	773	379	379
query74	10348	9415	9183	9183
query75	3291	3167	2710	2710
query76	3327	1188	811	811
query77	598	382	287	287
query78	10311	10623	9629	9629
query79	3066	902	641	641
query80	700	555	482	482
query81	504	265	223	223
query82	448	127	95	95
query83	172	166	148	148
query84	281	101	82	82
query85	796	398	312	312
query86	367	306	285	285
query87	4364	4272	4310	4272
query88	4787	2450	2406	2406
query89	408	327	299	299
query90	1866	197	204	197
query91	140	147	124	124
query92	64	61	53	53
query93	1808	904	560	560
query94	681	432	312	312
query95	351	292	282	282
query96	493	633	287	287
query97	3154	3247	3144	3144
query98	234	212	209	209
query99	1561	1421	1289	1289
Total cold run time: 295927 ms
Total hot run time: 193629 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

ClickBench: Total hot run time: 29.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40a4453cade1cabf36fb60a841a3abbeb79a6750, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.11	0.11
query5	0.53	0.53	0.54
query6	1.13	0.73	0.73
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.52
query10	0.54	0.54	0.55
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.63	0.61	0.60
query14	0.77	0.82	0.79
query15	0.87	0.84	0.83
query16	0.40	0.39	0.40
query17	1.08	1.01	1.04
query18	0.24	0.22	0.22
query19	1.82	1.87	1.82
query20	0.01	0.01	0.01
query21	15.43	0.91	0.60
query22	0.74	0.79	0.61
query23	15.15	1.43	0.62
query24	3.00	1.18	2.55
query25	0.25	0.08	0.10
query26	0.37	0.17	0.14
query27	0.06	0.04	0.04
query28	13.88	0.98	0.44
query29	12.56	3.99	3.25
query30	0.26	0.10	0.07
query31	2.81	0.63	0.39
query32	3.22	0.54	0.46
query33	3.00	3.01	3.06
query34	16.62	5.18	4.56
query35	4.55	4.57	4.56
query36	0.62	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 103.95 s
Total hot run time: 29.26 s

@morrySnow morrySnow changed the title branch-3.1: [fix](outfile) fix small file output with bz2 compression (#56368) branch-3.1: [fix](outfile) fix small file output with bz2 compression #56368 Oct 16, 2025
@morningman morningman merged commit cb07d50 into apache:branch-3.1 Oct 16, 2025
22 of 23 checks passed
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@morningman @hello-stephen @doris-robot @morrySnow