Skip to content

[enhancement](compaction) Optimize compaction policy for empty rowsets #56173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 4 commits into from
Oct 10, 2025
Merged

[enhancement](compaction) Optimize compaction policy for empty rowsets #56173

dataroaring merged 4 commits into from
Oct 10, 2025

Conversation

@luwei16
Copy link
Contributor

@luwei16 luwei16 commented Sep 17, 2025

When more than 5 consecutive empty rowsets are found and they account for over 30% of all input rowsets, prioritize handling empty rowsets

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@luwei16
Copy link
Contributor Author

luwei16 commented Sep 17, 2025

run buildall

@luwei16
Copy link
Contributor Author

luwei16 commented Sep 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 17, 2025

TPC-H: Total hot run time: 34618 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b96c5261be55938917dde7438d38b57649f4ac5d, data reload: false

------ Round 1 ----------------------------------
q1	17617	5236	5105	5105
q2	2007	317	207	207
q3	10301	1258	718	718
q4	10234	1020	517	517
q5	7534	2448	2361	2361
q6	180	165	135	135
q7	931	769	610	610
q8	9336	1309	1099	1099
q9	6946	5203	5176	5176
q10	6951	2408	1963	1963
q11	485	305	286	286
q12	361	356	234	234
q13	17804	3620	3052	3052
q14	253	228	222	222
q15	571	492	488	488
q16	1005	999	940	940
q17	621	882	352	352
q18	7554	7252	7020	7020
q19	1458	968	567	567
q20	339	334	228	228
q21	3685	2577	2356	2356
q22	1064	1002	982	982
Total cold run time: 107237 ms
Total hot run time: 34618 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5288	5164	5122	5122
q2	251	334	229	229
q3	2194	2669	2245	2245
q4	1371	1775	1345	1345
q5	4152	4516	4513	4513
q6	228	181	133	133
q7	1999	2045	1846	1846
q8	2687	2603	2670	2603
q9	7357	7217	7234	7217
q10	3149	3281	2865	2865
q11	584	534	511	511
q12	732	782	689	689
q13	3583	3870	3275	3275
q14	283	326	289	289
q15	528	479	492	479
q16	1050	1108	1051	1051
q17	1222	1524	1471	1471
q18	7922	7701	7178	7178
q19	765	794	968	794
q20	1916	1954	1804	1804
q21	4756	4297	4206	4206
q22	1088	1042	997	997
Total cold run time: 53105 ms
Total hot run time: 50862 ms

@doris-robot
Copy link

doris-robot commented Sep 17, 2025

TPC-DS: Total hot run time: 189328 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b96c5261be55938917dde7438d38b57649f4ac5d, data reload: false

query1	1079	464	424	424
query2	6549	1715	1696	1696
query3	6754	228	230	228
query4	27420	23907	23662	23662
query5	4451	642	487	487
query6	352	251	238	238
query7	4661	547	296	296
query8	320	273	258	258
query9	8680	2606	2617	2606
query10	492	365	299	299
query11	16085	15211	15144	15144
query12	170	123	116	116
query13	1691	572	445	445
query14	10802	9293	9332	9293
query15	211	196	175	175
query16	7760	709	521	521
query17	1231	745	609	609
query18	2049	446	345	345
query19	202	203	168	168
query20	135	125	133	125
query21	218	136	116	116
query22	4128	4170	4150	4150
query23	33924	32762	32813	32762
query24	8528	2402	2364	2364
query25	566	524	430	430
query26	1227	276	158	158
query27	2739	520	373	373
query28	4334	2217	2205	2205
query29	777	595	509	509
query30	293	231	198	198
query31	908	846	727	727
query32	80	72	67	67
query33	572	386	334	334
query34	792	866	527	527
query35	816	843	725	725
query36	996	1052	944	944
query37	118	122	84	84
query38	3554	3646	3535	3535
query39	1517	1439	1412	1412
query40	220	140	127	127
query41	67	66	64	64
query42	132	120	114	114
query43	487	506	482	482
query44	1333	839	851	839
query45	190	183	169	169
query46	860	1032	640	640
query47	1783	1796	1708	1708
query48	387	412	312	312
query49	779	514	431	431
query50	648	683	408	408
query51	4041	4003	3914	3914
query52	117	114	107	107
query53	243	278	205	205
query54	618	605	540	540
query55	91	90	84	84
query56	332	369	303	303
query57	1209	1211	1111	1111
query58	277	277	274	274
query59	2588	2694	2589	2589
query60	352	332	330	330
query61	207	154	181	154
query62	834	774	692	692
query63	225	193	186	186
query64	4457	1129	861	861
query65	4143	4021	4007	4007
query66	1094	441	334	334
query67	15544	15430	15119	15119
query68	9079	944	598	598
query69	493	310	281	281
query70	1391	1261	1280	1261
query71	547	334	316	316
query72	6099	4855	4966	4855
query73	722	625	364	364
query74	9062	8980	8631	8631
query75	4404	3385	2843	2843
query76	3732	1274	752	752
query77	819	411	342	342
query78	10098	10075	9095	9095
query79	2154	855	586	586
query80	670	554	498	498
query81	492	277	236	236
query82	428	158	129	129
query83	298	267	276	267
query84	302	109	101	101
query85	921	473	421	421
query86	336	314	293	293
query87	3877	3771	3737	3737
query88	3221	2244	2268	2244
query89	394	334	307	307
query90	1926	214	210	210
query91	164	171	130	130
query92	87	69	60	60
query93	1538	995	646	646
query94	692	451	345	345
query95	405	316	302	302
query96	480	570	279	279
query97	3039	3019	2860	2860
query98	241	218	207	207
query99	1436	1437	1273	1273
Total cold run time: 279986 ms
Total hot run time: 189328 ms

@doris-robot
Copy link

doris-robot commented Sep 17, 2025

ClickBench: Total hot run time: 29.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b96c5261be55938917dde7438d38b57649f4ac5d, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.06	0.06
query3	0.24	0.08	0.08
query4	1.60	0.11	0.11
query5	0.28	0.27	0.28
query6	1.18	0.67	0.65
query7	0.03	0.03	0.03
query8	0.06	0.05	0.04
query9	0.62	0.53	0.51
query10	0.57	0.57	0.57
query11	0.15	0.11	0.11
query12	0.16	0.12	0.12
query13	0.63	0.63	0.62
query14	1.03	1.04	1.06
query15	0.86	0.84	0.86
query16	0.43	0.41	0.39
query17	1.04	1.05	1.07
query18	0.22	0.20	0.21
query19	1.95	1.79	1.83
query20	0.01	0.01	0.01
query21	15.43	0.94	0.58
query22	0.79	1.15	0.72
query23	14.91	1.37	0.66
query24	7.04	1.36	0.57
query25	0.50	0.17	0.10
query26	0.66	0.16	0.13
query27	0.07	0.05	0.04
query28	8.96	0.90	0.44
query29	12.58	3.92	3.28
query30	0.28	0.13	0.13
query31	2.84	0.57	0.39
query32	3.23	0.57	0.47
query33	3.04	3.16	3.11
query34	16.15	5.52	4.86
query35	4.94	4.91	4.92
query36	0.71	0.51	0.50
query37	0.11	0.07	0.08
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.19	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.03	0.04
query43	0.04	0.04	0.03
Total cold run time: 103.91 s
Total hot run time: 29.79 s

@luwei16
Copy link
Contributor Author

luwei16 commented Oct 8, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 8, 2025

TPC-DS: Total hot run time: 190655 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 26717c775fdc486fec7a61e45ff92f1aa632b2a6, data reload: false

query1	1091	434	405	405
query2	6574	1743	1735	1735
query3	6753	223	220	220
query4	27069	23803	23332	23332
query5	5199	654	485	485
query6	338	246	249	246
query7	4663	498	296	296
query8	338	251	238	238
query9	8684	2573	2543	2543
query10	540	345	279	279
query11	15270	15225	14802	14802
query12	179	118	115	115
query13	1669	560	423	423
query14	11360	9423	9517	9423
query15	215	191	177	177
query16	8273	742	555	555
query17	1663	854	721	721
query18	2357	464	348	348
query19	279	207	190	190
query20	134	150	137	137
query21	212	144	118	118
query22	4445	4700	4669	4669
query23	34487	33907	33964	33907
query24	8559	2487	2576	2487
query25	575	549	470	470
query26	1741	287	174	174
query27	2726	525	379	379
query28	4426	2208	2202	2202
query29	794	605	483	483
query30	294	223	196	196
query31	883	818	711	711
query32	81	76	72	72
query33	592	361	327	327
query34	781	866	528	528
query35	794	820	766	766
query36	990	998	948	948
query37	116	141	81	81
query38	3537	3617	3559	3559
query39	1455	1419	1417	1417
query40	222	126	115	115
query41	61	59	62	59
query42	119	113	117	113
query43	486	503	474	474
query44	1323	817	829	817
query45	182	181	173	173
query46	835	985	635	635
query47	1772	1792	1733	1733
query48	388	436	340	340
query49	776	524	416	416
query50	644	692	407	407
query51	3976	4028	3919	3919
query52	111	108	101	101
query53	254	279	200	200
query54	599	588	523	523
query55	96	83	91	83
query56	338	324	312	312
query57	1186	1177	1128	1128
query58	281	288	274	274
query59	2471	2646	2519	2519
query60	352	345	327	327
query61	154	157	158	157
query62	778	747	668	668
query63	235	199	197	197
query64	4501	1244	920	920
query65	4031	3916	3975	3916
query66	1048	429	350	350
query67	15500	15494	15400	15400
query68	8213	944	591	591
query69	511	324	294	294
query70	1393	1308	1357	1308
query71	499	339	328	328
query72	6012	4907	4907	4907
query73	688	598	358	358
query74	8903	9122	8654	8654
query75	4078	3368	2802	2802
query76	3791	1167	729	729
query77	816	476	314	314
query78	9611	9805	8826	8826
query79	2951	841	583	583
query80	704	558	492	492
query81	490	256	228	228
query82	443	160	133	133
query83	300	263	240	240
query84	305	117	89	89
query85	959	467	427	427
query86	341	303	308	303
query87	3819	3846	3720	3720
query88	3373	2221	2195	2195
query89	404	324	305	305
query90	2061	220	218	218
query91	162	159	134	134
query92	86	77	68	68
query93	2236	963	638	638
query94	691	438	334	334
query95	397	323	312	312
query96	475	586	278	278
query97	2941	2990	2886	2886
query98	251	219	219	219
query99	1452	1426	1302	1302
Total cold run time: 282211 ms
Total hot run time: 190655 ms

@doris-robot
Copy link

doris-robot commented Oct 8, 2025

ClickBench: Total hot run time: 30.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 26717c775fdc486fec7a61e45ff92f1aa632b2a6, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.06
query3	0.25	0.08	0.08
query4	1.61	0.12	0.11
query5	0.27	0.26	0.25
query6	1.19	0.65	0.64
query7	0.03	0.02	0.02
query8	0.06	0.04	0.04
query9	0.64	0.52	0.54
query10	0.58	0.57	0.58
query11	0.17	0.12	0.12
query12	0.15	0.12	0.12
query13	0.65	0.62	0.61
query14	1.04	1.02	1.01
query15	0.87	0.86	0.87
query16	0.42	0.41	0.41
query17	1.08	1.06	1.06
query18	0.21	0.21	0.20
query19	2.01	1.84	1.85
query20	0.01	0.03	0.02
query21	15.42	0.95	0.57
query22	0.76	1.14	0.61
query23	15.03	1.36	0.64
query24	7.25	1.40	0.97
query25	0.48	0.15	0.25
query26	0.58	0.16	0.14
query27	0.07	0.06	0.05
query28	9.79	1.34	0.92
query29	12.66	4.04	3.29
query30	0.28	0.13	0.12
query31	2.83	0.60	0.39
query32	3.25	0.57	0.47
query33	3.11	3.12	3.14
query34	16.19	5.49	4.80
query35	4.92	4.93	4.93
query36	0.69	0.52	0.50
query37	0.11	0.08	0.08
query38	0.07	0.04	0.05
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.05
Total cold run time: 105.28 s
Total hot run time: 30.64 s

@doris-robot
Copy link

doris-robot commented Oct 8, 2025

BE UT Coverage Report

Increment line coverage 85.37% (35/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.50% (17714/33740)
Line Coverage 37.68% (160856/426913)
Region Coverage 32.16% (122860/381979)
Branch Coverage 33.56% (53885/160559)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 8, 2025

BE Regression && UT Coverage Report

Increment line coverage 85.37% (35/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.19% (23533/33058)
Line Coverage 57.64% (245818/426482)
Region Coverage 52.78% (204161/386807)
Branch Coverage 54.50% (87950/161370)

@luwei16
Copy link
Contributor Author

luwei16 commented Oct 9, 2025

run cloud_p0

@luwei16
Copy link
Contributor Author

luwei16 commented Oct 9, 2025

run p0

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 9, 2025

BE Regression && UT Coverage Report

Increment line coverage 85.37% (35/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.19% (23534/33058)
Line Coverage 57.64% (245829/426482)
Region Coverage 52.80% (204237/386807)
Branch Coverage 54.51% (87969/161370)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 9, 2025

BE Regression && UT Coverage Report

Increment line coverage 85.37% (35/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.19% (23534/33058)
Line Coverage 57.64% (245829/426482)
Region Coverage 52.80% (204237/386807)
Branch Coverage 54.51% (87969/161370)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 10, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 10, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 10, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Oct 10, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit b858541 into apache:master Oct 10, 2025
28 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 10, 2025
@luwei16
[enhancement](compaction) Optimize compaction policy for empty rowsets (
c5b4b09 
#56173)

When more than 5 consecutive empty rowsets are found and they account
for over 30% of all input rowsets, prioritize handling empty rowsets
@github-actions github-actions bot mentioned this pull request Oct 10, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Oct 10, 2025
@luwei16
[enhancement](compaction) Optimize compaction policy for empty rowsets (
cdba17f 
#56173)

When more than 5 consecutive empty rowsets are found and they account
for over 30% of all input rowsets, prioritize handling empty rowsets
@github-actions github-actions bot mentioned this pull request Oct 10, 2025
Merged
morrySnow pushed a commit that referenced this pull request Oct 11, 2025
@github-actions @luwei16
branch-3.1: [enhancement](compaction) Optimize compaction policy for …
5960d98 
…empty rowsets #56173 (#56768)

Cherry-picked from #56173

Co-authored-by: Luwei <814383175@qq.com>
yiguolei pushed a commit that referenced this pull request Oct 14, 2025
@github-actions @luwei16
branch-4.0: [enhancement](compaction) Optimize compaction policy for …
e291a5e 
…empty rowsets #56173 (#56769)

Cherry-picked from #56173

Co-authored-by: Luwei <814383175@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

@w41ter w41ter Awaiting requested review from w41ter w41ter is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.2-merged dev/4.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@luwei16 @hello-stephen @doris-robot @gavinchou @dataroaring @yiguolei @morrySnow