Skip to content

branch-3.1: [enhancement](compaction) Optimize compaction policy for empty rowsets #56173 #56768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Oct 11, 2025
Merged

branch-3.1: [enhancement](compaction) Optimize compaction policy for empty rowsets #56173 #56768

morrySnow merged 1 commit into from
Oct 11, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Oct 10, 2025

Cherry-picked from #56173

@luwei16
[enhancement](compaction) Optimize compaction policy for empty rowsets (
c5b4b09 
#56173)

When more than 5 consecutive empty rowsets are found and they account
for over 30% of all input rowsets, prioritize handling empty rowsets
@github-actions github-actions bot requested a review from morrySnow as a code owner October 10, 2025 02:50
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Oct 10, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

TPC-H: Total hot run time: 32522 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c5b4b0900163e3b414267be0062c00e9db57a065, data reload: false

------ Round 1 ----------------------------------
q1	17610	5557	5428	5428
q2	2034	386	280	280
q3	12298	1229	755	755
q4	10301	865	458	458
q5	8958	2403	2128	2128
q6	185	163	134	134
q7	910	763	632	632
q8	9338	1460	1172	1172
q9	5286	4967	4861	4861
q10	6751	2268	1824	1824
q11	474	278	263	263
q12	333	353	206	206
q13	17778	3579	2971	2971
q14	231	227	211	211
q15	519	459	463	459
q16	435	435	368	368
q17	589	861	361	361
q18	6913	6466	6288	6288
q19	1226	957	529	529
q20	328	338	211	211
q21	2780	2139	1991	1991
q22	1024	1036	992	992
Total cold run time: 106301 ms
Total hot run time: 32522 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5489	5467	5502	5467
q2	241	329	231	231
q3	2290	2622	2354	2354
q4	1345	1782	1378	1378
q5	4400	4968	5031	4968
q6	169	156	131	131
q7	2115	1954	1813	1813
q8	2648	2849	2705	2705
q9	7249	7297	7223	7223
q10	3052	3254	2729	2729
q11	580	521	484	484
q12	660	763	640	640
q13	3390	3795	3148	3148
q14	281	295	285	285
q15	508	482	465	465
q16	430	481	428	428
q17	1250	1728	1273	1273
q18	7794	7462	7357	7357
q19	851	1184	1079	1079
q20	2053	2046	1919	1919
q21	5317	4937	4626	4626
q22	1093	1145	1054	1054
Total cold run time: 53205 ms
Total hot run time: 51757 ms

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

BE UT Coverage Report

Increment line coverage 85.37% (35/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.72% (12850/28105)
Line Coverage 36.61% (115021/314181)
Region Coverage 34.15% (65626/192153)
Branch Coverage 31.17% (34456/110532)

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

TPC-DS: Total hot run time: 193174 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c5b4b0900163e3b414267be0062c00e9db57a065, data reload: false

query1	934	408	402	402
query2	7027	1922	1920	1920
query3	8688	197	194	194
query4	33931	23828	23674	23674
query5	3595	572	452	452
query6	327	190	172	172
query7	4204	489	309	309
query8	294	235	222	222
query9	9498	2623	2612	2612
query10	479	323	260	260
query11	18070	15671	15262	15262
query12	159	120	110	110
query13	1560	543	413	413
query14	9541	7204	7208	7204
query15	237	190	185	185
query16	8084	673	505	505
query17	1564	776	599	599
query18	2158	425	325	325
query19	224	195	180	180
query20	132	123	117	117
query21	215	130	112	112
query22	4632	4682	4585	4585
query23	35040	34240	34281	34240
query24	7610	2697	2693	2693
query25	509	507	445	445
query26	913	279	174	174
query27	2054	490	366	366
query28	5345	2284	2269	2269
query29	613	579	470	470
query30	237	195	161	161
query31	1023	910	824	824
query32	96	59	58	58
query33	509	406	314	314
query34	753	858	543	543
query35	784	834	763	763
query36	1037	1073	960	960
query37	108	86	71	71
query38	3992	3986	4025	3986
query39	1538	1471	1491	1471
query40	210	124	108	108
query41	51	50	45	45
query42	130	108	104	104
query43	527	527	493	493
query44	1356	852	827	827
query45	181	183	183	183
query46	895	1064	675	675
query47	1978	1964	1911	1911
query48	416	436	367	367
query49	734	493	401	401
query50	688	706	452	452
query51	7291	7336	7284	7284
query52	102	97	94	94
query53	241	256	194	194
query54	570	563	478	478
query55	77	78	77	77
query56	276	288	267	267
query57	1254	1265	1231	1231
query58	244	215	213	213
query59	3061	3183	2995	2995
query60	288	283	280	280
query61	119	113	116	113
query62	811	754	700	700
query63	233	193	195	193
query64	3786	1007	647	647
query65	3387	3340	3309	3309
query66	971	414	315	315
query67	16587	15943	15582	15582
query68	7945	836	553	553
query69	499	317	268	268
query70	1181	1097	1065	1065
query71	414	288	273	273
query72	5106	3771	3833	3771
query73	647	744	360	360
query74	10326	9522	8844	8844
query75	3739	3165	2680	2680
query76	3550	1181	758	758
query77	767	366	280	280
query78	10294	10500	9515	9515
query79	4664	864	593	593
query80	645	520	454	454
query81	492	262	222	222
query82	231	117	92	92
query83	167	161	141	141
query84	284	95	83	83
query85	753	368	295	295
query86	348	333	286	286
query87	4362	4335	4271	4271
query88	4156	2425	2406	2406
query89	440	331	297	297
query90	2128	190	187	187
query91	135	140	110	110
query92	62	57	54	54
query93	3161	881	542	542
query94	660	422	308	308
query95	349	277	272	272
query96	494	628	284	284
query97	3175	3315	3214	3214
query98	213	204	201	201
query99	1419	1399	1335	1335
Total cold run time: 296984 ms
Total hot run time: 193174 ms

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

ClickBench: Total hot run time: 28.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c5b4b0900163e3b414267be0062c00e9db57a065, data reload: false

query1	0.03	0.03	0.04
query2	0.06	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.11	0.10
query5	0.53	0.54	0.53
query6	1.13	0.73	0.73
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.59	0.51	0.50
query10	0.56	0.55	0.54
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.62	0.60	0.59
query14	0.77	0.80	0.79
query15	0.84	0.83	0.82
query16	0.39	0.38	0.39
query17	1.02	1.00	0.99
query18	0.24	0.23	0.22
query19	1.90	1.82	1.78
query20	0.01	0.01	0.02
query21	15.43	0.94	0.58
query22	0.74	0.81	0.67
query23	15.08	1.44	0.62
query24	3.68	0.48	1.26
query25	0.16	0.15	0.07
query26	0.35	0.16	0.14
query27	0.05	0.04	0.03
query28	12.77	1.06	0.44
query29	12.53	3.96	3.20
query30	0.26	0.10	0.06
query31	2.81	0.61	0.37
query32	3.23	0.54	0.46
query33	3.02	3.01	3.02
query34	16.66	5.40	4.52
query35	4.56	4.55	4.57
query36	0.64	0.50	0.47
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.17	0.12	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.36 s
Total hot run time: 28.34 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 10, 2025

BE Regression && UT Coverage Report

Increment line coverage 85.37% (35/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.54% (21158/27643)
Line Coverage 69.88% (219055/313463)
Region Coverage 67.79% (130860/193043)
Branch Coverage 61.36% (68162/111088)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 10, 2025

BE Regression && UT Coverage Report

Increment line coverage 85.37% (35/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.48% (21141/27643)
Line Coverage 69.79% (218762/313463)
Region Coverage 67.71% (130713/193043)
Branch Coverage 61.28% (68072/111088)

@morrySnow morrySnow merged commit 5960d98 into branch-3.1 Oct 11, 2025
22 of 23 checks passed
@github-actions github-actions bot deleted the auto-pick-56173-branch-3.1 branch October 11, 2025 03:33
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @morrySnow @dataroaring @luwei16