Skip to content

branch-3.1: [fix](mow) delete bitmap is not deleted if commit compaction job failed #56758 #56785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Oct 13, 2025
Merged

branch-3.1: [fix](mow) delete bitmap is not deleted if commit compaction job failed #56758 #56785

morrySnow merged 2 commits into from
Oct 13, 2025

Conversation

@mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Oct 10, 2025 •
edited by morrySnow
Loading

picked from #56758

@mymeiyi
[fix](mow) delete bitmap is not deleted if commit compaction job fail…
d951790 
…ed (apache#56758)

delete bitmap kv is leaked:
```
[delete bitmap check fails] can't find corresponding rowset for delete bitmap instance_id=113100978, tablet_id=1759979382398, rowset_id=0200000000000027bc46eb5855e1afdc75b79046710c329f, version=13, segment_id=0
```
this is because:
1. when update delete bitmap for  compaction, does not write pending key
2. when commit compaction job failed, the delete bitmap kv is not
deleted
3. when delete tmp rowset kv, the delete bitmap kv is not deleted
@mymeiyi
fix
fbe464e
@mymeiyi mymeiyi requested a review from morrySnow as a code owner October 10, 2025 06:27
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Oct 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

Cloud UT Coverage Report

Increment line coverage 84.21% (16/19) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.42% (1238/1502)
Line Coverage 66.31% (22379/33749)
Region Coverage 67.65% (11192/16545)
Branch Coverage 57.42% (5931/10330)

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

TPC-H: Total hot run time: 32777 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fbe464e4fc14d504cec504de3e81a0a642efb6df, data reload: false

------ Round 1 ----------------------------------
q1	17611	5508	5470	5470
q2	2027	403	283	283
q3	12282	1249	763	763
q4	10543	868	452	452
q5	9569	2403	2112	2112
q6	184	167	133	133
q7	907	744	618	618
q8	9340	1443	1123	1123
q9	5235	4961	4926	4926
q10	6793	2263	1852	1852
q11	479	279	263	263
q12	342	358	225	225
q13	17781	3569	2981	2981
q14	225	233	209	209
q15	533	471	461	461
q16	413	427	363	363
q17	598	851	355	355
q18	6860	6507	6452	6452
q19	1218	963	546	546
q20	320	338	207	207
q21	2794	2196	2011	2011
q22	1045	1047	972	972
Total cold run time: 107099 ms
Total hot run time: 32777 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5552	5494	5492	5492
q2	234	323	234	234
q3	2274	2672	2308	2308
q4	1381	1818	1388	1388
q5	4452	5018	5133	5018
q6	179	177	136	136
q7	2163	2091	1878	1878
q8	2689	2836	2773	2773
q9	7493	7303	7244	7244
q10	3015	3207	2812	2812
q11	575	515	525	515
q12	697	770	590	590
q13	3417	3817	3188	3188
q14	298	286	266	266
q15	531	465	457	457
q16	439	495	443	443
q17	1226	1750	1247	1247
q18	7565	7451	7306	7306
q19	846	1138	1107	1107
q20	2047	2046	1897	1897
q21	5468	5052	4606	4606
q22	1093	1106	1038	1038
Total cold run time: 53634 ms
Total hot run time: 51943 ms

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

TPC-DS: Total hot run time: 193036 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fbe464e4fc14d504cec504de3e81a0a642efb6df, data reload: false

query1	939	408	407	407
query2	6290	1964	1938	1938
query3	8691	201	197	197
query4	33675	23801	23752	23752
query5	3725	617	470	470
query6	291	197	186	186
query7	4204	491	316	316
query8	321	256	245	245
query9	9558	2619	2609	2609
query10	476	314	264	264
query11	18046	15283	15307	15283
query12	156	104	108	104
query13	1563	539	421	421
query14	9043	7079	6832	6832
query15	267	196	187	187
query16	8082	712	501	501
query17	1575	723	579	579
query18	2150	432	327	327
query19	202	202	158	158
query20	123	122	117	117
query21	206	136	117	117
query22	4482	4639	4437	4437
query23	35430	33864	34004	33864
query24	8036	2728	2742	2728
query25	522	489	425	425
query26	811	283	175	175
query27	2386	483	355	355
query28	5629	2256	2207	2207
query29	716	603	471	471
query30	255	195	162	162
query31	1003	935	872	872
query32	90	61	65	61
query33	545	393	326	326
query34	746	875	547	547
query35	817	824	742	742
query36	1012	1045	981	981
query37	111	98	73	73
query38	4133	4166	4039	4039
query39	1551	1501	1475	1475
query40	212	133	119	119
query41	53	52	50	50
query42	135	112	115	112
query43	517	529	501	501
query44	1376	849	830	830
query45	186	184	172	172
query46	916	1069	694	694
query47	1996	1970	1963	1963
query48	437	445	369	369
query49	759	536	446	446
query50	725	708	437	437
query51	7462	7455	7207	7207
query52	99	106	98	98
query53	234	264	195	195
query54	554	566	466	466
query55	86	85	85	85
query56	281	274	282	274
query57	1289	1264	1235	1235
query58	239	215	229	215
query59	3090	3099	3069	3069
query60	286	281	263	263
query61	116	116	113	113
query62	798	740	725	725
query63	241	203	203	203
query64	3702	1018	665	665
query65	3368	3288	3282	3282
query66	791	432	305	305
query67	16253	15772	15712	15712
query68	7746	842	559	559
query69	503	302	269	269
query70	1226	1159	1113	1113
query71	383	299	269	269
query72	5712	3688	3840	3688
query73	638	749	361	361
query74	10292	9103	9029	9029
query75	3225	3141	2670	2670
query76	3172	1195	788	788
query77	573	391	285	285
query78	10427	10586	9591	9591
query79	3093	895	602	602
query80	628	514	453	453
query81	498	260	228	228
query82	459	119	87	87
query83	177	165	145	145
query84	288	92	73	73
query85	778	393	301	301
query86	358	301	300	300
query87	4382	4279	4296	4279
query88	4543	2415	2395	2395
query89	408	330	299	299
query90	1847	187	191	187
query91	135	145	110	110
query92	63	56	51	51
query93	1276	902	551	551
query94	689	409	308	308
query95	344	278	275	275
query96	498	605	289	289
query97	3171	3245	3194	3194
query98	236	211	199	199
query99	1532	1391	1333	1333
Total cold run time: 293359 ms
Total hot run time: 193036 ms

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

ClickBench: Total hot run time: 28.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fbe464e4fc14d504cec504de3e81a0a642efb6df, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.07
query4	1.66	0.11	0.10
query5	0.54	0.51	0.53
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.60	0.49	0.50
query10	0.54	0.55	0.55
query11	0.17	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	0.80	0.78	0.79
query15	0.83	0.82	0.83
query16	0.39	0.39	0.37
query17	1.03	1.08	1.05
query18	0.23	0.22	0.21
query19	1.87	1.79	1.74
query20	0.01	0.01	0.01
query21	15.40	0.92	0.58
query22	0.74	0.68	0.65
query23	15.23	1.43	0.61
query24	3.59	0.42	1.01
query25	0.21	0.11	0.07
query26	0.32	0.14	0.13
query27	0.05	0.06	0.05
query28	13.50	1.06	0.43
query29	12.57	3.94	3.26
query30	0.26	0.09	0.07
query31	2.83	0.59	0.38
query32	3.22	0.54	0.47
query33	3.00	3.00	3.04
query34	16.52	5.12	4.50
query35	4.59	4.52	4.59
query36	0.65	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.1 s
Total hot run time: 28.26 s

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Oct 11, 2025

run p0

@morrySnow morrySnow changed the title branch-3.1: [fix](mow) delete bitmap is not deleted if commit compaction job failed (#56758) branch-3.1: [fix](mow) delete bitmap is not deleted if commit compaction job failed #56758 Oct 13, 2025
@morrySnow morrySnow merged commit 0b4b731 into apache:branch-3.1 Oct 13, 2025
23 of 24 checks passed
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mymeiyi @hello-stephen @doris-robot @morrySnow