Skip to content

[fix](mow) delete bitmap is not deleted if commit compaction job failed #56758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
w41ter merged 1 commit into from
Oct 10, 2025
Merged

[fix](mow) delete bitmap is not deleted if commit compaction job failed #56758

w41ter merged 1 commit into from
Oct 10, 2025

Conversation

@mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Oct 9, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

delete bitmap kv is leaked:

[delete bitmap check fails] can't find corresponding rowset for delete bitmap instance_id=113100978, tablet_id=1759979382398, rowset_id=0200000000000027bc46eb5855e1afdc75b79046710c329f, version=13, segment_id=0

this is because:

  1. when update delete bitmap for compaction, does not write pending key
  2. when commit compaction job failed, the delete bitmap kv is not deleted
  3. when delete tmp rowset kv, the delete bitmap kv is not deleted

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Oct 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Oct 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 9, 2025

Cloud UT Coverage Report

Increment line coverage 73.68% (14/19) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.36% (1633/1959)
Line Coverage 67.92% (28884/42529)
Region Coverage 68.14% (14229/20882)
Branch Coverage 58.45% (7577/12964)

@doris-robot
Copy link

doris-robot commented Oct 9, 2025

TPC-DS: Total hot run time: 190287 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6ce1854a6c9741a21f057d07384563650cd741de, data reload: false

query1	1074	430	399	399
query2	6546	1710	1694	1694
query3	6750	221	217	217
query4	25876	24009	23158	23158
query5	5848	626	457	457
query6	361	265	207	207
query7	4652	487	304	304
query8	320	261	242	242
query9	8714	2566	2536	2536
query10	567	333	288	288
query11	15712	14970	14927	14927
query12	204	116	114	114
query13	1671	545	440	440
query14	11632	9223	9286	9223
query15	205	192	179	179
query16	7657	671	528	528
query17	1290	739	613	613
query18	2051	519	389	389
query19	240	214	177	177
query20	146	135	130	130
query21	213	152	134	134
query22	4645	4718	4711	4711
query23	34818	33969	33989	33969
query24	8361	2533	2568	2533
query25	612	572	458	458
query26	1238	283	158	158
query27	3488	529	351	351
query28	4789	2225	2185	2185
query29	806	675	516	516
query30	348	241	209	209
query31	954	868	758	758
query32	88	79	75	75
query33	599	423	359	359
query34	823	886	558	558
query35	833	872	794	794
query36	1012	991	908	908
query37	119	114	92	92
query38	3470	3530	3481	3481
query39	1469	1454	1458	1454
query40	225	129	122	122
query41	68	63	66	63
query42	125	112	110	110
query43	492	481	456	456
query44	1321	830	835	830
query45	186	184	174	174
query46	835	1007	642	642
query47	1784	1848	1773	1773
query48	410	448	320	320
query49	779	538	424	424
query50	637	693	441	441
query51	3908	3951	3866	3866
query52	112	110	104	104
query53	232	274	202	202
query54	588	584	523	523
query55	93	85	85	85
query56	325	315	321	315
query57	1195	1198	1147	1147
query58	285	278	278	278
query59	2600	2657	2584	2584
query60	349	333	333	333
query61	160	150	153	150
query62	838	734	671	671
query63	233	198	184	184
query64	4412	1141	832	832
query65	4067	3955	3966	3955
query66	1040	442	345	345
query67	15535	15439	15227	15227
query68	8338	938	596	596
query69	516	318	298	298
query70	1358	1268	1235	1235
query71	530	329	316	316
query72	6003	4902	4794	4794
query73	670	567	357	357
query74	8866	8873	8652	8652
query75	4197	3378	2840	2840
query76	3796	1158	734	734
query77	819	410	313	313
query78	9673	9753	8922	8922
query79	2204	848	586	586
query80	628	554	532	532
query81	509	267	234	234
query82	491	166	135	135
query83	265	269	245	245
query84	257	110	94	94
query85	876	463	480	463
query86	338	310	295	295
query87	3756	3735	3709	3709
query88	3679	2217	2191	2191
query89	387	324	292	292
query90	2025	219	219	219
query91	165	182	138	138
query92	82	69	68	68
query93	1678	976	668	668
query94	681	439	313	313
query95	391	325	308	308
query96	480	580	276	276
query97	2963	3005	2866	2866
query98	242	214	216	214
query99	1447	1407	1303	1303
Total cold run time: 281345 ms
Total hot run time: 190287 ms

@doris-robot
Copy link

doris-robot commented Oct 9, 2025

ClickBench: Total hot run time: 30.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6ce1854a6c9741a21f057d07384563650cd741de, data reload: false

query1	0.06	0.05	0.05
query2	0.08	0.05	0.06
query3	0.26	0.08	0.08
query4	1.60	0.12	0.11
query5	0.27	0.26	0.26
query6	1.14	0.66	0.65
query7	0.04	0.03	0.03
query8	0.06	0.04	0.05
query9	0.62	0.53	0.54
query10	0.59	0.59	0.58
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.63	0.61	0.62
query14	1.02	1.04	1.02
query15	0.86	0.88	0.89
query16	0.39	0.40	0.40
query17	1.09	1.09	1.06
query18	0.22	0.20	0.20
query19	1.92	1.86	1.81
query20	0.02	0.01	0.01
query21	15.42	0.91	0.56
query22	0.77	1.09	0.87
query23	14.83	1.38	0.63
query24	7.31	0.92	0.76
query25	0.38	0.18	0.23
query26	0.62	0.17	0.13
query27	0.07	0.07	0.05
query28	8.96	1.35	0.93
query29	12.66	3.97	3.31
query30	0.28	0.14	0.12
query31	2.83	0.59	0.38
query32	3.24	0.56	0.47
query33	3.03	3.12	3.05
query34	16.01	5.49	4.89
query35	4.90	4.90	4.85
query36	0.70	0.51	0.50
query37	0.10	0.07	0.08
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.15
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 103.75 s
Total hot run time: 30.65 s

@doris-robot
Copy link

doris-robot commented Oct 9, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.49% (17709/33739)
Line Coverage 37.66% (160772/426879)
Region Coverage 32.15% (122788/381957)
Branch Coverage 33.55% (53855/160539)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 9, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.18% (23529/33057)
Line Coverage 57.60% (245643/426448)
Region Coverage 52.80% (204231/386785)
Branch Coverage 54.53% (87987/161350)

dataroaring
dataroaring approved these changes Oct 10, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 10, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 10, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 10, 2025

PR approved by anyone and no changes requested.

@w41ter w41ter merged commit bf943cc into apache:master Oct 10, 2025
29 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 10, 2025
@mymeiyi
[fix](mow) delete bitmap is not deleted if commit compaction job fail…
29cb54b 
…ed (#56758)

delete bitmap kv is leaked:
```
[delete bitmap check fails] can't find corresponding rowset for delete bitmap instance_id=113100978, tablet_id=1759979382398, rowset_id=0200000000000027bc46eb5855e1afdc75b79046710c329f, version=13, segment_id=0
```
this is because: 
1. when update delete bitmap for  compaction, does not write pending key
2. when commit compaction job failed, the delete bitmap kv is not
deleted
3. when delete tmp rowset kv, the delete bitmap kv is not deleted
@github-actions github-actions bot mentioned this pull request Oct 10, 2025
Merged
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Oct 10, 2025
@mymeiyi
[fix](mow) delete bitmap is not deleted if commit compaction job fail…
d951790 
…ed (apache#56758)

delete bitmap kv is leaked:
```
[delete bitmap check fails] can't find corresponding rowset for delete bitmap instance_id=113100978, tablet_id=1759979382398, rowset_id=0200000000000027bc46eb5855e1afdc75b79046710c329f, version=13, segment_id=0
```
this is because:
1. when update delete bitmap for  compaction, does not write pending key
2. when commit compaction job failed, the delete bitmap kv is not
deleted
3. when delete tmp rowset kv, the delete bitmap kv is not deleted
@mymeiyi mymeiyi mentioned this pull request Oct 10, 2025
Merged
yiguolei pushed a commit that referenced this pull request Oct 11, 2025
@github-actions @mymeiyi
branch-4.0: [fix](mow) delete bitmap is not deleted if commit compact…
1bd31df 
…ion job failed #56758 (#56779)

Cherry-picked from #56758

Co-authored-by: meiyi <meiyi@selectdb.com>
morrySnow pushed a commit that referenced this pull request Oct 13, 2025
@mymeiyi
branch-3.1: [fix](mow) delete bitmap is not deleted if commit compact…
0b4b731 
…ion job failed #56758 (#56785)

picked from #56758
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@w41ter w41ter w41ter approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.2-merged dev/4.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@mymeiyi @Thearas @doris-robot @hello-stephen @w41ter @dataroaring @yiguolei @morrySnow