Skip to content

storage: GC segments contained by the pack (#6010)#6011

Merged
JaySon-Huang merged 1 commit intopingcap:release-6.3from
breezewish:wenxuan/gc-6.3
Sep 22, 2022
Merged

storage: GC segments contained by the pack (#6010)#6011
JaySon-Huang merged 1 commit intopingcap:release-6.3from
breezewish:wenxuan/gc-6.3

Conversation

@breezewish
Copy link
Member

@breezewish breezewish commented Sep 22, 2022

What problem does this PR solve?

Issue Number: close #5975

Cherry pick from #6010

What is changed and how it works?

Previously, we count segment rows by using the sum rows of overlapped packs. When the segment rows are < 0.7 rows of DTFiles, the DTFile will be regarded as "reclaimable" and the stable will be re-generated.

In this PR, we count segment rows by using the sum rows of fully contained packs. This means, the calculated segment rows will be smaller than before, and segments will more likely being GCed.

Here is an illustration of the new contained rule:

image

Consider the case where there is only 1 pack in the DTFile. When segment is fully or partially contained by this pack, the segment will be GCed, no matter how many actual rows are there in the segment. This leads to extra GCs in some cases, but the negative impact is small, as the cost of rewriting 1 pack should be small.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@breezewish
storage: GC segments contained by the pack (pingcap#6010)
418fde3 
Signed-off-by: Wish <breezewish@outlook.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 22, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • JaySon-Huang
  • flowbehappy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/cherry-pick-not-approved size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 22, 2022
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 22, 2022
@flowbehappy flowbehappy self-requested a review September 22, 2022 15:10
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 22, 2022
@JaySon-Huang
Copy link
Contributor

JaySon-Huang commented Sep 22, 2022

/merge

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 22, 2022

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 22, 2022

This pull request has been accepted and is ready to merge.

DetailsCommit hash: 418fde3

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 22, 2022
@sre-bot sre-bot added the cherry-pick-approved Cherry pick PR approved by release team. label Sep 22, 2022
@sre-bot
Copy link
Collaborator

sre-bot commented Sep 22, 2022

Coverage for changed files

Filename                                                        Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/FailPoint.cpp                                                608               221    63.65%           8                 2    75.00%          84                36    57.14%         206               108    47.57%
Storages/DeltaMerge/DeltaMergeStore.cpp                             897               156    82.61%          46                 2    95.65%        1312               229    82.55%         470               152    67.66%
Storages/DeltaMerge/DeltaMergeStore_InternalBg.cpp                  510               180    64.71%          16                 1    93.75%         630               152    75.87%         228                87    61.84%
Storages/DeltaMerge/Segment.cpp                                    1141               276    75.81%          55                 1    98.18%        1348               115    91.47%         358               103    71.23%
Storages/DeltaMerge/StableValueSpace.cpp                            136                23    83.09%          18                 1    94.44%         332                48    85.54%          90                25    72.22%
Storages/DeltaMerge/StableValueSpace.h                               19                 5    73.68%          16                 2    87.50%          32                18    43.75%           2                 2     0.00%
Storages/DeltaMerge/tests/gtest_dm_delta_merge_store.cpp            908               335    63.11%          68                 1    98.53%        2822                69    97.55%         264               183    30.68%
Storages/DeltaMerge/tests/gtest_dm_segment.cpp                      402               131    67.41%          36                 0   100.00%        1134                 6    99.47%         164                77    53.05%
Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.cpp         238                36    84.87%          20                 1    95.00%         277                12    95.67%          66                29    56.06%
Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.h             2                 0   100.00%           2                 0   100.00%           6                 0   100.00%           0                 0         -
Storages/DeltaMerge/tests/gtest_dm_store_background.cpp             235                85    63.83%          30                 0   100.00%         328                 4    98.78%          66                50    24.24%
Storages/DeltaMerge/tests/gtest_segment.cpp                         836               280    66.51%          46                 0   100.00%         877                23    97.38%         236               178    24.58%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                              5932              1728    70.87%         361                11    96.95%        9182               712    92.25%        2150               994    53.77%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18911      8115             57.09%    220105  83604        62.02%

full coverage report (for internal network access only)

@JaySon-Huang JaySon-Huang merged commit 2dfb554 into pingcap:release-6.3 Sep 22, 2022
@breezewish breezewish deleted the wenxuan/gc-6.3 branch September 22, 2022 16:18
@VelocityLight VelocityLight added do-not-merge/cherry-pick-not-approved and removed cherry-pick-approved Cherry pick PR approved by release team. labels Sep 28, 2022
@VelocityLight VelocityLight added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Nov 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@flowbehappy flowbehappy flowbehappy approved these changes

@JaySon-Huang JaySon-Huang JaySon-Huang approved these changes

Assignees

No one assigned

Labels

cherry-pick-approved Cherry pick PR approved by release team. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/cherry-pick-for-release-6.3

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@breezewish @ti-chi-bot @JaySon-Huang @sre-bot @flowbehappy @VelocityLight