Skip to content

PageStorage: Fix pages are not deleted under some cases#5069

Merged
ti-chi-bot merged 4 commits intopingcap:masterfrom
JaySon-Huang:fix_page_undeleted
Jun 3, 2022
Merged

PageStorage: Fix pages are not deleted under some cases#5069
ti-chi-bot merged 4 commits intopingcap:masterfrom
JaySon-Huang:fix_page_undeleted

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Jun 3, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: close #5054

Problem Summary: as the issue described. V2.data.getAliveExternalPageIds returns the dtfile ids after deleted, so the deleted dtfiles can not be removed from disk. Restart the TiFlash process can workaround the issue as the page ids are deleted when restored from v2 wal.

What is changed and how it works?

  • StoragePool only need to remove DTFiles ids that are not exist in the V3.data, cause we perform force transform from V2.data to V3.data when restore
  • Keep tombstone when merging the delta version for page entries in V2, or the page id will shown in getAliveExternalPageIds after deleted

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code
  • Manual test steps:
    • Deploy cluster and load some data into TiFlash with storage.format_version = 3
    • Change the storage.format_version = 4
    • Use alter table t compact tiflash replica to perform delta-merge
    • Check that the old DTFiles are removed

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@JaySon-Huang
Add debug log and ut
02bfc09 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
keep tombstone when merging v2 in-mem delta versions
5dc7699 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Remove debug lines
50a7d30 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Remove useless callback in mixed mode
68e5d85 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jun 3, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • flowbehappy
  • hehechen

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. and removed do-not-merge/needs-linked-issue labels Jun 3, 2022
@JaySon-Huang JaySon-Huang mentioned this pull request Jun 3, 2022
Merged
12 tasks
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 3, 2022
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jun 3, 2022

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Jun 3, 2022

Coverage for changed files

Filename                                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/StoragePool.cpp                            223                98    56.05%          30                 8    73.33%         478               187    60.88%         146                78    46.58%
Page/Page.h                                            61                 6    90.16%          15                 1    93.33%          80                17    78.75%          38                12    68.42%
Page/V2/PageEntries.h                                 128                14    89.06%          49                11    77.55%         300                54    82.00%          64                 5    92.19%
Page/V3/tests/gtest_page_storage_mix_mode.cpp        2854               765    73.20%          22                 0   100.00%         832                21    97.48%         820               422    48.54%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                3266               883    72.96%         116                20    82.76%        1690               279    83.49%        1068               517    51.59%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18282      9732             46.77%    205081  97591        52.41%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 3, 2022
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jun 3, 2022

/merge

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jun 3, 2022

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jun 3, 2022

This pull request has been accepted and is ready to merge.

DetailsCommit hash: 68e5d85

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 3, 2022
@sre-bot
Copy link
Collaborator

sre-bot commented Jun 3, 2022

Coverage for changed files

Filename                                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/StoragePool.cpp                            223                98    56.05%          30                 8    73.33%         478               187    60.88%         146                78    46.58%
Page/Page.h                                            61                 6    90.16%          15                 1    93.33%          80                17    78.75%          38                12    68.42%
Page/V2/PageEntries.h                                 128                14    89.06%          49                11    77.55%         300                54    82.00%          64                 5    92.19%
Page/V3/tests/gtest_page_storage_mix_mode.cpp        2854               765    73.20%          22                 0   100.00%         832                21    97.48%         820               422    48.54%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                3266               883    72.96%         116                20    82.76%        1690               279    83.49%        1068               517    51.59%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18282      9732             46.77%    205081  97591        52.41%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot merged commit b49a787 into pingcap:master Jun 3, 2022
ti-chi-bot pushed a commit that referenced this pull request Jun 3, 2022
@JaySon-Huang
PageStorage: Fix pages are not deleted under some cases (#5069) (#5070)
df059cc 
close #5054
ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Jun 3, 2022
@JaySon-Huang @ti-chi-bot
PageStorage: Fix pages are not deleted under some cases (pingcap#5069)
bf315ca 
close pingcap#5054
@ti-chi-bot ti-chi-bot mentioned this pull request Jun 3, 2022
Closed
12 tasks
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jun 3, 2022

In response to a cherrypick label: new pull request created: #5071.

@JaySon-Huang JaySon-Huang deleted the fix_page_undeleted branch June 3, 2022 08:20
@sre-bot
Copy link
Collaborator

sre-bot commented Jun 3, 2022

Coverage for changed files

Filename                                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/StoragePool.cpp                            223                98    56.05%          30                 8    73.33%         478               187    60.88%         146                78    46.58%
Page/Page.h                                            61                 6    90.16%          15                 1    93.33%          80                17    78.75%          38                12    68.42%
Page/V2/PageEntries.h                                 128                14    89.06%          49                11    77.55%         300                54    82.00%          64                 5    92.19%
Page/V3/tests/gtest_page_storage_mix_mode.cpp        2854               765    73.20%          22                 0   100.00%         832                21    97.48%         820               422    48.54%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                3266               883    72.96%         116                20    82.76%        1690               279    83.49%        1068               517    51.59%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18282      9732             46.77%    205081  97618        52.40%

full coverage report (for internal network access only)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@flowbehappy flowbehappy flowbehappy approved these changes

@jiaqizho jiaqizho Awaiting requested review from jiaqizho

+1 more reviewer

@hehechen hehechen hehechen approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

The store size of TiFlash unexpectedly increases with update sql

5 participants

@JaySon-Huang @ti-chi-bot @sre-bot @flowbehappy @hehechen