Skip to content

Use an adaptive aggressive GC strategy on Delta (PageStorage)#1552

Merged
ti-srebot merged 11 commits intopingcap:masterfrom
JaySon-Huang:page_storage_ctl
Mar 12, 2021
Merged

Use an adaptive aggressive GC strategy on Delta (PageStorage)#1552
ti-srebot merged 11 commits intopingcap:masterfrom
JaySon-Huang:page_storage_ctl

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Mar 12, 2021 •
edited
Loading

What problem does this PR solve?

Issue Number: close #1550

Problem Summary: TiFlash OOM cause by too many legacy PageFile are not GC

What is changed and how it works?

  • Add some settings to make TiFlash can overwrite gc threshold of PageFile
  • Try to do more aggressive GC when there are many Legacy PageFiles
  • Add a build target page_ctl to do manual gc offline

Related changes

  • Need to cherry-pick to the release branch 4.0

Check List

Tests

  • Manual test (add detailed scripts or steps below)

Side effects

  • N/A

Release note

  • Use an adaptive aggressive data GC strategy to avoid OOM

@JaySon-Huang JaySon-Huang added the needs-cherry-pick-release-4.0 PR which needs to be cherry-picked to release-4.0 label Mar 12, 2021
@JaySon-Huang JaySon-Huang self-assigned this Mar 12, 2021
@JaySon-Huang JaySon-Huang changed the title Page storage ctl Try to do more aggressive GC on Delta (PageStorage) Mar 12, 2021
@JaySon-Huang
Add a ctl to do gc on page-storage data
fbb2530 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Add settings to control page-storage gc
8892f32 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Try to reload page_storage GC settings
43fffce 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Try to do more aggressive compaction
80cb63f 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Rename struct
63e20ed 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
fix bug
2c34c93 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Adjust the threshold
e292678 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang JaySon-Huang changed the title Try to do more aggressive GC on Delta (PageStorage) Use an adaptive aggressive GC strategy on Delta (PageStorage) Mar 12, 2021
@JaySon-Huang JaySon-Huang added the type/bugfix This PR fixes a bug. label Mar 12, 2021
@JaySon-Huang
Remove archived file
b20c8e0 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Mar 12, 2021

/run-all-tests

flowbehappy
flowbehappy approved these changes Mar 12, 2021
View reviewed changes
Copy link
Contributor

@flowbehappy flowbehappy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 12, 2021
JaySon-Huang and others added 3 commits March 12, 2021 23:25
@JaySon-Huang
Fix compile
67d53fb 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Merge branch 'master' into page_storage_ctl
3c3982c
@JaySon-Huang
Make removing Legacy PageFile atomic
17c16a5 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang JaySon-Huang added the status/can-merge Indicates a PR has been approved by a committer. label Mar 12, 2021
@ti-srebot
Copy link
Collaborator

ti-srebot commented Mar 12, 2021

/run-all-tests

@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Mar 12, 2021

/rebuild

@ti-srebot ti-srebot merged commit 1769158 into pingcap:master Mar 12, 2021
@JaySon-Huang JaySon-Huang deleted the page_storage_ctl branch March 12, 2021 20:01
@ti-srebot ti-srebot mentioned this pull request Mar 12, 2021
Merged
@ti-srebot
Copy link
Collaborator

ti-srebot commented Mar 12, 2021

cherry pick to release-4.0 in PR #1553

JaySon-Huang added a commit that referenced this pull request Mar 15, 2021
@ti-srebot @JaySon-Huang
Use an adaptive aggressive GC strategy on Delta (PageStorage) (#1552) (…
5782d41 
…#1553)

* cherry pick #1552 to release-4.0
* Fix conflict

Co-authored-by: JaySon-Huang <jayson.hjs@gmail.com>
LittleFall referenced this pull request in LittleFall/tics Mar 21, 2021
@ti-srebot @JaySon-Huang @LittleFall
Use an adaptive aggressive GC strategy on Delta (PageStorage) (#1552)…
081454c 
… (#1553)

* cherry pick #1552 to release-4.0
* Fix conflict

Co-authored-by: JaySon-Huang <jayson.hjs@gmail.com>
This was referenced Mar 23, 2021
Closed
Closed
This was referenced Jun 17, 2021
Merged
Closed
This was referenced Jun 18, 2021
Merged
Merged
Merged
@JaySon-Huang JaySon-Huang mentioned this pull request Jul 2, 2021
Merged
This was referenced Jul 6, 2021
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@flowbehappy flowbehappy flowbehappy approved these changes

Assignees

@JaySon-Huang JaySon-Huang

Labels

needs-cherry-pick-release-4.0 PR which needs to be cherry-picked to release-4.0 status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1. type/bugfix This PR fixes a bug.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

TiFlash OOM because of too many legacy PageFile are not GC

3 participants

@JaySon-Huang @ti-srebot @flowbehappy