Skip to content

Make the max expected number of legacy files and max valid rate configurable#2187

Merged
ti-srebot merged 4 commits intopingcap:masterfrom
JaySon-Huang:fix_page_file_duplicated
Jun 18, 2021
Merged

Make the max expected number of legacy files and max valid rate configurable#2187
ti-srebot merged 4 commits intopingcap:masterfrom
JaySon-Huang:fix_page_file_duplicated

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Jun 17, 2021

Signed-off-by: JaySon-Huang jayson.hjs@gmail.com

What problem does this PR solve?

Issue Number: related to #2169 , #1552

Problem Summary:
In #1552, we introduce an adaptive aggressive GC strategy for PageStorage. If there are more than 100 legacy files, we will set the gc_max_valid_rate to 1.0 try to compact all PageFiles even if there are no invalid pages inside one PageFile.
It brings some trouble in extreme cases like #2169 more worst and generates more write flow.

What is changed and how it works?

  • Add two (no public) variables dt_page_num_max_expect_legacy_files and dt_page_num_max_gc_valid_rate to make then configurable.
  • Change the default value of gc_max_valid_rate_bound from 1.00 to 0.95
  • Apply those variables for RegionPersister

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch:

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

Release note

  • No release note

@JaySon-Huang JaySon-Huang added the type/bugfix This PR fixes a bug. label Jun 17, 2021
@JaySon-Huang JaySon-Huang requested a review from flowbehappy June 17, 2021 08:14
@JaySon-Huang JaySon-Huang self-assigned this Jun 17, 2021
@JaySon-Huang JaySon-Huang mentioned this pull request Jun 17, 2021
Closed
@JaySon-Huang
Make the max expected number of legacy files and max valid rate confi…
7012a82 
…gurable

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Fix link error
da40bde 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang JaySon-Huang force-pushed the fix_page_file_duplicated branch from a4455f8 to da40bde Compare June 17, 2021 10:35
@JaySon-Huang
Add missing pragma once
5913b1c 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
flowbehappy
flowbehappy approved these changes Jun 18, 2021
View reviewed changes
Copy link
Contributor

@flowbehappy flowbehappy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 18, 2021
@flowbehappy flowbehappy added needs-cherry-pick-release-5.0 PR which needs to be cherry-picked to release-5.0 needs-cherry-pick-release-5.1 PR which needs to be cherry-picked to release-5.1 labels Jun 18, 2021
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jun 18, 2021

/merge

@ti-srebot
Copy link
Collaborator

ti-srebot commented Jun 18, 2021

Your auto merge job has been accepted, waiting for:

  • 2173

@ti-srebot ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 18, 2021
@ti-srebot
Copy link
Collaborator

ti-srebot commented Jun 18, 2021

/run-all-tests

@ti-srebot ti-srebot merged commit 4290c0b into pingcap:master Jun 18, 2021
@ti-srebot ti-srebot mentioned this pull request Jun 18, 2021
Merged
@ti-srebot
Copy link
Collaborator

ti-srebot commented Jun 18, 2021

cherry pick to release-5.0 in PR #2204

@ti-srebot ti-srebot mentioned this pull request Jun 18, 2021
Merged
@ti-srebot
Copy link
Collaborator

ti-srebot commented Jun 18, 2021

cherry pick to release-5.1 in PR #2205

@JaySon-Huang JaySon-Huang deleted the fix_page_file_duplicated branch June 18, 2021 05:01
@JaySon-Huang JaySon-Huang added the needs-cherry-pick-release-4.0 PR which needs to be cherry-picked to release-4.0 label Jun 18, 2021
@ti-srebot ti-srebot mentioned this pull request Jun 18, 2021
Merged
@ti-srebot
Copy link
Collaborator

ti-srebot commented Jun 18, 2021

cherry pick to release-4.0 in PR #2206

ti-srebot added a commit that referenced this pull request Jun 23, 2021
@ti-srebot
Make the max expected number of legacy files and max valid rate confi…
ee59f2c 
…gurable (#2187) (#2204)
JaySon-Huang added a commit that referenced this pull request Jun 28, 2021
@ti-srebot @JaySon-Huang
Make the max expected number of legacy files and max valid rate confi…
a425943 
…gurable (#2187) #2205

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: JaySon <tshent@qq.com>
LittleFall pushed a commit that referenced this pull request Jun 28, 2021
@ti-srebot @JaySon-Huang @LittleFall
Make the max expected number of legacy files and max valid rate confi…
db04580 
…gurable (#2187) #2205

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: JaySon <tshent@qq.com>
ti-srebot added a commit that referenced this pull request Jul 1, 2021
@ti-srebot
Make the max expected number of legacy files and max valid rate confi…
25ef6d1 
…gurable (#2187) (#2206)
@JaySon-Huang JaySon-Huang mentioned this pull request Jul 2, 2021
Merged
This was referenced Jul 6, 2021
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@flowbehappy flowbehappy flowbehappy approved these changes

Assignees

@JaySon-Huang JaySon-Huang

Labels

needs-cherry-pick-release-4.0 PR which needs to be cherry-picked to release-4.0 needs-cherry-pick-release-5.0 PR which needs to be cherry-picked to release-5.0 needs-cherry-pick-release-5.1 PR which needs to be cherry-picked to release-5.1 status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1. type/bugfix This PR fixes a bug.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JaySon-Huang @ti-srebot @flowbehappy