Skip to content

Make the max expected number of legacy files and max valid rate configurable (#2187)#2205

Merged
JaySon-Huang merged 2 commits intopingcap:release-5.1from
ti-srebot:release-5.1-4290c0b5308b
Jun 28, 2021
Merged

Make the max expected number of legacy files and max valid rate configurable (#2187)#2205
JaySon-Huang merged 2 commits intopingcap:release-5.1from
ti-srebot:release-5.1-4290c0b5308b

Conversation

@ti-srebot
Copy link
Collaborator

@ti-srebot ti-srebot commented Jun 18, 2021

cherry-pick #2187 to release-5.1
You can switch your code base to this Pull Request by using git-extras:

# In tics repo:
git pr https://github.com/pingcap/tics/pull/2205

After apply modifications, you can push your change to this PR via:

git push git@github.com:ti-srebot/tics.git pr/2205:release-5.1-4290c0b5308b

Signed-off-by: JaySon-Huang jayson.hjs@gmail.com

What problem does this PR solve?

Issue Number: related to #2169 , #1552

Problem Summary:
In #1552, we introduce an adaptive aggressive GC strategy for PageStorage. If there are more than 100 legacy files, we will set the gc_max_valid_rate to 1.0 try to compact all PageFiles even if there are no invalid pages inside one PageFile.
It brings some trouble in extreme cases like #2169 more worst and generates more write flow.

What is changed and how it works?

  • Add two (no public) variables dt_page_num_max_expect_legacy_files and dt_page_num_max_gc_valid_rate to make then configurable.
  • Change the default value of gc_max_valid_rate_bound from 1.00 to 0.95
  • Apply those variables for RegionPersister

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch:

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

Release note

  • No release note

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot added CHERRY-PICK cherry pick status/LGT1 Indicates that a PR has LGTM 1. type/bugfix This PR fixes a bug. labels Jun 18, 2021
@ti-srebot ti-srebot requested a review from flowbehappy June 18, 2021 04:51
@ti-srebot ti-srebot added this to the v5.1.0 milestone Jun 18, 2021
Copy link
Contributor

@flowbehappy flowbehappy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JaySon-Huang JaySon-Huang modified the milestones: v5.1.0, v5.1.1 Jun 24, 2021
@JaySon-Huang
Copy link
Contributor

/merge

@JaySon-Huang
Copy link
Contributor

/run-all-tests

@JaySon-Huang JaySon-Huang merged commit a425943 into pingcap:release-5.1 Jun 28, 2021
@JaySon-Huang JaySon-Huang deleted the release-5.1-4290c0b5308b branch June 28, 2021 07:24
LittleFall pushed a commit that referenced this pull request Jun 28, 2021
…gurable (#2187) #2205

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: JaySon <tshent@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CHERRY-PICK cherry pick status/LGT1 Indicates that a PR has LGTM 1. type/bugfix This PR fixes a bug.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants