Skip to content

schedule: revise region_scatter distribution for multi groups (#3422)#3620

Merged
Yisaer merged 5 commits intotikv:release-4.0from
ti-srebot:release-4.0-9e60f4d367e3
May 10, 2021
Merged

schedule: revise region_scatter distribution for multi groups (#3422)#3620
Yisaer merged 5 commits intotikv:release-4.0from
ti-srebot:release-4.0-9e60f4d367e3

Conversation

@ti-srebot
Copy link
Contributor

@ti-srebot ti-srebot commented Apr 21, 2021 •
edited
Loading

cherry-pick #3422 to release-4.0
You can switch your code base to this Pull Request by using git-extras:

# In pd repo:
git pr https://github.com/tikv/pd/pull/3620

After apply modifications, you can push your change to this PR via:

git push git@github.com:ti-srebot/pd.git pr/3620:release-4.0-9e60f4d367e3

Signed-off-by: Song Gao disxiaofei@163.com

What problem does this PR solve?

ref pingcap/tidb#22788

Currently region-scatter didn't work well for multi regions in different groups.

For example, there are 50 regions and 6 stores in the cluster. Each region belong to a dedicated group.

region scatter may locate them in the store1/store2/store3. Thus from each region in the group, the distribution is well. And for the whole cluster, the distribution is not well.

What is changed and how it works?

During region scatter, we should consider whether the target store already have the most peer or leader, if it is, we should skip it.

Check List

Tests

  • Unit test

I also tested with the testcases in this document: https://docs.google.com/document/d/1cGZU2DCm0UuIueqAQP8TGqTivyavCvB_QiwIzqvP_rA/edit

All the test case results didn't change.

Release note

  • No release note

@Yisaer @ti-srebot
cherry pick tikv#3422 to release-4.0
fa2e920 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Apr 21, 2021
Merged
@ti-srebot ti-srebot added component/cluster Cluster logic. status/LGT2 Indicates that a PR has LGTM 2. type/cherry-pick labels Apr 21, 2021
@ti-chi-bot ti-chi-bot removed the status/LGT2 Indicates that a PR has LGTM 2. label Apr 21, 2021
@ti-srebot ti-srebot added this to the v4.0.13 milestone Apr 21, 2021
@ti-chi-bot ti-chi-bot requested review from HunDunDM and Yisaer April 21, 2021 04:46
@ti-srebot
Copy link
Contributor Author

ti-srebot commented Apr 21, 2021

@Yisaer you're already a collaborator in bot's repo.

@nolouch
Copy link
Contributor

nolouch commented Apr 25, 2021

plz fix the conflict

@rleungx rleungx added type/cherry-pick-for-release-4.0 The PR belongs to release-4.0 cherry pick. and removed type/cherry-pick-for-release-1.0 labels Apr 25, 2021
@zhouqiang-cl zhouqiang-cl added the cherry-pick-approved Cherry pick PR approved by release team. label May 8, 2021
@zhouqiang-cl
Copy link
Contributor

zhouqiang-cl commented May 8, 2021

/merge

@ti-chi-bot
Copy link
Member

ti-chi-bot commented May 8, 2021

@zhouqiang-cl: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented May 8, 2021

@zhouqiang-cl: /merge in this pull request requires 2 /lgtm.

Details

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Yisaer added 4 commits May 10, 2021 21:32
@Yisaer
Merge remote-tracking branch 'upstream/release-4.0' into release-4.0-…
3a4afbb 
…9e60f4d367e3
@Yisaer
fix conflict
0f6e099 
Signed-off-by: yisaer <disxiaofei@163.com>
@Yisaer
remove test
2f80aa1 
Signed-off-by: yisaer <disxiaofei@163.com>
@Yisaer
remove test
5dc1b74 
Signed-off-by: yisaer <disxiaofei@163.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented May 10, 2021

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Yisaer

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@sonarqubecloud
Copy link

sonarqubecloud bot commented May 10, 2021

SonarCloud Quality Gate failed.

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 14 Code Smells

No Coverage information No Coverage information
3.1% 3.1% Duplication

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 10, 2021
@codecov
Copy link

codecov bot commented May 10, 2021

Codecov Report

Merging #3620 (5dc1b74) into release-4.0 (6b1f8ab) will decrease coverage by 0.03%.
The diff coverage is 92.59%.

Impacted file tree graph

@@               Coverage Diff               @@
##           release-4.0    #3620      +/-   ##
===============================================
- Coverage        74.60%   74.56%   -0.04%     
===============================================
  Files              211      211              
  Lines            19908    19933      +25     
===============================================
+ Hits             14852    14863      +11     
- Misses            3715     3728      +13     
- Partials          1341     1342       +1
Flag Coverage Δ
unittests 74.56% <92.59%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
server/schedule/region_scatterer.go 76.01% <90.47%> (+1.26%) ⬆️
pkg/cache/ttl.go 97.77% <100.00%> (+0.15%) ⬆️
server/schedulers/shuffle_hot_region.go 56.88% <0.00%> (-11.93%) ⬇️
server/kv/etcd_kv.go 72.46% <0.00%> (-7.25%) ⬇️
server/heartbeat_streams.go 65.93% <0.00%> (-6.60%) ⬇️
server/grpc_service.go 59.87% <0.00%> (-0.41%) ⬇️
server/server.go 73.47% <0.00%> (-0.33%) ⬇️
server/config/persist_options.go 87.92% <0.00%> (+0.75%) ⬆️
client/client.go 70.05% <0.00%> (+1.01%) ⬆️
server/tso/tso.go 78.98% <0.00%> (+1.44%) ⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b1f8ab...5dc1b74. Read the comment docs.

@Yisaer Yisaer merged commit b06e23e into tikv:release-4.0 May 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Yisaer Yisaer Yisaer approved these changes

@JmPotato JmPotato Awaiting requested review from JmPotato

@lhy1024 lhy1024 Awaiting requested review from lhy1024

@nolouch nolouch Awaiting requested review from nolouch

@rleungx rleungx Awaiting requested review from rleungx

@HunDunDM HunDunDM Awaiting requested review from HunDunDM

Assignees

@Yisaer Yisaer

Labels

cherry-pick-approved Cherry pick PR approved by release team. component/cluster Cluster logic. status/LGT1 Indicates that a PR has LGTM 1. type/cherry-pick-for-release-4.0 The PR belongs to release-4.0 cherry pick.

Projects

None yet

Milestone

v4.0.13

Development

Successfully merging this pull request may close these issues.

6 participants

@ti-srebot @nolouch @zhouqiang-cl @ti-chi-bot @Yisaer @rleungx