raftstore: report busy to PD when restarting if exists apply log lags. by LykxSassinator · Pull Request #16239 · tikv/tikv

LykxSassinator · 2023-12-26T03:12:28Z

What is changed and how it works?

Issue Number: Ref #15874

What's Changed:

This pr used to mitigate the dropping trend when rolling restart TiKV nodes, derived from #11859 and #11921.

This pr inspect the gap of each peer's `applied_log_index` and `commit_log_index` when restarting.

And if the gap exceeds the `leader_transfer_max_log_lag`, the related peer will be marked
as `pending for recovery` state. After the gap is less than `leader_transfer_max_log_lag`,
it means that the pending logs is acceptable.

Only if the count of ready peers exceeds the given configuration, that is,
`min_recovery_ready_region_percent`, this store is ready for re-balancing leaders. Before
this stage, the state of this store will be marked `is_busy` to avoid transferring leaders to it.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Release note

None.

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

ti-chi-bot · 2023-12-26T03:12:30Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Connor1996
overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

ti-chi-bot · 2024-01-13T08:52:27Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

Derived from tikv#11859. Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

overvenus · 2024-02-02T04:16:01Z

/merge

ti-chi-bot · 2024-02-02T04:16:03Z

@overvenus: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2024-02-02T04:16:05Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: 3064494

tikv#16239) ref tikv#15874 This pr inspect the gap of each peer's `applied_log_index` and `commit_log_index` when restarting. And if the gap exceeds the `leader_transfer_max_log_lag`, the related peer will be marked as `pending for recovery` state. After the gap is less than `leader_transfer_max_log_lag`, it means that the pending logs is acceptable. Only if the count of ready peers exceeds the given configuration, that is, `min_recovery_ready_region_percent`, this store is ready for re-balancing leaders. Before this stage, the state of this store will be marked `is_busy` to avoid transferring leaders to it. Signed-off-by: lucasliang <nkcs_lykx@hotmail.com> Signed-off-by: dbsid <chenhuansheng@pingcap.com>

tikv#16239) ref tikv#15874 This pr inspect the gap of each peer's `applied_log_index` and `commit_log_index` when restarting. And if the gap exceeds the `leader_transfer_max_log_lag`, the related peer will be marked as `pending for recovery` state. After the gap is less than `leader_transfer_max_log_lag`, it means that the pending logs is acceptable. Only if the count of ready peers exceeds the given configuration, that is, `min_recovery_ready_region_percent`, this store is ready for re-balancing leaders. Before this stage, the state of this store will be marked `is_busy` to avoid transferring leaders to it. Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

ti-chi-bot · 2024-07-30T02:38:29Z

In response to a cherrypick label: new pull request created to branch release-7.5: #17328.

ref tikv#15874 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2024-07-30T02:38:30Z

In response to a cherrypick label: new pull request created to branch release-7.1: #17329.

ref #15874 This pr is used to cp #16239 & #16494 & #16738 to v7.1, used to inspect the gap of each peer's `applied_log_index` and `commit_log_index` when restarting. Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

raftstore: delay the tick of raft when applying raft logs.

a8cd38b

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

LykxSassinator added 4 commits December 26, 2023 11:45

Fix clippy warnings.

2efa258

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

Bugfix for the missing RaftTick.

760b55f

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

Merge branch 'master' into delay_raft_hb

6b46df3

Backport from "raftstore: Report is apply busy to PD tikv#11921."

71eeb25

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 13, 2024

LykxSassinator added 2 commits January 17, 2024 04:46

Merge branch 'master' into delay_raft_hb

cfb72bf

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

Add log lag check for updating is_busy when recovery.

1c50bd9

Derived from tikv#11859. Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 17, 2024

Revert "report apply status".

bff1c2b

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 17, 2024

LykxSassinator added 2 commits January 17, 2024 08:45

Fix clippy errors.

c5e84fa

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

Revert unnecessary annotations.

f862ce9

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

LykxSassinator changed the title ~~raftstore: delay the tick of raft when applying raft logs.~~ raftstore: report busy to PD when restarting if exists apply log lags. Jan 18, 2024

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/needs-linked-issue do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jan 18, 2024

Merge branch 'master' into delay_raft_hb

08757c4

LykxSassinator removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 18, 2024

LykxSassinator added 3 commits January 19, 2024 16:36

Merge branch 'master' into delay_raft_hb

8e3c2bd

Merge branch 'master' into delay_raft_hb

fb07dbe

Merge branch 'master' into delay_raft_hb

25ea268

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Feb 2, 2024

ti-chi-bot bot merged commit 997eabc into tikv:master Feb 2, 2024

ti-chi-bot bot added this to the Pool milestone Feb 2, 2024

LykxSassinator deleted the delay_raft_hb branch February 2, 2024 07:26

This was referenced Apr 11, 2024

raftstore: bugfix of the calculation on the log gap between leader and peer when restart. #16738

Merged

raftstore: the calculation of unapplied index is not accurate about busy_on_apply state. #16800

Closed

ti-chi-bot mentioned this pull request May 8, 2024

raftstore: bugfix of the calculation on the log gap between leader and peer when restart. (#16738) #16961

Merged

9 tasks

LykxSassinator mentioned this pull request Jun 5, 2024

TiKV restart process optimization #15874

Closed

3 tasks

LykxSassinator mentioned this pull request Jul 29, 2024

raftstore: cherry-pick #16239 & #16494 & #16738 to v7.1 #17318

Merged

9 tasks

LykxSassinator added a commit to LykxSassinator/tikv that referenced this pull request Jul 29, 2024

raftstore: cherry-pick tikv#16239 & tikv#16494 & tikv#16738 to v7.5

3b1f30a

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

LykxSassinator mentioned this pull request Jul 29, 2024

raftstore: cherry-pick #16239 & #16494 & #16738 to v7.5. #17319

Merged

9 tasks

ti-chi-bot added needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. labels Jul 30, 2024

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Jul 30, 2024

This is an automated cherry-pick of tikv#16239

e515b99

ref tikv#15874 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Jul 30, 2024

This is an automated cherry-pick of tikv#16239

8663b7b

ref tikv#15874 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Jul 30, 2024

raftstore: report busy to PD when restarting if exists apply log lags. (#16239) #17328

Closed

9 tasks

ti-chi-bot mentioned this pull request Jul 30, 2024

raftstore: report busy to PD when restarting if exists apply log lags. (#16239) #17329

Closed

9 tasks

LykxSassinator mentioned this pull request Feb 20, 2025

raftstore: avoid early hibernate if pending on applying logs when restart #18236

Merged

9 tasks

ti-chi-bot mentioned this pull request Apr 14, 2025

raftstore: avoid early hibernate if pending on applying logs when restart (#18236) #18393

Merged

9 tasks

This was referenced Jun 30, 2025

raftstore: avoid early hibernate if pending on applying logs when restart (#18236) #18601

Merged

raftstore: avoid early hibernate if pending on applying logs when restart (#18236) #18602

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

raftstore: report busy to PD when restarting if exists apply log lags.#16239

raftstore: report busy to PD when restarting if exists apply log lags.#16239
ti-chi-bot[bot] merged 40 commits intotikv:masterfrom
LykxSassinator:delay_raft_hb

LykxSassinator commented Dec 26, 2023 •

edited

Loading

Uh oh!

ti-chi-bot bot commented Dec 26, 2023 •

edited

Loading

Uh oh!

ti-chi-bot bot commented Jan 13, 2024

Uh oh!

overvenus commented Feb 2, 2024

Uh oh!

ti-chi-bot bot commented Feb 2, 2024

Uh oh!

ti-chi-bot bot commented Feb 2, 2024

Uh oh!

ti-chi-bot commented Jul 30, 2024

Uh oh!

ti-chi-bot commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

LykxSassinator commented Dec 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is changed and how it works?

Related changes

Check List

Release note

Uh oh!

ti-chi-bot bot commented Dec 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ti-chi-bot bot commented Jan 13, 2024

Uh oh!

overvenus commented Feb 2, 2024

Uh oh!

ti-chi-bot bot commented Feb 2, 2024

Uh oh!

ti-chi-bot bot commented Feb 2, 2024

Uh oh!

ti-chi-bot commented Jul 30, 2024

Uh oh!

ti-chi-bot commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

LykxSassinator commented Dec 26, 2023 •

edited

Loading

ti-chi-bot bot commented Dec 26, 2023 •

edited

Loading