Skip to content

[hotfix-to-7.5.6] raftstore: remove stale ranges by DeleteByKeys rather than ingesting. (#18040)#18519

Merged
LykxSassinator merged 2 commits intotikv:add-index-ingest-allow-writefrom
LykxSassinator:cp_pr2_to_hotfix
Jun 6, 2025
Merged

[hotfix-to-7.5.6] raftstore: remove stale ranges by DeleteByKeys rather than ingesting. (#18040)#18519
LykxSassinator merged 2 commits intotikv:add-index-ingest-allow-writefrom
LykxSassinator:cp_pr2_to_hotfix

Conversation

@LykxSassinator
Copy link
Contributor

@LykxSassinator LykxSassinator commented Jun 6, 2025

This is a manual cherry-pick of #18040

What is changed and how it works?

Issue Number: Close #18107, Ref #18042

This PR mainly contains the following parts for optimization on scaling, used to mitigate the impacts, introduced by unnecessary ingesting sst files:

  • Directly clearing stale ranges by DeleteByKeys during the balancing regions process.
  • Do not clear data of offline stores during the scale-in process, as this data will be automatically cleared when the corresponding node goes offline.

What's Changed:

Optimizing the processing of clearing stale-ranges by DeleteByKeys, rather than ingesting.

This optimization is used to mitigate the impacts, introduced by unnecessary ingesting sst
files on latency when scaling.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Taking the grpc messages duration in TiKV metrics panel as examples, positive performance feedbacks on reducing the long-tail latency can be reviewed from the following comparison results:

Commit Comparisons on grpc messages duration
master image
This PR image

Meanwhile, the following E2E long-tail reduction also proves that this PR makes positive improvements effects:

Commit Scale-Out (Before -> Scaling) Scale-in (Before -> Scaling)
master P99: 7.2ms -> ~ 11.3 ms P999: 62ms -> ~89.2 ms P99: 7.2ms -> ~ 16.1 ms P999: 62ms -> ~100 ms
This PR P99: 7.2 ms -> ~10.6 ms P999: 62 ms -> ~76.9ms P99: 7.2ms -> ~ 15.3 ms P999: 62ms -> ~96 ms

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Release note

Optimizing the processing of clearing stale-ranges by DeleteByKeys to mitigate the impacts
on latency.

@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has signed the dco. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 6, 2025
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 6, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jun 6, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-06-06 10:06:27.239064235 +0000 UTC m=+3965.467379519: ☑️ agreed by overvenus.

@ti-chi-bot ti-chi-bot bot added the approved label Jun 6, 2025
@LykxSassinator
Merge branch 'add-index-ingest-allow-write' into cp_pr2_to_hotfix
3ca1619 
Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>
@LykxSassinator
Copy link
Contributor Author

LykxSassinator commented Jun 6, 2025

/merge

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jun 6, 2025

@LykxSassinator: We have migrated to builtin LGTM and approve plugins for reviewing.

👉 Please use /approve when you want approve this pull request.

The changes announcement: Proposal: Strengthen configuration change approval.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@LykxSassinator
Copy link
Contributor Author

LykxSassinator commented Jun 6, 2025

/approve

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jun 6, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: LykxSassinator, overvenus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [LykxSassinator,overvenus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@LykxSassinator LykxSassinator merged commit 6a80753 into tikv:add-index-ingest-allow-write Jun 6, 2025
2 of 3 checks passed
@LykxSassinator LykxSassinator deleted the cp_pr2_to_hotfix branch June 6, 2025 10:56
ti-chi-bot bot pushed a commit that referenced this pull request Jun 10, 2025
@overvenus @LykxSassinator
Rollup of ingest sst optimizations (#18521)
7852e09 
 

Rollup of following sst ingestion optimizations

* raftstore: directly write kvs rather than ingestion when merging small regions. (#17408) (#18518)
* raftstore: remove stale ranges by DeleteByKeys rather than ingesting. (#18040) (#18519)
* raftstore: support rocksdb writes during ingestion #18096 (#18520)
* storage: support online config for flow-control module (#17396). (#18523)
* sst_importer: allow write during ingesting sst (#18514) (#18522)

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>
Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: lucasliang <nkcs_lykx@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@overvenus overvenus overvenus approved these changes

Assignees

No one assigned

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. needs-1-more-lgtm Indicates a PR needs 1 more LGTM. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@LykxSassinator @overvenus