raftstore: add io throughput metrics for snapshot generation by hhwyt · Pull Request #18110 · tikv/tikv

hhwyt · 2025-01-09T12:25:43Z

This PR adds an I/O throughput metric(generate-io) for snapshot generation. It measures the read disk I/O throughput for RocksDB scans during the snapshot generation process.

What is changed and how it works?

Issue Number: Close #18140

What's Changed:

Add io throughput metrics for snapshot generation.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Release note

none

Signed-off-by: hhwyt <hhwyt1@gmail.com>

ti-chi-bot · 2025-01-09T12:25:53Z

Hi @hhwyt. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: hhwyt <hhwyt1@gmail.com>

hhwyt · 2025-01-20T00:50:50Z

@glorv @LykxSassinator PTAL, thx~

hhwyt · 2025-01-20T00:53:43Z

components/file_system/src/lib.rs

        assert!(!file.exists());
    }

+    #[cfg(feature = "failpoints")]


This PR also refactors code related to failpoint tests.

glorv

LGTM

ti-chi-bot · 2025-01-20T07:52:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: glorv, LykxSassinator

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [LykxSassinator,glorv]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-01-20T07:52:21Z

[LGTM Timeline notifier]

Timeline:

2025-01-20 03:40:37.240466623 +0000 UTC m=+65764.571386028: ☑️ agreed by glorv.
2025-01-20 07:52:20.124432775 +0000 UTC m=+80867.455352174: ☑️ agreed by LykxSassinator.

Signed-off-by: hhwyt <hhwyt1@gmail.com>

ti-chi-bot · 2025-01-20T09:05:28Z

@hhwyt: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

hhwyt · 2025-01-20T09:51:39Z

/retest

ti-chi-bot · 2025-01-20T09:51:55Z

@hhwyt: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

glorv · 2025-01-20T10:04:01Z

/retest

ti-chi-bot · 2025-01-20T10:27:47Z

In response to a cherrypick label: new pull request created to branch release-8.5: #18144.

) close tikv#18140 Add io throughput metrics for snapshot generation. Signed-off-by: hhwyt <hhwyt1@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> (cherry picked from commit da63ac3)

…#18144) close #18140 Add io throughput metrics for snapshot generation. Signed-off-by: hhwyt <hhwyt1@gmail.com> Co-authored-by: hhwyt <hhwyt1@gmail.com>

) close tikv#18140 Add io throughput metrics for snapshot generation. Signed-off-by: hhwyt <hhwyt1@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: xzhangxian1008 <xzhangxian@foxmail.com>

close tikv#18140 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-07-18T23:03:26Z

In response to a cherrypick label: new pull request created to branch release-7.5: #18725.
But this PR has conflicts, please resolve them!

raftstore: add io throughput metrics for snapshot generation

2f5c17e

Signed-off-by: hhwyt <hhwyt1@gmail.com>

ti-chi-bot bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jan 9, 2025

hhwyt added 2 commits January 10, 2025 19:22

rafstore: fix io-bytes calculation

dcacd2a

Signed-off-by: hhwyt <hhwyt1@gmail.com>

rafstore: fix testexport

feae0a1

Signed-off-by: hhwyt <hhwyt1@gmail.com>

ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 10, 2025

hhwyt commented Jan 20, 2025

View reviewed changes

glorv approved these changes Jan 20, 2025

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jan 20, 2025

LykxSassinator approved these changes Jan 20, 2025

View reviewed changes

ti-chi-bot bot added the lgtm label Jan 20, 2025

ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jan 20, 2025

ti-chi-bot bot added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Jan 20, 2025

hhwyt added 2 commits January 20, 2025 16:00

rafstore: fix lint

9eeaf76

Signed-off-by: hhwyt <hhwyt1@gmail.com>

raftstore: fix unit tests

a38d1eb

Signed-off-by: hhwyt <hhwyt1@gmail.com>

hhwyt force-pushed the snapshot-real-io branch from cebdb5b to a38d1eb Compare January 20, 2025 08:46

hhwyt and others added 2 commits January 20, 2025 16:56

raftstore: fix lint

b5f263d

Signed-off-by: hhwyt <hhwyt1@gmail.com>

Merge branch 'master' into snapshot-real-io

de6cd2b

ti-chi-bot bot merged commit da63ac3 into tikv:master Jan 20, 2025
1 check passed

ti-chi-bot bot added this to the Pool milestone Jan 20, 2025

ti-chi-bot mentioned this pull request Jan 20, 2025

raftstore: add io throughput metrics for snapshot generation (#18110) #18144

Merged

9 tasks

ti-chi-bot bot added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Jul 18, 2025

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Jul 18, 2025

This is an automated cherry-pick of tikv#18110

018d732

close tikv#18140 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Jul 18, 2025

raftstore: add io throughput metrics for snapshot generation (#18110) #18725

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

raftstore: add io throughput metrics for snapshot generation#18110

raftstore: add io throughput metrics for snapshot generation#18110
ti-chi-bot[bot] merged 7 commits intotikv:masterfrom
hhwyt:snapshot-real-io

hhwyt commented Jan 9, 2025 •

edited

Loading

Uh oh!

ti-chi-bot bot commented Jan 9, 2025

Uh oh!

hhwyt commented Jan 20, 2025

Uh oh!

hhwyt Jan 20, 2025

Uh oh!

glorv left a comment

Uh oh!

ti-chi-bot bot commented Jan 20, 2025

Uh oh!

ti-chi-bot bot commented Jan 20, 2025

Uh oh!

ti-chi-bot bot commented Jan 20, 2025

Uh oh!

hhwyt commented Jan 20, 2025

Uh oh!

ti-chi-bot bot commented Jan 20, 2025

Uh oh!

glorv commented Jan 20, 2025

Uh oh!

Uh oh!

ti-chi-bot commented Jan 20, 2025

Uh oh!

ti-chi-bot commented Jul 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

hhwyt commented Jan 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is changed and how it works?

Related changes

Check List

Release note

Uh oh!

ti-chi-bot bot commented Jan 9, 2025

Uh oh!

hhwyt commented Jan 20, 2025

Uh oh!

hhwyt Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

glorv left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Jan 20, 2025

Uh oh!

ti-chi-bot bot commented Jan 20, 2025

[LGTM Timeline notifier]

Uh oh!

ti-chi-bot bot commented Jan 20, 2025

Uh oh!

hhwyt commented Jan 20, 2025

Uh oh!

ti-chi-bot bot commented Jan 20, 2025

Uh oh!

glorv commented Jan 20, 2025

Uh oh!

Uh oh!

ti-chi-bot commented Jan 20, 2025

Uh oh!

ti-chi-bot commented Jul 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hhwyt commented Jan 9, 2025 •

edited

Loading