util: record schedule wait duration of yatp pool by sticnarf · Pull Request #12441 · tikv/tikv

sticnarf · 2022-04-27T15:58:54Z

What is changed and how it works?

Issue Number: Close #12359

What's Changed:

This commit makes use of the `schedule_time` in yatp to calculate
the wait duration of each wake.

The wait duration panel is added to the grafana for the unified read
pool and the txn scheduler pool.

Check List

Tests

Unit test
Manual test

Release note

None

ti-chi-bot · 2022-04-27T15:58:55Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

cfzjywxk
you06

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

cfzjywxk · 2022-05-06T08:20:52Z

metrics/grafana/tikv_details.json

+            {
+              "exemplar": true,
+              "expr": "rate(tikv_yatp_pool_schedule_wait_duration_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", name=~\"unified-read.*\"}[1m])",
+              "format": "heatmap",


Should we add some avg and P99/max like lines or panels?

The unified read pool and scheduler pool are the most latency-critical pools. So, I use heatmap to show the distribution more clearly.

Now, I add avg and P99 lines to server, showing the wait duration of all pools

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

cfzjywxk

LGTM

sticnarf

/merge

ti-chi-bot · 2022-05-12T03:22:24Z

@sticnarf: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sticnarf · 2022-05-12T06:32:54Z

/test

sticnarf · 2022-05-12T08:03:17Z

/test

sticnarf · 2022-06-01T08:28:00Z

/merge

ti-chi-bot · 2022-06-01T08:28:01Z

@sticnarf: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-06-01T08:28:03Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: b749520

sticnarf · 2022-06-01T09:51:42Z

/test

ti-chi-bot · 2022-06-01T10:36:39Z

@sticnarf: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sticnarf · 2022-06-01T11:12:03Z

/test

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

sticnarf · 2022-06-01T11:55:06Z

/merge

ti-chi-bot · 2022-06-01T11:55:07Z

@sticnarf: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-06-01T11:55:09Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: c9d8f22

This reverts commit a5987f3. Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

sticnarf force-pushed the pool-sched-duration-metrics branch from 1bdb281 to 5b3d28c Compare April 28, 2022 12:00

sticnarf force-pushed the pool-sched-duration-metrics branch from 5b3d28c to ff44ce0 Compare May 6, 2022 07:07

util: record schedule wait duration of yatp pool

2d050a9

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

sticnarf force-pushed the pool-sched-duration-metrics branch from ff44ce0 to 2d050a9 Compare May 6, 2022 08:00

sticnarf marked this pull request as ready for review May 6, 2022 08:05

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 6, 2022

sticnarf requested review from cfzjywxk and you06 May 6, 2022 08:06

cfzjywxk reviewed May 6, 2022

View reviewed changes

add avg and 99% schedule wait duration panels

48fa1ad

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 6, 2022

cfzjywxk approved these changes May 6, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 6, 2022

you06 approved these changes May 11, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 11, 2022

sticnarf commented May 12, 2022

View reviewed changes

Merge branch 'master' into pool-sched-duration-metrics

b749520

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 1, 2022

ti-chi-bot added 3 commits June 1, 2022 16:28

Merge branch 'master' into pool-sched-duration-metrics

0ae0633

Merge branch 'master' into pool-sched-duration-metrics

9a554f9

Merge branch 'master' into pool-sched-duration-metrics

9f446f0

ti-chi-bot added 2 commits June 1, 2022 17:56

Merge branch 'master' into pool-sched-duration-metrics

300ecb4

Merge branch 'master' into pool-sched-duration-metrics

dac7565

fix unit tests

c9d8f22

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Jun 1, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 1, 2022

ti-chi-bot merged commit a5987f3 into tikv:master Jun 1, 2022

BusyJay added a commit to BusyJay/tikv that referenced this pull request Aug 12, 2022

Revert "util: record schedule wait duration of yatp pool (tikv#12441)"

3f964ef

This reverts commit a5987f3. Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

sticnarf mentioned this pull request Aug 16, 2022

Storage pool regression on AWS EC2 instance type i3.4xlarge #13280

Closed

Conversation

sticnarf commented Apr 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is changed and how it works?

Check List

Release note

Uh oh!

ti-chi-bot commented Apr 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cfzjywxk May 6, 2022

Choose a reason for hiding this comment

Uh oh!

sticnarf May 6, 2022

Choose a reason for hiding this comment

Uh oh!

cfzjywxk left a comment

Choose a reason for hiding this comment

Uh oh!

sticnarf left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot commented May 12, 2022

Uh oh!

sticnarf commented May 12, 2022

Uh oh!

sticnarf commented May 12, 2022

Uh oh!

sticnarf commented Jun 1, 2022

Uh oh!

ti-chi-bot commented Jun 1, 2022

Uh oh!

ti-chi-bot commented Jun 1, 2022

Uh oh!

sticnarf commented Jun 1, 2022

Uh oh!

ti-chi-bot commented Jun 1, 2022

Uh oh!

sticnarf commented Jun 1, 2022

Uh oh!

sticnarf commented Jun 1, 2022

Uh oh!

ti-chi-bot commented Jun 1, 2022

Uh oh!

ti-chi-bot commented Jun 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sticnarf commented Apr 27, 2022 •

edited

Loading

ti-chi-bot commented Apr 27, 2022 •

edited

Loading