Skip to content

Add grpc wait duration metrics#18810

Merged
ti-chi-bot[bot] merged 9 commits intotikv:masterfrom
yibin87:add_metrics_for_grpc_wait
Aug 11, 2025
Merged

Add grpc wait duration metrics#18810
ti-chi-bot[bot] merged 9 commits intotikv:masterfrom
yibin87:add_metrics_for_grpc_wait

Conversation

@yibin87
Copy link
Contributor

@yibin87 yibin87 commented Aug 11, 2025 •
edited
Loading

What is changed and how it works?

Issue Number: Close #18809

What's Changed:

Set "server.grpc-compression-type: gzip" and read large row data to overload grpc thread.
When gRPC thread is overloading, we can see "RPC Layer Latency" is high:
img_v3_02p2_f804792c-cdc0-4a55-be82-9cf9ca8368ag

After adding grpc wait metrics, we can further detect the root cause and detect it earlier:
img_v3_02p2_e771189c-a335-43aa-b879-a6c2cab3abbg

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Release note

Add a new metrics which shows the grpc message waiting time.

@yibin87
Add grpc wait duration metrics
e291b94 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@ti-chi-bot ti-chi-bot bot added dco-signoff: yes Indicates the PR's author has signed the dco. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. contribution This PR is from a community contributor. labels Aug 11, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 11, 2025

Hi @yibin87. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot bot added needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 11, 2025
@yibin87
update tikv details dashboard scripts
6aeafdd 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@ti-chi-bot ti-chi-bot bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 11, 2025
@yibin87
update grafana files
c91f062 
Signed-off-by: yibin <huyibin@pingcap.com>
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 11, 2025
yibin87 added 3 commits August 11, 2025 15:13
@yibin87
Little update
a335e63 
Signed-off-by: yibin <huyibin@pingcap.com>
@yibin87
Little update
88a8d9d 
Signed-off-by: yibin <huyibin@pingcap.com>
@yibin87
Little fix
397eb40 
Signed-off-by: yibin <huyibin@pingcap.com>
@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Aug 11, 2025
MyonKeminta
MyonKeminta reviewed Aug 11, 2025
View reviewed changes
@yibin87
Address comments
a62baa9 
Signed-off-by: yibin <huyibin@pingcap.com>
@yibin87 yibin87 requested a review from MyonKeminta August 11, 2025 09:42
@yibin87
Copy link
Contributor Author

yibin87 commented Aug 11, 2025

/cc @overvenus @glorv

@ti-chi-bot ti-chi-bot bot requested review from glorv and overvenus August 11, 2025 09:43
@yibin87
Address comments
d29ef3a 
Signed-off-by: yibin <huyibin@pingcap.com>
@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Aug 11, 2025
glorv
glorv reviewed Aug 11, 2025
View reviewed changes
Copy link
Contributor

@glorv glorv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

@yibin87
Address comments
9d3d515 
Signed-off-by: yibin <huyibin@pingcap.com>
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 11, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: glorv, MyonKeminta

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Aug 11, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 11, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-08-11 09:50:51.682733984 +0000 UTC m=+261050.717317222: ☑️ agreed by MyonKeminta.
  • 2025-08-11 10:17:36.925040431 +0000 UTC m=+262655.959623659: ☑️ agreed by glorv.

@glorv
Copy link
Contributor

glorv commented Aug 11, 2025

/ok-to-test

@ti-chi-bot ti-chi-bot bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Aug 11, 2025
@ti-chi-bot ti-chi-bot bot merged commit 0e3f125 into tikv:master Aug 11, 2025
8 checks passed
@ti-chi-bot ti-chi-bot bot added this to the Pool milestone Aug 11, 2025
overvenus
overvenus reviewed Aug 11, 2025
View reviewed changes
@yibin87
Copy link
Contributor Author

yibin87 commented Aug 13, 2025

/cherry-pick-invite

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Aug 13, 2025

@yibin87 you're not a member of org tikv

@yibin87
Copy link
Contributor Author

yibin87 commented Aug 13, 2025

/cherry-pick release-8.5
/cherry-pick release-8.1

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Aug 13, 2025
@yibin87 @ti-chi-bot
This is an automated cherry-pick of tikv#18810
55b4481 
close tikv#18809

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Aug 13, 2025
Open
9 tasks
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Aug 13, 2025

@yibin87: new pull request created to branch release-8.1: #18825.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherry-pick release-8.5
/cherry-pick release-8.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Aug 13, 2025
@yibin87 @ti-chi-bot
This is an automated cherry-pick of tikv#18810
189f440 
close tikv#18809

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Aug 13, 2025

@yibin87: new pull request created to branch release-8.5: #18826.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherry-pick release-8.5
/cherry-pick release-8.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot mentioned this pull request Aug 13, 2025
Merged
9 tasks
@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Aug 13, 2025
3AceShowHand pushed a commit to 3AceShowHand/tikv that referenced this pull request Oct 13, 2025
@yibin87 @3AceShowHand
Add grpc wait duration metrics (tikv#18810)
e4af6f1 
close tikv#18809

Signed-off-by: yibin <huyibin@pingcap.com>
Signed-off-by: 3AceShowHand <jinl1037@hotmail.com>
yibin87 added a commit to ti-chi-bot/tikv that referenced this pull request Nov 10, 2025
@yibin87
This is an automated cherry-pick of tikv#18810
41c5a88 
close tikv#18809

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot bot pushed a commit that referenced this pull request Nov 11, 2025
@ti-chi-bot @yibin87
Add grpc wait duration metrics (#18810) (#18826)
7d661d2 
close #18809

Signed-off-by: yibin87 <huyibin@pingcap.com>

Co-authored-by: yibin87 <huyibin@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@overvenus overvenus overvenus left review comments

@glorv glorv glorv approved these changes

@MyonKeminta MyonKeminta MyonKeminta approved these changes

Assignees

No one assigned

Labels

approved contribution This PR is from a community contributor. dco-signoff: yes Indicates the PR's author has signed the dco. lgtm ok-to-test Indicates a PR is ready to be tested. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

Pool

Development

Successfully merging this pull request may close these issues.

Add grpc wait time metrics to better observe grpc overloading cases

5 participants

@yibin87 @glorv @ti-chi-bot @overvenus @MyonKeminta