Skip to content

copr: improve observability of total suspend time (#19178)#19195

Merged
ti-chi-bot[bot] merged 3 commits intotikv:release-8.5from
ti-chi-bot:cherry-pick-19178-to-release-8.5
Dec 11, 2025
Merged

copr: improve observability of total suspend time (#19178)#19195
ti-chi-bot[bot] merged 3 commits intotikv:release-8.5from
ti-chi-bot:cherry-pick-19178-to-release-8.5

Conversation

@ti-chi-bot
Copy link
Member

This is an automated cherry-pick of #19178

What is changed and how it works?

Issue Number: Close #19179

What's Changed:

Changes:
- Include coprocessor suspend time in the existing copr wait time metric
- Add panels for semaphore waiting time and the number of copr tasks
  waiting on the semaphore.

Background: 
- `total_suspend_time` measures how long a coprocessor task is not
actually being processed. This can be prolonged for several reasons,
including YATP scheduling wait and waiting for the concurrency limiter
semaphore. Previously, `total_suspend_time` was only visible in TiKV’s
slow query logs.
- Semaphore waiting is one of the main contributors to long suspend
times. The concurrency limiter exists to prioritize completing a limited
number of heavy tasks rather than spreading work thin across too many
tasks. In this mechanism, when a copr task runs for more than 5ms, it
must acquire a semaphore permit before continuing. The total number of 
permits defaults to the number of CPU cores.

Grafana example:
20251210-143717

image

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Release note

None

close tikv#19179

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot added dco-signoff: yes Indicates the PR's author has signed the dco. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-8.5 This PR is cherry-picked to release-8.5 from a source PR. labels Dec 10, 2025
@ti-chi-bot
Copy link
Member Author

@hbisheng This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 10, 2025

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: Bisheng Huang <hbisheng@gmail.com>
@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Dec 11, 2025
@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 11, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 11, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-12-11 05:35:08.433405854 +0000 UTC m=+1105653.247183426: ☑️ agreed by overvenus.
  • 2025-12-11 05:41:08.687492027 +0000 UTC m=+1106013.501269598: ☑️ agreed by glorv.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 11, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: glorv, LykxSassinator, overvenus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [LykxSassinator,glorv,overvenus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hbisheng
Copy link
Member

/unhold

@ti-chi-bot ti-chi-bot bot added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/cherry-pick-not-approved labels Dec 11, 2025
@ti-chi-bot ti-chi-bot bot merged commit 83ff1a8 into tikv:release-8.5 Dec 11, 2025
5 checks passed
@ti-chi-bot ti-chi-bot bot deleted the cherry-pick-19178-to-release-8.5 branch December 11, 2025 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved cherry-pick-approved Cherry pick PR approved by release team. dco-signoff: yes Indicates the PR's author has signed the dco. lgtm release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-8.5 This PR is cherry-picked to release-8.5 from a source PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants