batch_system: Add fsm schedule related metrics by Connor1996 · Pull Request #17723 · tikv/tikv

Connor1996 · 2024-10-28T08:17:40Z

What is changed and how it works?

Issue Number: close #18684

What's Changed:

Add fsm schedule related metrics

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Release note

None

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

overvenus · 2024-10-28T08:47:28Z

components/batch-system/src/batch.rs

+struct MetricsCollector<N: Fsm> {
+    timer: Instant, // time since polled
+    round: usize,   // how many round the fsm has been continuously polled
+    _phantom: std::marker::PhantomData<N>,


Suggested change

_phantom: std::marker::PhantomData<N>,

fsm_type: String,

How about adding a String? It makes the code cleaner.

the static metric vector get() only accepts the enum instead of string

components/raftstore/src/store/fsm/peer.rs

hhwyt · 2024-10-28T10:06:57Z

components/batch-system/src/metrics.rs

    )
    .unwrap();

+    pub static ref FSM_RESCHEDULE_COUNTER_VEC: IntCounterVec = register_int_counter_vec!(


Why aren't these reschedule metrics shown in the graph of your comments?

the screen is not big enough :)

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

SpadeA-Tang · 2024-10-29T06:58:30Z

components/batch-system/src/scheduler.rs

        };

-        match sender.send(FsmTypes::Normal(fsm), None) {
+        match sender.send(FsmTypes::Normal((fsm, Instant::now_coarse())), None) {


The precision of now_coarse is typically ms level and I think it may mislead someone.

I think ms is enough here

hhwyt · 2024-10-29T07:57:40Z

LGTM

ti-chi-bot · 2024-10-29T08:10:25Z

@hhwyt: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

components/batch-system/src/fsm.rs

components/batch-system/src/metrics.rs

hbisheng · 2024-10-31T09:09:29Z

components/batch-system/src/metrics.rs

+    pub static ref FSM_POLL_ROUND: FsmPollRoundVec =
+        auto_flush_from!(FSM_POLL_ROUND_VEC, FsmPollRoundVec);
+
+    pub static ref FSM_COUNT_PER_POLL_VEC: HistogramVec =


It would be even better if we can see this metric per poller. But that might not be easy to achieve.

I don't think per poller gives extra information

metrics/grafana/tikv_details.dashboard.py

Co-authored-by: Bisheng Huang <hbisheng@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

ti-chi-bot · 2024-11-01T07:23:30Z

[LGTM Timeline notifier]

Timeline:

2024-10-29 09:40:53.904124196 +0000 UTC m=+342766.743279742: ☑️ agreed by SpadeA-Tang.
2024-11-01 07:23:29.244021496 +0000 UTC m=+593722.083177035: ☑️ agreed by glorv.

Connor1996 · 2024-11-01T07:24:02Z

/approve

ti-chi-bot · 2024-11-01T07:24:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Connor1996, glorv, hhwyt, SpadeA-Tang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [Connor1996,SpadeA-Tang,glorv]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ref #15990 Add fsm schedule related metrics Signed-off-by: Connor <zbk602423539@gmail.com> Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Bisheng Huang <hbisheng@gmail.com> Signed-off-by: Neil Shen <overvenus@gmail.com>

ref tikv#15990 Add fsm schedule related metrics Signed-off-by: Connor <zbk602423539@gmail.com> Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Bisheng Huang <hbisheng@gmail.com> Signed-off-by: Neil Shen <overvenus@gmail.com>

ref #15990 Add fsm schedule related metrics Signed-off-by: Connor <zbk602423539@gmail.com> Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: Connor <zbk602423539@gmail.com> Co-authored-by: Bisheng Huang <hbisheng@gmail.com>

ref tikv#15990 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-06-17T02:58:18Z

In response to a cherrypick label: new pull request created to branch release-7.5: #18557.
But this PR has conflicts, please resolve them!

ref tikv#15990 Add fsm schedule related metrics Signed-off-by: Connor <zbk602423539@gmail.com> Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Bisheng Huang <hbisheng@gmail.com> Signed-off-by: Neil Shen <overvenus@gmail.com>

ref tikv#15990 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-06-17T04:51:07Z

In response to a cherrypick label: new pull request created to branch release-8.1: #18558.
But this PR has conflicts, please resolve them!

ref #15990, close #18684 Add fsm schedule related metrics Signed-off-by: Connor <zbk602423539@gmail.com> Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: Connor <zbk602423539@gmail.com> Co-authored-by: Bisheng Huang <hbisheng@gmail.com>

ref tikv#15990 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-08-29T03:59:07Z

In response to a cherrypick label: new pull request created to branch release-8.5: #18893.
But this PR has conflicts, please resolve them!

close tikv#18800, ref tikv#17723 1. in raftstore add apply_msg_len.flush() to trigger metric dump; 2. in raftsotre_v2 add corresponding hist.observe() in handle_all_tasks(); Signed-off-by: squalfof <squalfof@gmail.com>

close tikv#18800, ref tikv#17723 In raftstore/ add apply_msg_len.flush() to trigger metric dump. Signed-off-by: squalfof <squalfof@gmail.com>

ti-chi-bot bot added dco-signoff: yes Indicates the PR's author has signed the dco. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 28, 2024

add batch system metrics

150433a

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Connor1996 force-pushed the batch-system-metrics branch from a952cc7 to 150433a Compare October 28, 2024 08:18

Merge remote-tracking branch 'upstream/master' into batch-system-metrics

498a205

overvenus reviewed Oct 28, 2024

View reviewed changes

hhwyt reviewed Oct 28, 2024

View reviewed changes

address comment

b795f9a

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

SpadeA-Tang reviewed Oct 29, 2024

View reviewed changes

hhwyt approved these changes Oct 29, 2024

View reviewed changes

SpadeA-Tang approved these changes Oct 29, 2024

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Oct 29, 2024

glorv reviewed Oct 30, 2024

View reviewed changes

components/batch-system/src/fsm.rs Outdated Show resolved Hide resolved

hbisheng reviewed Oct 31, 2024

View reviewed changes

Connor1996 and others added 8 commits November 1, 2024 09:57

Update components/batch-system/src/metrics.rs

b473638

Co-authored-by: Bisheng Huang <hbisheng@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

Update components/batch-system/src/metrics.rs

f4ba469

Co-authored-by: Bisheng Huang <hbisheng@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

Update components/batch-system/src/metrics.rs

9fa72d4

Co-authored-by: Bisheng Huang <hbisheng@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

Update metrics/grafana/tikv_details.dashboard.py

7777b30

Co-authored-by: Bisheng Huang <hbisheng@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

Update metrics/grafana/tikv_details.dashboard.py

d1f856f

Co-authored-by: Bisheng Huang <hbisheng@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

use const type

2638faa

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

update description

b71bab8

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Merge remote-tracking branch 'upstream/master' into batch-system-metrics

3bc590f

glorv approved these changes Nov 1, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 1, 2024

ti-chi-bot bot merged commit f51e848 into tikv:master Nov 1, 2024

ti-chi-bot bot added this to the Pool milestone Nov 1, 2024

overvenus added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Jun 17, 2025

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Jun 17, 2025

This is an automated cherry-pick of tikv#17723

4900248

ref tikv#15990 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Jun 17, 2025

batch_system: Add fsm schedule related metrics (#17723) #18557

Merged

9 tasks

overvenus added the needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. label Jun 17, 2025

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Jun 17, 2025

This is an automated cherry-pick of tikv#17723

9d156db

ref tikv#15990 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Jun 17, 2025

batch_system: Add fsm schedule related metrics (#17723) #18558

Open

9 tasks

overvenus mentioned this pull request Jul 8, 2025

Add fsm schedule related metrics #18684

Closed

ti-chi-bot bot added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Aug 29, 2025

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Aug 29, 2025

This is an automated cherry-pick of tikv#17723

88c29ad

ref tikv#15990 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Aug 29, 2025

batch_system: Add fsm schedule related metrics (#17723) #18893

Open

9 tasks

ti-chi-bot bot removed the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Aug 29, 2025

hbisheng mentioned this pull request Oct 13, 2025

Missing metrics for apply msg length distribution #18800

Closed

squalfof mentioned this pull request Oct 16, 2025

raftstore*: Add missing metric apply_msg_len #19058

Merged

9 tasks

squalfof added a commit to squalfof/tikv that referenced this pull request Oct 21, 2025

raftstore*: Add missing metric apply_msg_len

272d335

close tikv#18800, ref tikv#17723 In raftstore/ add apply_msg_len.flush() to trigger metric dump. Signed-off-by: squalfof <squalfof@gmail.com>

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Nov 5, 2025

raftstore*: Add missing metric apply_msg_len

749f147

close tikv#18800, ref tikv#17723 In raftstore/ add apply_msg_len.flush() to trigger metric dump. Signed-off-by: squalfof <squalfof@gmail.com>

Conversation

Connor1996 commented Oct 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is changed and how it works?

Related changes

Check List

Release note

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hhwyt commented Oct 29, 2024

Uh oh!

ti-chi-bot bot commented Oct 29, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ti-chi-bot bot commented Nov 1, 2024

[LGTM Timeline notifier]

Uh oh!

Connor1996 commented Nov 1, 2024

Uh oh!

ti-chi-bot bot commented Nov 1, 2024

Uh oh!

ti-chi-bot commented Jun 17, 2025

Uh oh!

ti-chi-bot commented Jun 17, 2025

Uh oh!

ti-chi-bot commented Aug 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Connor1996 commented Oct 28, 2024 •

edited

Loading