Skip to content

admission: export elastic CPU utilization % as a metric, natively#92470

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
irfansharif:221124.elastic-cpu-metric
Nov 29, 2022
Merged

admission: export elastic CPU utilization % as a metric, natively#92470
craig[bot] merged 1 commit intocockroachdb:masterfrom
irfansharif:221124.elastic-cpu-metric

Conversation

@irfansharif
Copy link
Copy Markdown
Contributor

@irfansharif irfansharif commented Nov 24, 2022

Fixes #89814. With the elastic CPU limiter (#86638) closely tracking acquired/returned CPU nanoseconds, it's possible to compute what % CPU utilization it's nominally overseeing. In experimentation we've been using prometheus where this CPU % is being computed using:

(
  rate(admission_elastic_cpu_acquired_nanos[$__rate_interval]) -
  rate(admission_elastic_cpu_acquired_nanos[$__rate_interval])
) / admission_elastic_cpu_max_available_nanos

This timeseries math is not possible in CRDB natively, but it's a useful one to have to observe the CPU limiter in action. This PR computes this number within CRDB and exports it as a metric. Below we see the two different forms of this metric, one computed as described above (smoother) and the version introduced in this PR.

image

Release note: None

@irfansharif irfansharif requested a review from a team as a code owner November 24, 2022 20:35
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@irfansharif irfansharif force-pushed the 221124.elastic-cpu-metric branch 3 times, most recently from 2dd8063 to 0208774 Compare November 24, 2022 22:12
Fixes cockroachdb#89814. With the elastic CPU limiter (cockroachdb#86638) closely tracking
acquired/returned CPU nanoseconds, it's possible to compute what % CPU
utilization it's nominally overseeing. In experimentation we've been
using prometheus where this CPU % is being computed using:

    (
      rate(admission_elastic_cpu_acquired_nanos[$__rate_interval]) -
      rate(admission_elastic_cpu_acquired_nanos[$__rate_interval])
    ) / admission_elastic_cpu_max_available_nanos

This timeseries math is not possible in CRDB natively, but it's a useful
one to have to observe the CPU limiter in action. This PR computes this
number within CRDB and exports it as a metric.

Release note: None
@irfansharif irfansharif force-pushed the 221124.elastic-cpu-metric branch from 0208774 to 1b239d8 Compare November 25, 2022 14:54
@irfansharif irfansharif requested a review from a team November 25, 2022 14:54
@irfansharif
Copy link
Copy Markdown
Contributor Author

(Bump.)

Copy link
Copy Markdown
Collaborator

@dhartunian dhartunian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: thanks for the nice screenshot

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained

@irfansharif
Copy link
Copy Markdown
Contributor Author

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Nov 29, 2022

Build succeeded:

@irfansharif irfansharif deleted the 221124.elastic-cpu-metric branch November 30, 2022 01:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

admission: export elastic CPU utilization % as a metric, natively

3 participants