-
Notifications
You must be signed in to change notification settings - Fork 4.1k
admission: fix ioTokensExhaustedDurationMetric on 22.1 #86963
Copy link
Copy link
Closed
Labels
A-admission-controlC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-admission-controlAdmission ControlAdmission Controlbranch-release-22.1Used to mark GA and release blockers, technical advisories, and bugs for 22.1Used to mark GA and release blockers, technical advisories, and bugs for 22.1
Description
Describe the problem
The ioTokensExhaustedDurationMetric is busted on v22.1 because every time setAvailableIOTokensLocked is called when sg.availableIOTokens < 0, it re-adds now() - sg.exhaustedStart to the metric without resetting the value of sg.exhaustedStart. On a node that's constantly overloaded, it ends up with a ioTokensExhaustedDurationMetric that increases roughly quadratically, such that taking its rate gives a linearly increasing graph over time, which doesn't seem to be the metric's intent. See this where @a-robinson spotted it.
Jira issue: CRDB-19062
Epic CRDB-54646
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-admission-controlC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-admission-controlAdmission ControlAdmission Controlbranch-release-22.1Used to mark GA and release blockers, technical advisories, and bugs for 22.1Used to mark GA and release blockers, technical advisories, and bugs for 22.1