tsdb: metrics graphs reporting abnormally high values #99486
Description
Describe the problem
Original thread: https://cockroachlabs.slack.com/archives/C01CNRP6TSN/p1679669195626919
The roachprod cluster in the above linked Slack thread is experiencing abnormally high stats readings in DB Console metrics charts. For example, normalized CPU Usage charts are reading > 1000% per node, memory usage per-node > 250GB, etc.
We recently merged #98077, which modified the TSDB query code to work for in-process tenants. It's possible we introduced a bug into the aggregation logic.
To Reproduce
- Set up a roach prod cluster (not multi-tenant - note the specific roachprod cluster where this was discovered did not have multiple tenants).
- Generate a workload against the cluster.
- Observe abnormally high metric readings.
Additional data / screenshots
Jira issue: CRDB-25898
Metadata
Metadata
Assignees
Labels
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Failures and bugs on the master branch.Used to mark GA and release blockers, technical advisories, and bugs for 23.1Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.