What is changed and how it works?

Issue Number: ref #12355

What's Changed:
After this change, the metrics registry is divided into 4 registries:

• HIGH_PRIORITY_REGISTRY. The core metrics that should be always fetched, if server.simplified-metrics is set to true, we try to reduce some counter and histogram samples but still ensure it won't affact the grafana charts.
• FULL_HISTOGRAM_REGISTRY. These metrics are also high priority, but due to the expression used in grafana or the charts type("flamegraph"), the sample data should not be simplified, so always return the original full sample data.
• UNUSED_METRICS_REGISTRY. These metrics are not used in tikv's grafana, so do not return them in the simplified mode.
• default. The rest are in the system default registry. In the simplified mode, these samples will be returned will a smaller frequency, by not the frequency is 1/2, that is one in 2 request includes them.

Add a new server config server.simplified-metrics to control whether the metrics API is running under the simplified-mode.
If this config is set to true(false by default), tikv will return simplified metrics. If set to false, it will still return the full metrics as before.

Sample data simplification strategy:

• Counter. Filter out all counters that its value is 0.
• Histogram. If the histogram's max accumulate count is 0, it will be filtered out. Else, any bucket with its value equal to the former and successive bucket's value will be filtered out.

NOTE:
After the compact of histogram, there is no information loss in theory (as we only filtered out buckets will 0 sample values), due to the expr implementation of prometheus, some type of expression or charts can not be drawn correctly, so we keep the effected metrics at the FULL_HISTOGRAM_REGISTRY. Currently, the affected charts are: expression contains delta function and charts with flamegraph.

Side Effect:
due to the reduction of some metrics, when server.simplified-metrics is set to true, there are some negative side effects to the grafana data:

• Some expression will show no data instead of 0 data because we filter out 0 data for counter and histogram.
• The precision of some charts will decrease by half because the sample duration is decreased from 15s to 30s.

Benchmark result:
With server.simplified-metrics set to false, the response data size is about 1.3MB, total samples count is: 17081.
With server.simplified-metrics set to true, when only high priority metrics are returned, the data size is about 125KB, the samples count is about 1494. when the default metrics is also returned, the data size is 226KB, samples count is 2609.

Thus the overall sample reduced percentage when server.simplified-metrics set to true and false is: (1494+2609)/2/17081 = 0.12.

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• Need to cherry-pick to the release branch

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression
  • Consumes more CPU
  • Consumes more MEM
• Breaking backward compatibility

Release note

None