Skip to content

prometheus "name" label in top K metric should be unique per metric #158

@weyrick

Description

@weyrick

Right now all TopK metrics render their prometheus output with a "name" label containing the frequent item:

dns_top_qname3{instance="gw",name=".fe.apple-dns.net"} 2
dns_top_qname3{instance="gw",name="clientservices.googleapis.com"} 2
dns_top_qname3{instance="gw",name="gs.apple.com"} 2
dns_top_qname3{instance="gw",name="cdn2.ghostery.com"} 2
dns_top_qtype{instance="gw",name="A"} 30
dns_top_qtype{instance="gw",name="HTTPS"} 8
dns_top_qtype{instance="gw",name="AAAA"} 2

This makes "name" very high cardinality, and also hampers data exploration and graphing in e.g. Grafana, which is unable the create an automatic filter on "name" because it is ambiguous which name should be filtered.

We should change this label to be specific per TopK metric, for example "qtype3" and "qtype" in the example above. This will shard the cardinality out, and give us a way to take advantage of Grafana "Filter by Value"
Screen Shot 2021-12-09 at 1.19.39 PM.png

HOWEVER this is a breaking change and will break existing dashboards that already use "name" so we need to handle that appropriately.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions