-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Description
Greetings! Here's a feature request to track. Other folks that use Datadog for monitoring would likely be interested in this as well.
The statsd metrics that Envoy produces are geared towards the generic statsd implementation. For folks that use Datadog for monitoring this is suboptimal, because Datadog's statsd implementation supports metric tags (ie. dimensions). See:
- https://docs.datadoghq.com/guides/dogstatsd/
- https://www.datadoghq.com/blog/the-power-of-tagged-metrics/
- https://help.datadoghq.com/hc/en-us/articles/204312749-Getting-started-with-tags
Ex: For a given metric like a cluster's upstream_cx_rx_bytes_buffered, a single Envoy instance currently produces metrics with names:
envoy.cluster.CLUSTER_A.upstream_cx_rx_bytes_buffered
envoy.cluster.CLUSTER_B.upstream_cx_rx_bytes_buffered
...
envoy.cluster.CLUSTER_N.upstream_cx_rx_bytes_buffered
... for each cluster. In Datadog best practices, one would prefer a single metric-name like envoy.cluster.upstream_cx_rx_bytes_buffered which is tagged by the respective cluster name. This makes it possible to aggregate & monitor across many clusters, and be able to find anomalies or get alerts from any misbehaving cluster(s).
(And yes, part of the problem here is that Datadog doesn't support templating/string interpolation of metrics names in their monitors & dashboards.)