outbound: Report per-route-backend request count metrics#2377
Merged
Conversation
When performing policy-based routing, proxies may dispatch requests through per-route backend configurations. In order to illustrate how routing rules apply and how backend distributions are being honored, this change adds two new metrics: * `outbound_http_route_backend_requests_total`; and * `outbound_grpc_route_backend_requests_total` Each of these metrics includes labels that identify a routes parent (i.e. a Service), the route resource being used, and the backend resource being used. This implementation does NOT implement any form of metrics eviction for these new metrics. This is tolerable for the short term as the cardinality of services and routes is generally much less than the cardinality of individual endpoints (where we do require timeout/eviction for metrics).
hawkw
approved these changes
Apr 4, 2023
Contributor
hawkw
left a comment
There was a problem hiding this comment.
looks good to me, the use of ExtractParam for metrics is very cool and makes me want to rewrite the rest of our metrics code (eventually). no blockers from me!
Comment on lines
+132
to
+150
| impl<T> svc::ExtractParam<RequestCount, Http<T>> for ExtractMetrics { | ||
| fn extract_param(&self, params: &Http<T>) -> RequestCount { | ||
| RequestCount(self.metrics.http_requests_total( | ||
| params.params.concrete.parent_ref.clone(), | ||
| params.params.route_ref.clone(), | ||
| params.params.concrete.backend_ref.clone(), | ||
| )) | ||
| } | ||
| } | ||
|
|
||
| impl<T> svc::ExtractParam<RequestCount, Grpc<T>> for ExtractMetrics { | ||
| fn extract_param(&self, params: &Grpc<T>) -> RequestCount { | ||
| RequestCount(self.metrics.grpc_requests_total( | ||
| params.params.concrete.parent_ref.clone(), | ||
| params.params.route_ref.clone(), | ||
| params.params.concrete.backend_ref.clone(), | ||
| )) | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
neat, this could be a nice pattern to use for other metrics stuff eventually!
| assert_eq!( | ||
| lines, | ||
| vec![ | ||
| r#"outbound_http_route_backend_requests_total{parent_group="core",parent_kind="Service",parent_namespace="ns",parent_name="papa",parent_port="7979",parent_section_name="",route_group="policy.linkerd.io",route_kind="HTTPRoute",route_namespace="ns",route_name="default",backend_group="core",backend_kind="Service",backend_namespace="ns",backend_name="default",backend_port="8080",backend_section_name=""} 1"#, |
Contributor
There was a problem hiding this comment.
nit, not actually important: it seems like it might be nicer to use multi-line string literals here with \ to escape whitespace? for example, this is the same string:
Suggested change
| r#"outbound_http_route_backend_requests_total{parent_group="core",parent_kind="Service",parent_namespace="ns",parent_name="papa",parent_port="7979",parent_section_name="",route_group="policy.linkerd.io",route_kind="HTTPRoute",route_namespace="ns",route_name="default",backend_group="core",backend_kind="Service",backend_namespace="ns",backend_name="default",backend_port="8080",backend_section_name=""} 1"#, | |
| "outbound_http_route_backend_requests_total{\ | |
| parent_group=\"core\",\ | |
| parent_kind=\"Service\",\ | |
| parent_namespace=\"ns\",\ | |
| parent_name=\"papa\",\ | |
| parent_port=\"7979\",\ | |
| parent_section_name=\"\",\ | |
| route_group=\"policy.linkerd.io\",\ | |
| route_kind=\"HTTPRoute\", | |
| route_namespace=\"ns\",\ | |
| route_name=\"default\",\ | |
| backend_group=\"core\",\ | |
| backend_kind=\"Service\",\ | |
| backend_namespace=\"ns\",\ | |
| backend_name=\"default\",\ | |
| backend_port=\"8080\",\ | |
| backend_section_name=\"\"\ | |
| } 1", |
but the label values are a little easier to read...
Member
Author
There was a problem hiding this comment.
yeah this test could definitely be prettier in a few ways. going to leave the mush of labels for now.
hawkw
added a commit
to linkerd/linkerd2
that referenced
this pull request
Apr 4, 2023
This proxy release adds new `outbound_http_route_backend_requests_total` and `outbound_grpc_route_backend_requests_total` metrics, which can be used to track how routing rules and backend distributions apply to requests. These metrics contain labels describing the route's parent (i.e. a Service), the route resource being used, and the backend resource being used by each request. In addition, this proxy release adds new `INFO`-level logs for failure accrual, and updates logging contexts with Kubernetes resource metadata. --- * outbound: Emit INFO-level logs on failure accrual changes (linkerd/linkerd2-proxy#2373) * app: Rename Metrics types as InboundMetrics and OutboundMetrics (linkerd/linkerd2-proxy#2376) * outbound: Configure balancers with service metadata (linkerd/linkerd2-proxy#2374) * outbound: Report per-route-backend request count metrics (linkerd/linkerd2-proxy#2372) * outbound: Report per-route-backend request count metrics (linkerd/linkerd2-proxy#2377)
hawkw
added a commit
to linkerd/linkerd2
that referenced
this pull request
Apr 5, 2023
This proxy release adds new `outbound_http_route_backend_requests_total` and `outbound_grpc_route_backend_requests_total` metrics, which can be used to track how routing rules and backend distributions apply to requests. These metrics contain labels describing the route's parent (i.e. a Service), the route resource being used, and the backend resource being used by each request. In addition, this proxy release adds new `INFO`-level logs for failure accrual, and updates logging contexts with Kubernetes resource metadata. --- * outbound: Emit INFO-level logs on failure accrual changes (linkerd/linkerd2-proxy#2373) * app: Rename Metrics types as InboundMetrics and OutboundMetrics (linkerd/linkerd2-proxy#2376) * outbound: Configure balancers with service metadata (linkerd/linkerd2-proxy#2374) * outbound: Report per-route-backend request count metrics (linkerd/linkerd2-proxy#2372) * outbound: Report per-route-backend request count metrics (linkerd/linkerd2-proxy#2377)
hawkw
added a commit
to linkerd/website
that referenced
this pull request
Apr 10, 2023
This branch adds documentation for the new metrics added in linkerd/linkerd2-proxy#2377 and linkerd/linkerd2-proxy#2380.
hawkw
added a commit
to linkerd/website
that referenced
this pull request
Apr 10, 2023
This branch adds documentation for the new metrics added in linkerd/linkerd2-proxy#2377 and linkerd/linkerd2-proxy#2380.
cratelyn
added a commit
to linkerd/website
that referenced
this pull request
Nov 14, 2025
the documentation of our proxy metrics has not kept pace with all of the exciting telemetry that has been introduced since #1599 documented the state of our authorization policy metrics. this commit reworks the documentation to exhaustively document the families of metrics exported by the proxy. this commit does not introduce mention of the _inbound_ metrics that have been added, but does rename this section to "_Endpoint Metrics_" in order to be compatible with the future addition of `inbound_http_route_*`, `inbound_http_route_backend_*`, `inbound_grpc_route*`, and `inbound_grpc_route_backend_*` metrics. see: * linkerd/linkerd2-proxy#2377 * linkerd/linkerd2-proxy#2380 * linkerd/linkerd2-proxy#3086 * linkerd/linkerd2-proxy#3308 * linkerd/linkerd2-proxy#3334 Signed-off-by: katelyn martin <kate@buoyant.io>
kflynn
pushed a commit
to linkerd/website
that referenced
this pull request
Nov 20, 2025
* feat(proxy-metrics): document outbound policy routing metrics the documentation of our proxy metrics has not kept pace with all of the exciting telemetry that has been introduced since #1599 documented the state of our authorization policy metrics. this commit reworks the documentation to exhaustively document the families of metrics exported by the proxy. this commit does not introduce mention of the _inbound_ metrics that have been added, but does rename this section to "_Endpoint Metrics_" in order to be compatible with the future addition of `inbound_http_route_*`, `inbound_http_route_backend_*`, `inbound_grpc_route*`, and `inbound_grpc_route_backend_*` metrics. see: * linkerd/linkerd2-proxy#2377 * linkerd/linkerd2-proxy#2380 * linkerd/linkerd2-proxy#3086 * linkerd/linkerd2-proxy#3308 * linkerd/linkerd2-proxy#3334 Signed-off-by: katelyn martin <kate@buoyant.io> * chore(markdownlint): allow duplicate "labels" headers Signed-off-by: katelyn martin <kate@buoyant.io> --------- Signed-off-by: katelyn martin <kate@buoyant.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When performing policy-based routing, proxies may dispatch requests through per-route backend configurations. In order to illustrate how routing rules apply and how backend distributions are being honored, this change adds two new metrics:
outbound_http_route_backend_requests_total; andoutbound_grpc_route_backend_requests_totalEach of these metrics includes labels that identify a routes parent (i.e. a Service), the route resource being used, and the backend resource being used.
This implementation does NOT implement any form of metrics eviction for these new metrics. This is tolerable for the short term as the cardinality of services and routes is generally much less than the cardinality of individual endpoints (where we do require timeout/eviction for metrics).