outbound: Report per-route-backend request count metrics#2372
Merged
Conversation
52e55bb to
f51a651
Compare
When performing policy-based routing, proxies may dispatch requests through per-route backend configurations. In order to illustrate how routing rules apply and how backend distributions are being honored, this change adds two new metrics: * `outbound_http_route_backend_requests_total`; and * `outbound_grpc_route_backend_requests_total` Each of these metrics includes labels that identify a routes parent (i.e. a Service), the route resource being used, and the backend resource being used. This implementation does NOT implement any form of metrics eviction for these new metrics. This is tolerable for the short term as the cardinality of services and routes is generally much less than the cardinality of individual endpoints (where we do require timeout/eviction for metrics).
f51a651 to
df78f29
Compare
olix0r
commented
Apr 2, 2023
Comment on lines
+80
to
+83
| write!(f, "{scope}_group=\"{}\"", meta.group())?; | ||
| write!(f, ",{scope}_kind=\"{}\"", meta.kind())?; | ||
| write!(f, ",{scope}_namespace=\"{}\"", meta.namespace())?; | ||
| write!(f, ",{scope}_name=\"{}\"", meta.name())?; |
Member
Author
There was a problem hiding this comment.
I don’t think these labels are sufficient.
Without port information, we can’t differentiate between ports on a service.
Should port information be reflected through meta.section()? Or do we need a separate field to reflect ports?
How are ports handled in the Gateway API?
Member
Author
There was a problem hiding this comment.
Gateway API parent references may include either/both sectionName and port (number) fields.
hawkw
added a commit
to linkerd/linkerd2
that referenced
this pull request
Apr 4, 2023
This proxy release adds new `outbound_http_route_backend_requests_total` and `outbound_grpc_route_backend_requests_total` metrics, which can be used to track how routing rules and backend distributions apply to requests. These metrics contain labels describing the route's parent (i.e. a Service), the route resource being used, and the backend resource being used by each request. In addition, this proxy release adds new `INFO`-level logs for failure accrual, and updates logging contexts with Kubernetes resource metadata. --- * outbound: Emit INFO-level logs on failure accrual changes (linkerd/linkerd2-proxy#2373) * app: Rename Metrics types as InboundMetrics and OutboundMetrics (linkerd/linkerd2-proxy#2376) * outbound: Configure balancers with service metadata (linkerd/linkerd2-proxy#2374) * outbound: Report per-route-backend request count metrics (linkerd/linkerd2-proxy#2372) * outbound: Report per-route-backend request count metrics (linkerd/linkerd2-proxy#2377)
hawkw
added a commit
to linkerd/linkerd2
that referenced
this pull request
Apr 5, 2023
This proxy release adds new `outbound_http_route_backend_requests_total` and `outbound_grpc_route_backend_requests_total` metrics, which can be used to track how routing rules and backend distributions apply to requests. These metrics contain labels describing the route's parent (i.e. a Service), the route resource being used, and the backend resource being used by each request. In addition, this proxy release adds new `INFO`-level logs for failure accrual, and updates logging contexts with Kubernetes resource metadata. --- * outbound: Emit INFO-level logs on failure accrual changes (linkerd/linkerd2-proxy#2373) * app: Rename Metrics types as InboundMetrics and OutboundMetrics (linkerd/linkerd2-proxy#2376) * outbound: Configure balancers with service metadata (linkerd/linkerd2-proxy#2374) * outbound: Report per-route-backend request count metrics (linkerd/linkerd2-proxy#2372) * outbound: Report per-route-backend request count metrics (linkerd/linkerd2-proxy#2377)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When performing policy-based routing, proxies may dispatch requests through per-route backend configurations. In order to illustrate how routing rules apply and how backend distributions are being honored, this change adds two new metrics:
outbound_http_route_backend_requests_total; andoutbound_grpc_route_backend_requests_totalEach of these metrics includes labels that identify a routes parent (i.e. a Service), the route resource being used, and the backend resource being used.
This implementation does NOT implement any form of metrics eviction for these new metrics. This is tolerable for the short term as the cardinality of services and routes is generally much less than the cardinality of individual endpoints (where we do require timeout/eviction for metrics).