outbound: Fix incorrect l5d-proxy-connection logs#2344
Merged
Conversation
160133d to
6d1468b
Compare
275c301 to
3b6e905
Compare
We've received reports of proxies inexplicibly emiting log lines including "Received unmeshed response with l5d-proxy-connection set". These messages may arise when the endpoint stack returns a synthesized response. Furthermore, we have a note in the code explaining that this connection closure logic should not apply to load balanced requests, though it currently is. This change fixes both of these issues: * The proxy_connection_close module is renamed to handle_proxy_error_headers. * It is now only applied in the endpoint stack. It doesn't make any sense for it to be applied in the server stack, since we'll already have cleared any headers set by peers. Removing this module prevents the application of teardown logic on synthetic responses. * `NewHandleProxyErrorHeaders` is now configured by a `Closable` parameter that determines whether teardown logic should be applied. This parameter is only enabled when forwarding to a single endpoint. No teardown logic is applied when load balancing. * In a future change, we should stop emitting l5d-proxy-connection when synthesizing outbound responses.
3b6e905 to
78ffb2d
Compare
hawkw
approved these changes
Mar 20, 2023
Comment on lines
+60
to
+61
| X: svc::ExtractParam<Closable, T>, | ||
| X: svc::ExtractParam<tls::ConditionalClientTls, T>, |
Contributor
There was a problem hiding this comment.
do we really want to use the same ExtractParam impl to extract both of these? i realize this probably doesn't actually matter as it looks like in practice, the ExtractParam impl is currently always (), and this is just here for future-proofing...
Member
Author
There was a problem hiding this comment.
Yeah, we can change it later if it needs to be changed.
olix0r
added a commit
to linkerd/linkerd2
that referenced
this pull request
Mar 21, 2023
Both outbound and gateway proxies now resolve client policies from the OutboundPolicies API. When the outbound proxy attempts to discover a policy and the policy controller returns NotFound, it synthesizes a default policy from the discovered ServiceProfile. However, when the gateway proxy receives a NotFound, it will currently fail the connection, based on the assumption that only valid cluster DNS names are gatewayed (and not arbitrary IPs that might be forwards). Unfortunately, this is not quite true. Gateway proxies may attempt to discover cluster DNS names that are Pod DNS names, rather than Service DNS names, and the policy controller will return NotFound for those names. This branch therefore changes the gateway proxy to also synthesize default ClientPolicies based on the ServiceProfile when receiving a NotFound status. Some of the code for synthesizing a client policy from a ServiceProfile that's currently used in the outbound proxy was factored out so that it could be reused here. --- * gateway: move discovery resolver into its own file (linkerd/linkerd2-proxy#2343) * outbound: Fix incorrect l5d-proxy-connection logs (linkerd/linkerd2-proxy#2344) * gateway: synthesize ClientPolicies when the controller returns `NotFound` (linkerd/linkerd2-proxy#2333) Signed-off-by: Oliver Gould <ver@buoyant.io>
olix0r
added a commit
to linkerd/linkerd2
that referenced
this pull request
Mar 21, 2023
* proxy: v2.193.0 This proxy release changes the multicluster gateway to discover Gateway API routes via the `OutboundPolicy` API. This builds on the similar changes to the outbound proxy in v2.192. --- * gateway: discover client policies from the policy controller (linkerd/linkerd2-proxy#2315) * build(deps): bump windows_x86_64_msvc from 0.42.1 to 0.42.2 (linkerd/linkerd2-proxy#2319) * build(deps): bump proc-macro2 from 1.0.51 to 1.0.52 (linkerd/linkerd2-proxy#2320) * outbound: Apply filters to outbound requests (linkerd/linkerd2-proxy#2260) * test: add mock client policy resolver (linkerd/linkerd2-proxy#2314) * build(deps): bump tj-actions/changed-files from 35.6.4 to 35.7.0 (linkerd/linkerd2-proxy#2318) * build(deps): bump axum from 0.6.10 to 0.6.11 (linkerd/linkerd2-proxy#2321) * build(deps): bump ryu from 1.0.12 to 1.0.13 (linkerd/linkerd2-proxy#2322) * build(deps): bump windows_x86_64_gnullvm from 0.42.1 to 0.42.2 (linkerd/linkerd2-proxy#2323) * outbound: Eagerly cancel synthesized profile task (linkerd/linkerd2-proxy#2317) * outbound: Simplify discovery debug logging (linkerd/linkerd2-proxy#2316) * build(deps): bump tj-actions/changed-files from 35.6.1 to 35.6.4 (linkerd/linkerd2-proxy#2309) * proxy: v2.193.1 * outbound: fix `Balance::Dispatch` "authority" labels (linkerd/linkerd2-proxy#2332) * outbound: refactor `discover::resolver` into a method (linkerd/linkerd2-proxy#2331) * proxy: v2.193.2 Both outbound and gateway proxies now resolve client policies from the OutboundPolicies API. When the outbound proxy attempts to discover a policy and the policy controller returns NotFound, it synthesizes a default policy from the discovered ServiceProfile. However, when the gateway proxy receives a NotFound, it will currently fail the connection, based on the assumption that only valid cluster DNS names are gatewayed (and not arbitrary IPs that might be forwards). Unfortunately, this is not quite true. Gateway proxies may attempt to discover cluster DNS names that are Pod DNS names, rather than Service DNS names, and the policy controller will return NotFound for those names. This branch therefore changes the gateway proxy to also synthesize default ClientPolicies based on the ServiceProfile when receiving a NotFound status. Some of the code for synthesizing a client policy from a ServiceProfile that's currently used in the outbound proxy was factored out so that it could be reused here. --- * gateway: move discovery resolver into its own file (linkerd/linkerd2-proxy#2343) * outbound: Fix incorrect l5d-proxy-connection logs (linkerd/linkerd2-proxy#2344) * gateway: synthesize ClientPolicies when the controller returns `NotFound` (linkerd/linkerd2-proxy#2333) Signed-off-by: Oliver Gould <ver@buoyant.io> --------- Signed-off-by: Oliver Gould <ver@buoyant.io> Co-authored-by: Oliver Gould <ver@buoyant.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We've received reports of proxies inexplicibly emiting log lines including "Received unmeshed response with l5d-proxy-connection set". These messages may arise when the endpoint stack returns a synthesized response.
Furthermore, we have a note in the code explaining that this connection closure logic should not apply to load balanced requests, though it currently is.
This change fixes both of these issues:
Synthesizedmarker to responses so that other modules can determine when a response was generated locally.proxy_connection_closemodule is updated so that we ignore responses with this extension set.Connection: closebehavior unless the newClosablemarker extension is present.