outbound: Refactor stack target types by olix0r · Pull Request #2210 · linkerd/linkerd2-proxy

olix0r · 2023-02-03T16:43:22Z

To support a new configuration API, we want to update the outbound stack
to compose the following stacks:

Routing discovery;
Protocol switching;
Protocol-specific logical routing;
Concrete forwarding / load balancing;
Endpoint dispatching.

To do this, we update the existing stacks so that:

each stack accepts generic input targets (with enumerated Param
requirements); and
each module defines all necessary internal target types, including
those passed to inner stacks.

This decouples stack configuration from the specifics of discovery:
tests can satisfy stack parameters without having to stub out all of the
discovery types and we are setup to change discovery implementation
details moving forward.

This requires updating the gateway and ingress stacks in addition to the
(default) sidecar stack.

Generally, we prefer duplicating similar target types to coupling
modules. In followup changes, we can introduce macros to reduce Param
boilerplate if necessary.

Changes include:

A new Gateway builder type (analogous to Inbound and Outbound);
A new Gateway::server stack handles discovery and protocol
switching;
Service cacheing is performed for the http and opaq stacks (to
preserve load balancer state, etc) using only the information needed
for those stacks;
Concrete stacks provide the only data-plane queue. This enables
outer stacks to be cloneable. This also means that the concrete stacks
are the only stacks with stack metrics;
The switch_logical module is removed;
The http::detect module is moved into the protocol stack, which
now supports discovery-hinted protocols to bypass detection.
The logical and endpoint modules are removed in favor of
protocol-specific modules.

We want to change the shape of the outbound stack--removing `switch_logical` so that forwarding/balancing decisions are made in concrete stacks. We also want to move away from using profiles exclusively, adding new discovery types. This is complicated by our use of fixed target types because it's difficult to make incremental changes. Furthermore, a stack's requirements are not clearly documented. This change addresses this as follows: * The logical, concrete, and endpoint stacks are generic over their input types. Each stack enumerates required parameters from these targets. * Each of these stacks declare module-local types that are passed to inner services. These types must expose params to satisfy all of the requirements of inner stacks. * `tcp::concrete` and `tcp::logical` have been renamed to `opaque::concrete` and `opaque::logical` to reflect that they are an alternative to the http stack. The HTTP stack continues to incorporate the `tcp::endpoint` stack. The stack helpers have been renamed accordingly.

linkerd/app/outbound/src/ingress.rs

To support a new configuration API, we want to update the outbound stack to compose the following stacks: * Routing discovery; * Protocol switching; * Protocol-specific logical routing; * Concrete forwarding / load balancing; * Endpoint dispatching. To do this, we update the existing stacks so that: * each stack accepts generic input targets (with enumerated `Param` requirements); and * each module defines all necessary internal target types, including those passed to inner stacks. This decouples stack configuration from the specifics of discovery: tests can satisfy stack parameters without having to stub out all of the discovery types and we are setup to change discovery implementation details moving forward. This requires updating the gateway and ingress stacks in addition to the (default) sidecar stack. Generally, we prefer duplicating similar target types to coupling modules. In followup changes, we can introduce macros to reduce `Param` boilerplate if necessary. Changes include: * A new `Gateway` builder type (analogous to `Inbound` and `Outbound`); * A new `Gateway::server` stack handles discovery and protocol switching; * Service cacheing is performed for the http and opaq stacks (to preserve load balancer state, etc) using only the information needed for those stacks; * Concrete stacks provide the only data-plane queue. This enables outer stacks to be cloneable. This also means that the concrete stacks are the only stacks with stack metrics; * The `switch_logical` module is removed; * The `http::detect` module is moved into the `protocol` stack, which now supports discovery-hinted protocols to bypass detection. * The `logical` and `endpoint` modules are removed in favor of protocol-specific modules.

hawkw

Despite the size of the change, this branch ended up being surprisingly straightforward to review --- this makes a lot of sense! I commented on a lot of fairly small nits, but the overall approach is great.

linkerd/app/outbound/src/lib.rs

linkerd/app/outbound/src/opaq/concrete.rs

linkerd/app/outbound/src/protocol.rs

linkerd/app/outbound/src/opaq/logical.rs

linkerd/app/outbound/src/http/concrete.rs

linkerd/app/outbound/src/http.rs

This release includes many internal changes to prepare for the new client policy API. Stack metric label values have changed to reflect the new shape of the outbound proxy. --- * build(deps): bump try-lock from 0.2.3 to 0.2.4 (linkerd/linkerd2-proxy#2139) * build(deps): bump regex from 1.7.0 to 1.7.1 (linkerd/linkerd2-proxy#2145) * build(deps): bump tokio from 1.24.0 to 1.24.1 (linkerd/linkerd2-proxy#2144) * Parameterize the load balancer stack (linkerd/linkerd2-proxy#2142) * build(deps): bump prost-types from 0.11.5 to 0.11.6 (linkerd/linkerd2-proxy#2147) * build(deps): bump prost from 0.11.5 to 0.11.6 (linkerd/linkerd2-proxy#2148) * build(deps): bump prost-build from 0.11.5 to 0.11.6 (linkerd/linkerd2-proxy#2149) * stack: add `AnnotateError` middleware (linkerd/linkerd2-proxy#2158) * Fix proxy-core dependencies (linkerd/linkerd2-proxy#2163) * build(deps): bump tj-actions/changed-files from 35.3.1 to 35.4.1 (linkerd/linkerd2-proxy#2153) * orig_proto: don't set `connection: close` on errors (linkerd/linkerd2-proxy#2171) * build(deps): bump bumpalo from 3.11.1 to 3.12.0 (linkerd/linkerd2-proxy#2166) * build(deps): bump proc-macro2 from 1.0.49 to 1.0.50 (linkerd/linkerd2-proxy#2165) * build(deps): bump tj-actions/changed-files from 35.4.1 to 35.4.4 (linkerd/linkerd2-proxy#2172) * build(deps): bump windows_x86_64_msvc from 0.42.0 to 0.42.1 (linkerd/linkerd2-proxy#2164) * configure buffers from target `Param`s (linkerd/linkerd2-proxy#2173) * Simplify profile discovery (linkerd/linkerd2-proxy#2170) * stack: Unify AnnotateError and MapErr (linkerd/linkerd2-proxy#2180) * build(deps): bump windows_aarch64_msvc from 0.42.0 to 0.42.1 (linkerd/linkerd2-proxy#2176) * build(deps): bump async-trait from 0.1.61 to 0.1.63 (linkerd/linkerd2-proxy#2177) * build(deps): bump tokio from 1.24.1 to 1.24.2 (linkerd/linkerd2-proxy#2178) * Add the `meshtls-boring-fips` feature flag (linkerd/linkerd2-proxy#2168) * Update HTTP error responder to log version info (linkerd/linkerd2-proxy#2182) * build(deps): bump which from 4.3.0 to 4.4.0 (linkerd/linkerd2-proxy#2185) * build(deps): bump derive_arbitrary from 1.2.2 to 1.2.3 (linkerd/linkerd2-proxy#2184) * build(deps): bump unicode-bidi from 0.3.8 to 0.3.10 (linkerd/linkerd2-proxy#2183) * build(deps): bump linkerd/dev from 38 to 39 (linkerd/linkerd2-proxy#2175) * add target metadata to error contexts (linkerd/linkerd2-proxy#2162) * build(deps): bump rustls from 0.20.7 to 0.20.8 (linkerd/linkerd2-proxy#2187) * build(deps): bump ahash from 0.8.2 to 0.8.3 (linkerd/linkerd2-proxy#2188) * build(deps): bump matches from 0.1.9 to 0.1.10 (linkerd/linkerd2-proxy#2189) * Rename MakeThunk to NewThunk (linkerd/linkerd2-proxy#2197) * Add `NewQueueWithoutTimeout` (linkerd/linkerd2-proxy#2196) * Cache discovery results independently of proxy stacks (linkerd/linkerd2-proxy#2195) * core: Rename Stack utilities for clarity (linkerd/linkerd2-proxy#2199) * gateway: Unify discovery for HTTP & opaque stacks (linkerd/linkerd2-proxy#2198) * build(deps): bump libfuzzer-sys from 0.4.5 to 0.4.6 (linkerd/linkerd2-proxy#2193) * build(deps): bump arbitrary from 1.2.2 to 1.2.3 (linkerd/linkerd2-proxy#2191) * http: Remove `Clone` requirement in servers (linkerd/linkerd2-proxy#2200) * build(deps): bump either from 1.8.0 to 1.8.1 (linkerd/linkerd2-proxy#2192) * build(deps): bump bytes from 1.3.0 to 1.4.0 (linkerd/linkerd2-proxy#2202) * build(deps): bump cc from 1.0.78 to 1.0.79 (linkerd/linkerd2-proxy#2203) * build(deps): bump tj-actions/changed-files from 35.4.4 to 35.5.1 (linkerd/linkerd2-proxy#2211) * build(deps): bump tokio from 1.24.2 to 1.25.0 (linkerd/linkerd2-proxy#2206) * build(deps): bump miniz_oxide from 0.6.2 to 0.6.4 (linkerd/linkerd2-proxy#2207) * build(deps): bump async-trait from 0.1.63 to 0.1.64 (linkerd/linkerd2-proxy#2205) * Split outbound test modules into files (linkerd/linkerd2-proxy#2213) * Disable broken tests (linkerd/linkerd2-proxy#2214) * Add traceparent header parsing for w3c tracecontext (linkerd/linkerd2-proxy#2179) * Simplify the `Resolve` trait alias (linkerd/linkerd2-proxy#2218) * downgrade `miniz_oxide` from yanked 0.6.4 to 0.6.2 (linkerd/linkerd2-proxy#2219) * build(deps): bump heck from 0.4.0 to 0.4.1 (linkerd/linkerd2-proxy#2215) * ci: Fix check-each workflow (linkerd/linkerd2-proxy#2222) * Use a cascading stack with protocol detection (linkerd/linkerd2-proxy#2221) * ci: Fix quotation in list-crates (linkerd/linkerd2-proxy#2225) * outbound: Split out separate 'opaq' modules (linkerd/linkerd2-proxy#2224) * Update `linkerd2-proxy-api` to v0.8.0 (linkerd/linkerd2-proxy#2223) * outbound: Lint stack target types (linkerd/linkerd2-proxy#2226) * outbound: Split sidecar and ingress stack modules (linkerd/linkerd2-proxy#2227) * gateway: Split 'http' and 'opaq' modules (linkerd/linkerd2-proxy#2230) * test: Disable tap::rejects_incorrect_identity_when_identity_is_expected (linkerd/linkerd2-proxy#2231) * outbound: Improve discovery cache test (linkerd/linkerd2-proxy#2233) * integration: add destination update builders (linkerd/linkerd2-proxy#2232) * Rename linkerd-server-policy to linkerd-proxy-server-policy (linkerd/linkerd2-proxy#2235) * integration: add test for direct HTTP connections (linkerd/linkerd2-proxy#2234) * outbound: Refactor stack target types (linkerd/linkerd2-proxy#2210) * Add client-policy types (linkerd/linkerd2-proxy#2236) Signed-off-by: Oliver Gould <ver@buoyant.io>

This release includes many internal changes to prepare for the new client policy API. Stack metric label values have changed to reflect the new shape of the outbound proxy. This change also includes some test improvements that helped debug an issue while merging this. --- * build(deps): bump try-lock from 0.2.3 to 0.2.4 (linkerd/linkerd2-proxy#2139) * build(deps): bump regex from 1.7.0 to 1.7.1 (linkerd/linkerd2-proxy#2145) * build(deps): bump tokio from 1.24.0 to 1.24.1 (linkerd/linkerd2-proxy#2144) * Parameterize the load balancer stack (linkerd/linkerd2-proxy#2142) * build(deps): bump prost-types from 0.11.5 to 0.11.6 (linkerd/linkerd2-proxy#2147) * build(deps): bump prost from 0.11.5 to 0.11.6 (linkerd/linkerd2-proxy#2148) * build(deps): bump prost-build from 0.11.5 to 0.11.6 (linkerd/linkerd2-proxy#2149) * stack: add `AnnotateError` middleware (linkerd/linkerd2-proxy#2158) * Fix proxy-core dependencies (linkerd/linkerd2-proxy#2163) * build(deps): bump tj-actions/changed-files from 35.3.1 to 35.4.1 (linkerd/linkerd2-proxy#2153) * orig_proto: don't set `connection: close` on errors (linkerd/linkerd2-proxy#2171) * build(deps): bump bumpalo from 3.11.1 to 3.12.0 (linkerd/linkerd2-proxy#2166) * build(deps): bump proc-macro2 from 1.0.49 to 1.0.50 (linkerd/linkerd2-proxy#2165) * build(deps): bump tj-actions/changed-files from 35.4.1 to 35.4.4 (linkerd/linkerd2-proxy#2172) * build(deps): bump windows_x86_64_msvc from 0.42.0 to 0.42.1 (linkerd/linkerd2-proxy#2164) * configure buffers from target `Param`s (linkerd/linkerd2-proxy#2173) * Simplify profile discovery (linkerd/linkerd2-proxy#2170) * stack: Unify AnnotateError and MapErr (linkerd/linkerd2-proxy#2180) * build(deps): bump windows_aarch64_msvc from 0.42.0 to 0.42.1 (linkerd/linkerd2-proxy#2176) * build(deps): bump async-trait from 0.1.61 to 0.1.63 (linkerd/linkerd2-proxy#2177) * build(deps): bump tokio from 1.24.1 to 1.24.2 (linkerd/linkerd2-proxy#2178) * Add the `meshtls-boring-fips` feature flag (linkerd/linkerd2-proxy#2168) * Update HTTP error responder to log version info (linkerd/linkerd2-proxy#2182) * build(deps): bump which from 4.3.0 to 4.4.0 (linkerd/linkerd2-proxy#2185) * build(deps): bump derive_arbitrary from 1.2.2 to 1.2.3 (linkerd/linkerd2-proxy#2184) * build(deps): bump unicode-bidi from 0.3.8 to 0.3.10 (linkerd/linkerd2-proxy#2183) * build(deps): bump linkerd/dev from 38 to 39 (linkerd/linkerd2-proxy#2175) * add target metadata to error contexts (linkerd/linkerd2-proxy#2162) * build(deps): bump rustls from 0.20.7 to 0.20.8 (linkerd/linkerd2-proxy#2187) * build(deps): bump ahash from 0.8.2 to 0.8.3 (linkerd/linkerd2-proxy#2188) * build(deps): bump matches from 0.1.9 to 0.1.10 (linkerd/linkerd2-proxy#2189) * Rename MakeThunk to NewThunk (linkerd/linkerd2-proxy#2197) * Add `NewQueueWithoutTimeout` (linkerd/linkerd2-proxy#2196) * Cache discovery results independently of proxy stacks (linkerd/linkerd2-proxy#2195) * core: Rename Stack utilities for clarity (linkerd/linkerd2-proxy#2199) * gateway: Unify discovery for HTTP & opaque stacks (linkerd/linkerd2-proxy#2198) * build(deps): bump libfuzzer-sys from 0.4.5 to 0.4.6 (linkerd/linkerd2-proxy#2193) * build(deps): bump arbitrary from 1.2.2 to 1.2.3 (linkerd/linkerd2-proxy#2191) * http: Remove `Clone` requirement in servers (linkerd/linkerd2-proxy#2200) * build(deps): bump either from 1.8.0 to 1.8.1 (linkerd/linkerd2-proxy#2192) * build(deps): bump bytes from 1.3.0 to 1.4.0 (linkerd/linkerd2-proxy#2202) * build(deps): bump cc from 1.0.78 to 1.0.79 (linkerd/linkerd2-proxy#2203) * build(deps): bump tj-actions/changed-files from 35.4.4 to 35.5.1 (linkerd/linkerd2-proxy#2211) * build(deps): bump tokio from 1.24.2 to 1.25.0 (linkerd/linkerd2-proxy#2206) * build(deps): bump miniz_oxide from 0.6.2 to 0.6.4 (linkerd/linkerd2-proxy#2207) * build(deps): bump async-trait from 0.1.63 to 0.1.64 (linkerd/linkerd2-proxy#2205) * Split outbound test modules into files (linkerd/linkerd2-proxy#2213) * Disable broken tests (linkerd/linkerd2-proxy#2214) * Add traceparent header parsing for w3c tracecontext (linkerd/linkerd2-proxy#2179) * Simplify the `Resolve` trait alias (linkerd/linkerd2-proxy#2218) * downgrade `miniz_oxide` from yanked 0.6.4 to 0.6.2 (linkerd/linkerd2-proxy#2219) * build(deps): bump heck from 0.4.0 to 0.4.1 (linkerd/linkerd2-proxy#2215) * ci: Fix check-each workflow (linkerd/linkerd2-proxy#2222) * Use a cascading stack with protocol detection (linkerd/linkerd2-proxy#2221) * ci: Fix quotation in list-crates (linkerd/linkerd2-proxy#2225) * outbound: Split out separate 'opaq' modules (linkerd/linkerd2-proxy#2224) * Update `linkerd2-proxy-api` to v0.8.0 (linkerd/linkerd2-proxy#2223) * outbound: Lint stack target types (linkerd/linkerd2-proxy#2226) * outbound: Split sidecar and ingress stack modules (linkerd/linkerd2-proxy#2227) * gateway: Split 'http' and 'opaq' modules (linkerd/linkerd2-proxy#2230) * test: Disable tap::rejects_incorrect_identity_when_identity_is_expected (linkerd/linkerd2-proxy#2231) * outbound: Improve discovery cache test (linkerd/linkerd2-proxy#2233) * integration: add destination update builders (linkerd/linkerd2-proxy#2232) * Rename linkerd-server-policy to linkerd-proxy-server-policy (linkerd/linkerd2-proxy#2235) * integration: add test for direct HTTP connections (linkerd/linkerd2-proxy#2234) * outbound: Refactor stack target types (linkerd/linkerd2-proxy#2210) * Add client-policy types (linkerd/linkerd2-proxy#2236)

olix0r added 19 commits February 2, 2023 04:08

opaque generic

3a3a036

Rename tcp to opaque

10a63df

split http endpoint tests into a separate file

ad675a8

wip

a388186

wip

ee8b991

wip:ingress

ea1d173

wip: gateway compiles!

c13d439

make the http concrete stack generic

c551223

new concrete type

9e30f2d

Make the logical HTTP stack generic

93da668

Remove the old concrete target type

cccdbf6

more logical stack massaging

de1fd4c

cleanup

77d8c6b

make switch_logical generic

45f5e5e

Update concrete stacks to handle forwarding.

7fe8814

wip: http logical

3b37863

wip:detect

43bac57

ingress compiles

3b7ad30

olix0r commented Feb 3, 2023

View reviewed changes

linkerd/app/outbound/src/ingress.rs Show resolved Hide resolved

olix0r added 10 commits February 3, 2023 18:06

wip:gw

b83b297

split out sidecar module

62c47ac

wip:gateway

fd93314

wip

833d878

wip: make detect stack eager

417396a

wip: outbound

96d2c2f

http server cleanup

7f6053e

wip gateway

9de71bd

wip:gateway

5c1ccda

wip:gateway

e084085

olix0r added 4 commits February 14, 2023 21:46

simplify Discovery target

4708c05

smpfy

c196687

remove discover_cache utility

de5a766

Merge branch 'main' into ver/unforward

9cfdcab

olix0r mentioned this pull request Feb 14, 2023

outbound: Make stacks generic over target types #2208

Closed

olix0r added 11 commits February 14, 2023 23:13

gateway: comments/clarity

8a0e023

comments

f2ba079

more gateway simplification

6bd5aaa

cleanup outbound stack

4773292

fixup

ea830b9

reexport target

e33e2fa

Rename "Target" to "Logical", make modules private

3d230f2

unused impl

c76ca12

remove unused params

cfde3af

touchup

c628795

olix0r force-pushed the ver/unforward branch from 84e7184 to 7cf07e3 Compare February 15, 2023 02:21

olix0r changed the title ~~wip: Per-protocol outbound stacks~~ outbound: Refactor stack target types Feb 15, 2023

olix0r marked this pull request as ready for review February 15, 2023 02:22

olix0r requested a review from a team as a code owner February 15, 2023 02:22

olix0r added 2 commits February 15, 2023 03:21

Merge branch 'main' into ver/unforward

ee8ced4

Merge branch 'main' into ver/unforward

404e17f

olix0r self-assigned this Feb 15, 2023

hawkw approved these changes Feb 15, 2023

View reviewed changes

Review feedback

638535f

olix0r merged commit 91bbbd4 into main Feb 15, 2023

olix0r deleted the ver/unforward branch February 15, 2023 21:49

olix0r mentioned this pull request Feb 16, 2023

proxy: v2.190.1 linkerd/linkerd2#10342

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

outbound: Refactor stack target types#2210

outbound: Refactor stack target types#2210
olix0r merged 105 commits intomainfrom
ver/unforward

olix0r commented Feb 3, 2023 •

edited

Loading

Uh oh!

Uh oh!

hawkw left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

olix0r commented Feb 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

hawkw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

olix0r commented Feb 3, 2023 •

edited

Loading