see #4251, which is based upon this branch. that pull request proposes StreamLabel implementations that can be used to label metrics with HTTP or gRPC status codes, and for converting boxed errors into metric labels.

[..] I'm wary of having to express additional type assertions everywhere.
It would help to understand what flexibility we get for the verbosity
trade-off.

certainly. let me try and outline where i believe we are headed with this
change. first, let's restate where we are now, and where we would like to go.

we have a pair of MkStreamLabel and StreamLabel traits that are intended
for use in our metrics middleware layers. these allow us to abstract over
protocol-specific details for both HTTP and gRPC traffic, so that we can e.g.
count HTTP and gRPC status codes, without needing to reimplement separate
layers.

today, this pair of traits is used only within the
NewRecordResponse<L, X, M, N> layer, defined in linkerd-http-prom.
consumers of this middleware should record request duration or response duration
by using the NewRequestDuration<L, X, N> and NewResponseDuration<L, X, N>
aliases of this middleware. note that each of these include status code
telemetry as well.

today, the NewRecordResponse<L, X, M, N> layer is only used within the
outbound proxy. we would like to use it within the inbound proxy as well.
the inbound proxy's stack does not include the same retry facilities as the
outbound proxy, so two families of status code metrics would introduce a
redundant family of high-cardinality metrics.

so, our high-level goal is to decompose NewRecordResponse<L, X, M, N>
and in turn, NewRequestDuration<L, X, N> and NewResponseDuration<L, X, N>,
into two distinct layers that can be reused by both the inbound and outbound
proxies.

so, how does this change help us get there?

by loosening this constraint, we may define some reusable stream labeling
components (see #4251) for
common concerns like HTTP and gRPC status code labels or mapping errors to
labels, that can be shared by the inbound and outbound proxy. today, we can't
write such a component because this trait's prerequisite bounds demand that we
emit a type that implements EncodeLabelSet and friends.

for what it's worth, an alias like the LabelSet trait proposed here can also
help with some of the existing boilerplate in
linkerd_http_prom::record_response's submodules. see
1de0cd5.

now, note that MkStreamLabel and StreamLabel are currently tightly coupled
to this duration middleware. in
#4231, we hoisted this up a
level so that it could be reused in the future, but it currently is explicitly
aware of the notion of "status" and "duration" labels.

after we introduce a standalone status counting layer, we can remove these
metrics from the duration recording layer. and once that is done, we can in
turn, consolidate the separate StatusLabels and DurationLabels associated
types in MkStreamLabel and StreamLabel into a single associated type.

in this world, we end up with a StreamLabel trait that is shaped like this:

pub trait StreamLabel: Send + 'static {
    type Labels;
    fn init_response<B>(&mut self, rsp: &http::Response<B>);
    fn end_response(&mut self, trailers: Result<Option<&http::HeaderMap>, &Error>);
    fn labels(&self) -> Self::Labels;
}

there is, as you point out, a tradeoff here: we incur some complexity in
dependent code, but we will end up in a position that also greatly simplifies
the complex bounds in our duration middleware, and opens the door the sharing
labeling facilities between both proxies and other future middleware layers.

i've also noticed that after we extricate status code metrics from
the duration middleware, it wouldn't even need MkStreamLabel and StreamLabel.
today, we simply pass the route labels along, which are known in advance and
require no inspection of the response or trailers. should we remove that,
even more boilerplate will melt away in the wake of these changes!

so, i feel some conviction that in the long-term, this is worth the cost. i'm
curious to hear your thoughts. i hope that this helped illuminate the bigger
picture outside of the scope of this change, #4251, and #4252.