Another interesting case we should consider: If the resolver never returns a result and RPCs fail due to DEADLINE_EXCEEDED, the error message should indicate that the resolver never returned a result.

I agree, I also think this "waiting to acquire resource X" would be helpful debug info for a lot of other things that are currently buried in logs. For example, there are a number of interesting things that an RPC can "stall" on:

RPC can be stalled because the channel's LB policy hasn't picked a connection for the RPC
- the connection-pick might be incomplete because:
  a) name resolution hasn't completed
  b) load balancer isn't responding (if relevant to the LB policy in use)
  c) connections are all in "connecting" state.
  ... etc.
RPC can get stuck waiting for a slot in it's http/2 connection's "max concurrent streams" queue.
RPC can have an active send message that it hasn't yet written to the wire because it's waiting for space in the outgoing HTTP/2 flow control window
.....etc.

If these things could represented e.g. with some counters, and be accounted for on a per-RPC basis, then they could then be tied back to each RPC's context and be surfaced up to users either in an RPC's error message or perhaps alongside it - IMO that would useful to debug why some RPC's fail and potentially also why some RPC's are slow (e.g. determining that some specific high latency RPC took a long time because it was spending all of it's time waiting for name resolution to finish).

I'm not sure if perhaps the census context could be leveraged for these kinds of things....

RPC failure status should include all useful information about the failure #22883

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions