api: add new pendingStreamCreated on clientStreamTracer#10014
api: add new pendingStreamCreated on clientStreamTracer#10014YifeiZhuang merged 4 commits intogrpc:masterfrom
Conversation
core/src/main/java/io/grpc/internal/DelayedClientTransport.java
Outdated
Show resolved
Hide resolved
| headers.discardAll(tracingHeader); | ||
| headers.put(tracingHeader, span.getContext()); | ||
| } | ||
| span.addAnnotation("Real stream created."); |
There was a problem hiding this comment.
The existing annotation didn't include a period. We should do the same.
Is "real stream" meaningful to users? Should we instead say something like "Transport selected" or "Load balancer provided a transport"? Do we know what string C used here?
There was a problem hiding this comment.
C use "Delayed name resolution complete." and "Delayed LB pick complete."
It is a good idea to use the same descriptions, at least for the this new one.
grpc/grpc#31913
|
|
||
| @Override | ||
| public void createPendingStream() { | ||
| span.addAnnotation("Pending stream created."); |
There was a problem hiding this comment.
I don't this annotation gains us anything, as this will be the same as the start of the span.
There was a problem hiding this comment.
For The start of the span you mean when the stream tracer is created by the census interceptor, right?
That is around the same time as pending stream is created. The difference is that every stream has start of span moment, but only every stream has createPendingStream momemt.
There was a problem hiding this comment.
discussed offline, this needs to be added on parent span. Plus creatingPendingStream is not sufficiently mean name resolution delay. Maybe we will add follow up PR about name resolution delay timestamp.
2f7df74 to
5e2fc7a
Compare
b/259730220
go/grpc-java-new-trace-lines
Problem
Direct Path users (Big Table, Spanner) occasionally see spikes of RPC latencies and it is hard to debug the cause due to the lack of information. Users look for more information in tracing. Specifically, it requires a few timestamps in the RPC lifecycle .
Background
Before name resolution returns service configs, the ClientCalls are pendingCalls, the steam creation calls are saved.
After name resolution is done, calls are drained: this triggers stream creation, and at the same time the stream tracer is created from all the stream tracer factories. The stream created may not be real stream, instead it forks to two situations:
Client transport calls back streamCreate() on stream tracer (via StatsTraceContext).
Solution
public abstract class ClientStreamTracer extends StreamTracer {
public void pendingStreamCreated() {
}
…
}
cc. @mohanli-ml