-
Notifications
You must be signed in to change notification settings - Fork 4.1k
tracing: support span linking for spans shared by multiple traces #70864
Description
During query processing, we frequently wait for asynchronous operations that are shared by multiple callers. These operations are currently absent from traces. For example:
-
Raft processing: a single round of Raft can include log entries for many commands coming from many queries and clients.
-
Asynchronous transaction writes: most transaction writes are pipelined, i.e. sent off asynchronously.
-
Range descriptor lookups: if a descriptor lookup is already in flight, other callers will coalesce their lookups, i.e. wait for the result of the in-flight request.
In these cases (and probably many others), query tracing goes dark once it hits these operations, which is unfortunate -- especially in the case of Raft, where it would be very useful to follow e.g. the replication and fsync latencies of each query.
What we probably need to do here is to start a new span for these operations, and then return that span up to all callers who are waiting for the results of the operations, such that they can link the shared span into their own traces. Note that the callers may well arrive after the operation has started, as in the case of range descriptor lookup coalescing (i.e. a query can find an in-flight lookup that started 100ms ago, and wait for that to complete).
This may also require support for this in the trace data formats. @andreimatei mentioned in #70741 (review) that there is some support for this in OpenTelemetry, but that it's still somewhat in flux.
Jira issue: CRDB-10256