Skip to content

tracing: support span linking for spans shared by multiple traces #70864

@erikgrinaker

Description

@erikgrinaker

During query processing, we frequently wait for asynchronous operations that are shared by multiple callers. These operations are currently absent from traces. For example:

  • Raft processing: a single round of Raft can include log entries for many commands coming from many queries and clients.

  • Asynchronous transaction writes: most transaction writes are pipelined, i.e. sent off asynchronously.

  • Range descriptor lookups: if a descriptor lookup is already in flight, other callers will coalesce their lookups, i.e. wait for the result of the in-flight request.

In these cases (and probably many others), query tracing goes dark once it hits these operations, which is unfortunate -- especially in the case of Raft, where it would be very useful to follow e.g. the replication and fsync latencies of each query.

What we probably need to do here is to start a new span for these operations, and then return that span up to all callers who are waiting for the results of the operations, such that they can link the shared span into their own traces. Note that the callers may well arrive after the operation has started, as in the case of range descriptor lookup coalescing (i.e. a query can find an in-flight lookup that started 100ms ago, and wait for that to complete).

This may also require support for this in the trace data formats. @andreimatei mentioned in #70741 (review) that there is some support for this in OpenTelemetry, but that it's still somewhat in flux.

Jira issue: CRDB-10256

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-kv-observabilityC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-kvKV Team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions