tracing: centralized span registry #61407
Description
Describe the problem
The registry of inflight spans today only captures root spans. We construct an in-memory tree of child spans that derive from the parent spans, and are thus only accessible through tree traversal. This makes a few things more cumbersome, compared to the alternative (below).
cockroach/pkg/util/tracing/tracer.go
Lines 121 to 141 in 5b72a34
cockroach/pkg/util/tracing/crdbspan.go
Lines 74 to 76 in 5b72a34
Proposed improvement
We can centralize the storage of spans by storing them directly in the same in-memory registry (keyed by Span ID) and have parent spans only refer to child spans by IDs, rather than direct pointers. See #59310 for a prototype of what that could look like.
It comes with a few upsides.
- Memory accounting: we'll have a centralized place for where it all happens, instead of threading it in through individual spans and child spans. It also lets us introduce a global debug switch to drop all structured recordings we’re holding onto, or control what the RSS is for the current set of observable structured events.
- Historical introspection: by decoupling the garbage collection of structured events from the lifecycle of the span itself, we can go back in time (very briefly) and see what the collected events were. Right now we can only introspect in-flight spans. This might not come in super handy, but it also might, given spans are very short-lived in practice.
- We can refer to child spans by Span ID, instead of building an in-memory graph to traverse through. It helps obviate memory issues and trickiness we're observing with forked spans that outlive parents, and parents that might still want to hold onto references to child spans for traversal. [dnm] tracing: clean up use of auto collection across tasks #59391.
- Another perk of having span-centric storage comes from util/tracing,sql: add builtin to set trace spans' verbosity #61353. At the time of writing, only root spans are accessible through the registry. So a built-in that's able to selectively toggle the verbosity of a span is of limited use; we can't isolate a random child span because it's only indirectly accessible (we'd need to traverse the tree of the root span).