tracing: expand the active spans registry #72993
tracing: expand the active spans registry #72993craig[bot] merged 3 commits intocockroachdb:masterfrom
Conversation
c5e95c9 to
71c84a0
Compare
1815e4f to
6627c55
Compare
|
First commit is #73383 |
6627c55 to
7ab3eb4
Compare
rebased; all commits are original now. |
7ab3eb4 to
1303a78
Compare
rimadeodhar
left a comment
There was a problem hiding this comment.
Reviewed 1 of 1 files at r2, 5 of 5 files at r8, 5 of 5 files at r9, 3 of 3 files at r10, all commit messages.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @abarganier and @andreimatei)
-- commits, line 74 at r10:
Nit: Typo recordings
pkg/util/tracing/crdbspan.go, line 90 at r9 (raw file):
// If this parent is recording at the time when a child Finish()es, and the // respective childRef indicates that the child is to be included in the // paren't recording, then the child's recording is collected in
Nit: Superfluous apostrophe.
Benchmark the creation of both regular children and detached children. Their scalability is currently different, because the detached children are added to the registry. Release note: None
Before this patch, the registry only contained (local) roots. A child span was not part of the registry directly; it was part of it indirectly if the parent was recording at the moment when the child was opened. I believe the fact that a child of a non-recording span is not part of the registry is not principled, just path dependency. This patch makes child spans indirectly part of the registry even if the parent is not recording at the moment when the child is created, by having children always register themselves with the parent, even if the parent is not recording. I got rid of the maxChildrenPerSpan limit on the number of open children that a span keep track off. This limit was fairly non-sensical (it probably started as more sane but then was refactored into non-sense) as it applies to open spans but not to recordings of the finished children. The point of this limit originally was to bound the memory use of recordings. By acting only on open children, it had no such effect. Indeed, there's pretty much no reason to bound the number of open children tracked by a parent, and so this patch removes the limit. In the next commit we'll get a new limit that refers to sizes of recordings. --- This patch also improves the scalability of creating child spans by avoiding to attempt to remove them from the registry on Finish(). This attempt was useless because we knew that the child span was not part of the registry. It was also expensive, since it meant locking (and thus synchronizing on) the registry. Thus, we now only attempt to remove roots from the registry. The scalability increase can be seen in the following: GOGC=off make bench PKG=./pkg/util/tracing BENCHES=BenchmarkSpanCreation TESTFLAGS="-cpu=1,2,4,8,16,32 -count=5" SpanCreation/detached-child=false 1.95µs ±28% 2.04µs ±24% ~ (p=0.278 n=5+5) SpanCreation/detached-child=false-2 1.32µs ±11% 1.23µs ±15% ~ (p=0.310 n=5+5) SpanCreation/detached-child=false-4 1.51µs ± 5% 0.81µs ±11% -46.49% (p=0.008 n=5+5) SpanCreation/detached-child=false-8 1.86µs ± 2% 0.89µs ± 4% -52.24% (p=0.008 n=5+5) SpanCreation/detached-child=false-16 2.08µs ± 5% 0.85µs ± 4% -59.30% (p=0.008 n=5+5) SpanCreation/detached-child=false-32 2.21µs ± 2% 0.85µs ± 2% -61.71% (p=0.008 n=5+5) --- For tactical reasons, this patch also adds protection for creating a child span in a finished parent. This is illegal, as is any use of a span after Finish(). Still, almost every other use of a span has protection against use-after-Finish that makes the respective access behave reasonably. This protection will also act as a point where I'll add an assertion, at least in tests, verifying the absence of such illegal uses. So, this patch adds similar protection to creating a child in a Finish()ed span. In particular, it makes the new would-be child span act as a root and be registered in the active spans registry. Before, new span would be added to a finished parent, and so it would essentially fall by the wayside and not make it to the registry. Release note: None
Don't allow recordings to grow infinitely. Release note: None
1303a78 to
c4d3ad0
Compare
andreimatei
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 1 of 0 LGTMs obtained (waiting on @abarganier)
Previously, rimadeodhar (Rima Deodhar) wrote…
Nit: Typo recordings
done
pkg/util/tracing/crdbspan.go, line 90 at r9 (raw file):
Previously, rimadeodhar (Rima Deodhar) wrote…
Nit: Superfluous apostrophe.
done
andreimatei
left a comment
There was a problem hiding this comment.
TFTR!
bors r+
Reviewable status:
complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @abarganier and @rimadeodhar)
|
Build succeeded: |
This comment was claiming that a child's parent pointer is not set if the parent was not recording at the time the child was started, but that hasn't been true since cockroachdb#72993. Release note: None
73748: tracing: fix stale comment r=andreimatei a=andreimatei This comment was claiming that a child's parent pointer is not set if the parent was not recording at the time the child was started, but that hasn't been true since #72993. Release note: None Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
Before this patch, the registry only contained (local) roots. A child
span was not part of the registry directly; it was part of it indirectly
if the parent was recording at the moment when the child was opened.
I believe the fact that a child of a non-recording span is not part of
the registry is not principled, just path dependency. This patch makes
child spans indirectly part of the registry even if the parent is not
recording at the moment when the child is created, by having children
always register themselves with the parent, even if the parent is not
recording.
I got rid of the maxChildrenPerSpan limit on the number of open children
that a span keep track off. This limit was fairly non-sensical (it
probably started as more sane but then was refactored into non-sense) as
it applies to open spans but not to recordings of the finished children.
The point of this limit originally was to bound the memory use of
recordings. By acting only on open children, it had no such effect.
Indeed, there's pretty much no reason to bound the number of open
children tracked by a parent, and so this patch removes the limit. In
the next commit we'll get a new limit that refers to sizes of
recordings.
This patch also improves the scalability of creating child spans by
avoiding to attempt to remove them from the registry on Finish(). This
attempt was useless because we knew that the child span was not part of
the registry. It was also expensive, since it meant locking (and thus
synchronizing on) the registry. Thus, we now only attempt to remove
roots from the registry.
The scalability increase can be seen in the following:
GOGC=off make bench PKG=./pkg/util/tracing BENCHES=BenchmarkSpanCreation TESTFLAGS="-cpu=1,2,4,8,16,32 -count=5"
SpanCreation/detached-child=false 1.95µs ±28% 2.04µs ±24% ~ (p=0.278 n=5+5)
SpanCreation/detached-child=false-2 1.32µs ±11% 1.23µs ±15% ~ (p=0.310 n=5+5)
SpanCreation/detached-child=false-4 1.51µs ± 5% 0.81µs ±11% -46.49% (p=0.008 n=5+5)
SpanCreation/detached-child=false-8 1.86µs ± 2% 0.89µs ± 4% -52.24% (p=0.008 n=5+5)
SpanCreation/detached-child=false-16 2.08µs ± 5% 0.85µs ± 4% -59.30% (p=0.008 n=5+5)
SpanCreation/detached-child=false-32 2.21µs ± 2% 0.85µs ± 2% -61.71% (p=0.008 n=5+5)
For tactical reasons, this patch also adds protection for creating a
child span in a finished parent. This is illegal, as is any use of a
span after Finish(). Still, almost every other use of a span has
protection against use-after-Finish that makes the respective access
behave reasonably. This protection will also act as a point where I'll
add an assertion, at least in tests, verifying the absence of such
illegal uses. So, this patch adds similar protection to creating a child
in a Finish()ed span. In particular, it makes the new would-be child
span act as a root and be registered in the active spans registry.
Before, new span would be added to a finished parent, and so it would
essentially fall by the wayside and not make it to the registry.