Skip to content

tracing: expand the active spans registry #72993

Merged
craig[bot] merged 3 commits intocockroachdb:masterfrom
andreimatei:tracing.registry-contention
Dec 8, 2021
Merged

tracing: expand the active spans registry #72993
craig[bot] merged 3 commits intocockroachdb:masterfrom
andreimatei:tracing.registry-contention

Conversation

@andreimatei
Copy link
Copy Markdown
Contributor

@andreimatei andreimatei commented Nov 19, 2021

Before this patch, the registry only contained (local) roots. A child
span was not part of the registry directly; it was part of it indirectly
if the parent was recording at the moment when the child was opened.

I believe the fact that a child of a non-recording span is not part of
the registry is not principled, just path dependency. This patch makes
child spans indirectly part of the registry even if the parent is not
recording at the moment when the child is created, by having children
always register themselves with the parent, even if the parent is not
recording.

I got rid of the maxChildrenPerSpan limit on the number of open children
that a span keep track off. This limit was fairly non-sensical (it
probably started as more sane but then was refactored into non-sense) as
it applies to open spans but not to recordings of the finished children.
The point of this limit originally was to bound the memory use of
recordings. By acting only on open children, it had no such effect.
Indeed, there's pretty much no reason to bound the number of open
children tracked by a parent, and so this patch removes the limit. In
the next commit we'll get a new limit that refers to sizes of
recordings.


This patch also improves the scalability of creating child spans by
avoiding to attempt to remove them from the registry on Finish(). This
attempt was useless because we knew that the child span was not part of
the registry. It was also expensive, since it meant locking (and thus
synchronizing on) the registry. Thus, we now only attempt to remove
roots from the registry.

The scalability increase can be seen in the following:

GOGC=off make bench PKG=./pkg/util/tracing BENCHES=BenchmarkSpanCreation TESTFLAGS="-cpu=1,2,4,8,16,32 -count=5"

SpanCreation/detached-child=false 1.95µs ±28% 2.04µs ±24% ~ (p=0.278 n=5+5)
SpanCreation/detached-child=false-2 1.32µs ±11% 1.23µs ±15% ~ (p=0.310 n=5+5)
SpanCreation/detached-child=false-4 1.51µs ± 5% 0.81µs ±11% -46.49% (p=0.008 n=5+5)
SpanCreation/detached-child=false-8 1.86µs ± 2% 0.89µs ± 4% -52.24% (p=0.008 n=5+5)
SpanCreation/detached-child=false-16 2.08µs ± 5% 0.85µs ± 4% -59.30% (p=0.008 n=5+5)
SpanCreation/detached-child=false-32 2.21µs ± 2% 0.85µs ± 2% -61.71% (p=0.008 n=5+5)


For tactical reasons, this patch also adds protection for creating a
child span in a finished parent. This is illegal, as is any use of a
span after Finish(). Still, almost every other use of a span has
protection against use-after-Finish that makes the respective access
behave reasonably. This protection will also act as a point where I'll
add an assertion, at least in tests, verifying the absence of such
illegal uses. So, this patch adds similar protection to creating a child
in a Finish()ed span. In particular, it makes the new would-be child
span act as a root and be registered in the active spans registry.
Before, new span would be added to a finished parent, and so it would
essentially fall by the wayside and not make it to the registry.

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@andreimatei andreimatei force-pushed the tracing.registry-contention branch 4 times, most recently from c5e95c9 to 71c84a0 Compare November 21, 2021 23:36
@andreimatei andreimatei force-pushed the tracing.registry-contention branch 2 times, most recently from 1815e4f to 6627c55 Compare December 2, 2021 18:41
@andreimatei andreimatei marked this pull request as ready for review December 2, 2021 18:44
@andreimatei
Copy link
Copy Markdown
Contributor Author

First commit is #73383

@andreimatei andreimatei force-pushed the tracing.registry-contention branch from 6627c55 to 7ab3eb4 Compare December 2, 2021 22:27
@andreimatei
Copy link
Copy Markdown
Contributor Author

First commit is #73383

rebased; all commits are original now.

Copy link
Copy Markdown
Collaborator

@rimadeodhar rimadeodhar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 1 files at r2, 5 of 5 files at r8, 5 of 5 files at r9, 3 of 3 files at r10, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @abarganier and @andreimatei)


-- commits, line 74 at r10:
Nit: Typo recordings


pkg/util/tracing/crdbspan.go, line 90 at r9 (raw file):

	// If this parent is recording at the time when a child Finish()es, and the
	// respective childRef indicates that the child is to be included in the
	// paren't recording, then the child's recording is collected in

Nit: Superfluous apostrophe.

Benchmark the creation of both regular children and detached children.
Their scalability is currently different, because the detached children
are added to the registry.

Release note: None
Before this patch, the registry only contained (local) roots. A child
span was not part of the registry directly; it was part of it indirectly
if the parent was recording at the moment when the child was opened.

I believe the fact that a child of a non-recording span is not part of
the registry is not principled, just path dependency. This patch makes
child spans indirectly part of the registry even if the parent is not
recording at the moment when the child is created, by having children
always register themselves with the parent, even if the parent is not
recording.

I got rid of the maxChildrenPerSpan limit on the number of open children
that a span keep track off. This limit was fairly non-sensical (it
probably started as more sane but then was refactored into non-sense) as
it applies to open spans but not to recordings of the finished children.
The point of this limit originally was to bound the memory use of
recordings. By acting only on open children, it had no such effect.
Indeed, there's pretty much no reason to bound the number of open
children tracked by a parent, and so this patch removes the limit. In
the next commit we'll get a new limit that refers to sizes of
recordings.

---

This patch also improves the scalability of creating child spans by
avoiding to attempt to remove them from the registry on Finish(). This
attempt was useless because we knew that the child span was not part of
the registry. It was also expensive, since it meant locking (and thus
synchronizing on) the registry. Thus, we now only attempt to remove
roots from the registry.

The scalability increase can be seen in the following:

GOGC=off make bench PKG=./pkg/util/tracing BENCHES=BenchmarkSpanCreation TESTFLAGS="-cpu=1,2,4,8,16,32 -count=5"

SpanCreation/detached-child=false     1.95µs ±28%  2.04µs ±24%     ~     (p=0.278 n=5+5)
SpanCreation/detached-child=false-2   1.32µs ±11%  1.23µs ±15%     ~     (p=0.310 n=5+5)
SpanCreation/detached-child=false-4   1.51µs ± 5%  0.81µs ±11%  -46.49%  (p=0.008 n=5+5)
SpanCreation/detached-child=false-8   1.86µs ± 2%  0.89µs ± 4%  -52.24%  (p=0.008 n=5+5)
SpanCreation/detached-child=false-16  2.08µs ± 5%  0.85µs ± 4%  -59.30%  (p=0.008 n=5+5)
SpanCreation/detached-child=false-32  2.21µs ± 2%  0.85µs ± 2%  -61.71%  (p=0.008 n=5+5)

---

For tactical reasons, this patch also adds protection for creating a
child span in a finished parent. This is illegal, as is any use of a
span after Finish(). Still, almost every other use of a span has
protection against use-after-Finish that makes the respective access
behave reasonably.  This protection will also act as a point where I'll
add an assertion, at least in tests, verifying the absence of such
illegal uses. So, this patch adds similar protection to creating a child
in a Finish()ed span.  In particular, it makes the new would-be child
span act as a root and be registered in the active spans registry.
Before, new span would be added to a finished parent, and so it would
essentially fall by the wayside and not make it to the registry.

Release note: None
Don't allow recordings to grow infinitely.

Release note: None
@andreimatei andreimatei force-pushed the tracing.registry-contention branch from 1303a78 to c4d3ad0 Compare December 7, 2021 22:41
Copy link
Copy Markdown
Contributor Author

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @abarganier)


-- commits, line 74 at r10:

Previously, rimadeodhar (Rima Deodhar) wrote…

Nit: Typo recordings

done


pkg/util/tracing/crdbspan.go, line 90 at r9 (raw file):

Previously, rimadeodhar (Rima Deodhar) wrote…

Nit: Superfluous apostrophe.

done

Copy link
Copy Markdown
Contributor Author

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFTR!

bors r+

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @abarganier and @rimadeodhar)

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Dec 8, 2021

Build succeeded:

@craig craig bot merged commit e68d1df into cockroachdb:master Dec 8, 2021
andreimatei added a commit to andreimatei/cockroach that referenced this pull request Dec 13, 2021
This comment was claiming that a child's parent pointer is not set if
the parent was not recording at the time the child was started, but that
hasn't been true since cockroachdb#72993.

Release note: None
craig bot pushed a commit that referenced this pull request Jan 4, 2022
73748: tracing: fix stale comment r=andreimatei a=andreimatei

This comment was claiming that a child's parent pointer is not set if
the parent was not recording at the time the child was started, but that
hasn't been true since #72993.

Release note: None

Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
@andreimatei andreimatei deleted the tracing.registry-contention branch January 22, 2022 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants