Skip to content

tracing: fix lazytag export for Jaegar/otel#85419

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
aadityasondhi:aadityas/tracing-lazytags
Aug 19, 2022
Merged

tracing: fix lazytag export for Jaegar/otel#85419
craig[bot] merged 1 commit intocockroachdb:masterfrom
aadityasondhi:aadityas/tracing-lazytags

Conversation

@aadityasondhi
Copy link
Copy Markdown
Contributor

@aadityasondhi aadityasondhi commented Aug 1, 2022

Previously, lazytags would not render in the otel export to
Jaeger. This was because otel doesn't support an on-demand
nature of the lazytags in its interface.

This change manually adds lazytags as otel tags before an otel
span is finished, allowing them to render in Jaeger after they
are exported.

Resolves: #85166

Release note: None

Release justification: bug fixes

@aadityasondhi aadityasondhi requested review from a team and andreimatei August 1, 2022 18:45
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Copy link
Copy Markdown
Contributor

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good stuff?

Can you please add a test? See if TestOtelTracer can be tweaked to do it.

Did you happen to figure out why the rendering of the lock time in the diagnostics bundle was off?

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @aadityasondhi and @andreimatei)


pkg/util/tracing/span_inner.go line 107 at r1 (raw file):

	if s.otelSpan != nil {
		// It is ok to access s.crdb after calling s.crdb.finish() since it will be cleaned up by the parent of this func

Scratch this comment. Replace it with // Serialize the lazy tags.

If it stays, it needs a period, 80-col wrapping (ask me about an golang plugin if you don't have it) and it needs to be clear who the "it" is in "it will be cleaned up".


pkg/util/tracing/span_inner.go line 110 at r1 (raw file):

		s.crdb.mu.Lock()
		defer s.crdb.mu.Unlock()
		for _, kv := range s.crdb.mu.lazyTags {

let's make this a method on s.crdb that returns []atribute.KeyValue. I think you've copied this code from somewhere, right? It'd probably be good to share.

@aadityasondhi aadityasondhi force-pushed the aadityas/tracing-lazytags branch 2 times, most recently from d35cae2 to f09b3ef Compare August 8, 2022 14:54
Copy link
Copy Markdown
Contributor Author

@aadityasondhi aadityasondhi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added tests.

Re: lock wait time rendering, did some digging and it turns out when it is off for the diagnostics bundle, it is also off in the otel export to Jaeger. Interestingly, the same information is rendered correctly in the logs attached to these traces. I am still trying to narrow down where/how the wait time gets corrupted for lazy tags. My initial plan was to try and useTestContentionEventTracer to consistently reproduce this, but haven't had any luck yet. Would you have any ideas on what to try or where to look?

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei)


pkg/util/tracing/span_inner.go line 107 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

Scratch this comment. Replace it with // Serialize the lazy tags.

If it stays, it needs a period, 80-col wrapping (ask me about an golang plugin if you don't have it) and it needs to be clear who the "it" is in "it will be cleaned up".

Replaced with the suggested comment


pkg/util/tracing/span_inner.go line 110 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

let's make this a method on s.crdb that returns []atribute.KeyValue. I think you've copied this code from somewhere, right? It'd probably be good to share.

Done. Even though the switch structure is similar to the code in getRecordingNoChildrenLocked, it does slightly different things. Here we want to generate a flattened []atribute.KeyValue while there we are generating tag groups, and directly adding tags to the span.

Copy link
Copy Markdown
Contributor

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @aadityasondhi and @andreimatei)


pkg/util/tracing/crdbspan.go line 973 at r2 (raw file):

// It flattens lazy tags with children and attaches the parent's
// key as a prefix on each child.
func (s *crdbSpan) getLazyTagsKvLocked() []attribute.KeyValue {

KV in the function name - we generally capitalize all acronyms


pkg/util/tracing/span_inner.go line 110 at r1 (raw file):

Previously, aadityasondhi (Aaditya Sondhi) wrote…

Done. Even though the switch structure is similar to the code in getRecordingNoChildrenLocked, it does slightly different things. Here we want to generate a flattened []atribute.KeyValue while there we are generating tag groups, and directly adding tags to the span.

Consider making the method return []tracingpb.TagGroup, and then processing the tag groups here to turn them into what otel wants.
The part I'd like shared with getRecordingNoChildrenLocked is recognizing Stringers vs LaxyTags.

@aadityasondhi aadityasondhi force-pushed the aadityas/tracing-lazytags branch from f09b3ef to f74a3a7 Compare August 17, 2022 22:23
Copy link
Copy Markdown
Contributor Author

@aadityasondhi aadityasondhi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @andreimatei)


pkg/util/tracing/crdbspan.go line 973 at r2 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

KV in the function name - we generally capitalize all acronyms

Renamed the function as a part of the latest change.


pkg/util/tracing/span_inner.go line 110 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

Consider making the method return []tracingpb.TagGroup, and then processing the tag groups here to turn them into what otel wants.
The part I'd like shared with getRecordingNoChildrenLocked is recognizing Stringers vs LaxyTags.

Done.

@aadityasondhi aadityasondhi force-pushed the aadityas/tracing-lazytags branch from f74a3a7 to bb40949 Compare August 19, 2022 14:01
Previously, lazytags would not render in the otel export to
Jaeger. This was because otel doesn't support an on-demand
nature of the lazytags in its interface.

This change manually adds lazytags as otel tags before an otel
span is finished, allowing them to render in Jaeger after they
are exported.

Resolves: cockroachdb#85166

Release note: None

Release justification: bug fixes
@aadityasondhi aadityasondhi force-pushed the aadityas/tracing-lazytags branch from bb40949 to 1b2d4b1 Compare August 19, 2022 15:38
@aadityasondhi
Copy link
Copy Markdown
Contributor Author

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 19, 2022

Build succeeded:

@craig craig bot merged commit 677ef2c into cockroachdb:master Aug 19, 2022
@aadityasondhi aadityasondhi deleted the aadityas/tracing-lazytags branch August 19, 2022 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tracing: lazyTags don't make it to Jaeger

3 participants