tracing: fix lazytag export for Jaegar/otel#85419
tracing: fix lazytag export for Jaegar/otel#85419craig[bot] merged 1 commit intocockroachdb:masterfrom
Conversation
andreimatei
left a comment
There was a problem hiding this comment.
Good stuff?
Can you please add a test? See if TestOtelTracer can be tweaked to do it.
Did you happen to figure out why the rendering of the lock time in the diagnostics bundle was off?
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @aadityasondhi and @andreimatei)
pkg/util/tracing/span_inner.go line 107 at r1 (raw file):
if s.otelSpan != nil { // It is ok to access s.crdb after calling s.crdb.finish() since it will be cleaned up by the parent of this func
Scratch this comment. Replace it with // Serialize the lazy tags.
If it stays, it needs a period, 80-col wrapping (ask me about an golang plugin if you don't have it) and it needs to be clear who the "it" is in "it will be cleaned up".
pkg/util/tracing/span_inner.go line 110 at r1 (raw file):
s.crdb.mu.Lock() defer s.crdb.mu.Unlock() for _, kv := range s.crdb.mu.lazyTags {
let's make this a method on s.crdb that returns []atribute.KeyValue. I think you've copied this code from somewhere, right? It'd probably be good to share.
d35cae2 to
f09b3ef
Compare
aadityasondhi
left a comment
There was a problem hiding this comment.
Added tests.
Re: lock wait time rendering, did some digging and it turns out when it is off for the diagnostics bundle, it is also off in the otel export to Jaeger. Interestingly, the same information is rendered correctly in the logs attached to these traces. I am still trying to narrow down where/how the wait time gets corrupted for lazy tags. My initial plan was to try and useTestContentionEventTracer to consistently reproduce this, but haven't had any luck yet. Would you have any ideas on what to try or where to look?
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @andreimatei)
pkg/util/tracing/span_inner.go line 107 at r1 (raw file):
Previously, andreimatei (Andrei Matei) wrote…
Scratch this comment. Replace it with
// Serialize the lazy tags.If it stays, it needs a period, 80-col wrapping (ask me about an golang plugin if you don't have it) and it needs to be clear who the "it" is in "it will be cleaned up".
Replaced with the suggested comment
pkg/util/tracing/span_inner.go line 110 at r1 (raw file):
Previously, andreimatei (Andrei Matei) wrote…
let's make this a method on
s.crdbthat returns[]atribute.KeyValue. I think you've copied this code from somewhere, right? It'd probably be good to share.
Done. Even though the switch structure is similar to the code in getRecordingNoChildrenLocked, it does slightly different things. Here we want to generate a flattened []atribute.KeyValue while there we are generating tag groups, and directly adding tags to the span.
andreimatei
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 1 of 0 LGTMs obtained (waiting on @aadityasondhi and @andreimatei)
pkg/util/tracing/crdbspan.go line 973 at r2 (raw file):
// It flattens lazy tags with children and attaches the parent's // key as a prefix on each child. func (s *crdbSpan) getLazyTagsKvLocked() []attribute.KeyValue {
KV in the function name - we generally capitalize all acronyms
pkg/util/tracing/span_inner.go line 110 at r1 (raw file):
Previously, aadityasondhi (Aaditya Sondhi) wrote…
Done. Even though the switch structure is similar to the code in
getRecordingNoChildrenLocked, it does slightly different things. Here we want to generate a flattened[]atribute.KeyValuewhile there we are generating tag groups, and directly adding tags to the span.
Consider making the method return []tracingpb.TagGroup, and then processing the tag groups here to turn them into what otel wants.
The part I'd like shared with getRecordingNoChildrenLocked is recognizing Stringers vs LaxyTags.
f09b3ef to
f74a3a7
Compare
aadityasondhi
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @andreimatei)
pkg/util/tracing/crdbspan.go line 973 at r2 (raw file):
Previously, andreimatei (Andrei Matei) wrote…
KVin the function name - we generally capitalize all acronyms
Renamed the function as a part of the latest change.
pkg/util/tracing/span_inner.go line 110 at r1 (raw file):
Previously, andreimatei (Andrei Matei) wrote…
Consider making the method return
[]tracingpb.TagGroup, and then processing the tag groups here to turn them into what otel wants.
The part I'd like shared withgetRecordingNoChildrenLockedis recognizingStringersvsLaxyTags.
Done.
f74a3a7 to
bb40949
Compare
Previously, lazytags would not render in the otel export to Jaeger. This was because otel doesn't support an on-demand nature of the lazytags in its interface. This change manually adds lazytags as otel tags before an otel span is finished, allowing them to render in Jaeger after they are exported. Resolves: cockroachdb#85166 Release note: None Release justification: bug fixes
bb40949 to
1b2d4b1
Compare
|
bors r+ |
|
Build succeeded: |
Previously, lazytags would not render in the otel export to
Jaeger. This was because otel doesn't support an on-demand
nature of the lazytags in its interface.
This change manually adds lazytags as otel tags before an otel
span is finished, allowing them to render in Jaeger after they
are exported.
Resolves: #85166
Release note: None
Release justification: bug fixes