Suppose you have a span hierarchy of ({tracerName}:{spanKind}:{spanName}):
tracerA:server:parent -> tracerB:internal:child -> tracerC:client:grandchild
And you decide that internal spans from tracerB are too noisy. You want to turn them off while:
- Not losing downstream spans. You want to keep
tracerC:client:grandchild because it contains important information about calls to downstream systems.
- Not producing broken traces.
tracerC:client:grandchild's parent should be tracerA:server:parent, not tracerB:internal:child.
How can you accomplish this? Well, you pretty much can't. A couple of problems prevent it:
- Samplers dont have access to scope, so there's no SDK configuration mechanism which can do this.
- Even if Samplers could make decisions based on scope, the sampling decision options don't make the desired behavior easy.
- If a sampler returns
DROP for tracerB and delegates to ParentBased{root=AlwaysOn} for other spans: Then all descendants of tracerB:internal:child will be dropped as well. Not what we want.
- If a sampler returns
DROP for tracerB and delegates to ParentBased{root=AlwaysOn,localParentNotSampled=AlwaysOn} for other spans: Then we end up with a broken trace since tracerC:client:grandchild will assign its parent to be tracerB:internal:child, which is dropped. Not what we want.
- If a sampler returns
DROP for tracerB and delegates to ParentBased{root=AlwaysOn,localParentNotSampled=AlwaysOn} and instrumentation is aware and only sets spans in context if Span.isRecording()=true: Then we achieve the desired affect. tracerB:internal:child is dropped and tracerC:client:grandchild will assign its parent to be tracerA:server:parent, so we have a complete trace.
- Even if Samplers could make decisions based on scope, its cumbersome to configure a sampler to have special rules for a scope. It's simpler to think of a sampling policy as orthogonal to the decision of which scopes / instrumentation libraries should be enabled.
So there is no way to get the behavior we want today. And even if samplers were given access to scope, we'd still need: a complex / non-standard sampling policy and coordination with instrumentation (i.e. checking Span.isRecording()==true). Ouch.
This is related to #530 and #1588, but the problem framing here is a little different so I think it warrants a separate issue.
Originally opened as open-telemetry/opentelemetry-java#6197 but think we should provide a spec level solution for this.
Suppose you have a span hierarchy of ({tracerName}:{spanKind}:{spanName}):
And you decide that internal spans from
tracerBare too noisy. You want to turn them off while:tracerC:client:grandchildbecause it contains important information about calls to downstream systems.tracerC:client:grandchild's parent should betracerA:server:parent, nottracerB:internal:child.How can you accomplish this? Well, you pretty much can't. A couple of problems prevent it:
DROPfortracerBand delegates toParentBased{root=AlwaysOn}for other spans: Then all descendants oftracerB:internal:childwill be dropped as well. Not what we want.DROPfortracerBand delegates toParentBased{root=AlwaysOn,localParentNotSampled=AlwaysOn}for other spans: Then we end up with a broken trace sincetracerC:client:grandchildwill assign its parent to betracerB:internal:child, which is dropped. Not what we want.DROPfortracerBand delegates toParentBased{root=AlwaysOn,localParentNotSampled=AlwaysOn}and instrumentation is aware and only sets spans in context ifSpan.isRecording()=true: Then we achieve the desired affect.tracerB:internal:childis dropped andtracerC:client:grandchildwill assign its parent to betracerA:server:parent, so we have a complete trace.So there is no way to get the behavior we want today. And even if samplers were given access to scope, we'd still need: a complex / non-standard sampling policy and coordination with instrumentation (i.e. checking
Span.isRecording()==true). Ouch.This is related to #530 and #1588, but the problem framing here is a little different so I think it warrants a separate issue.
Originally opened as open-telemetry/opentelemetry-java#6197 but think we should provide a spec level solution for this.