Skip to content

Turn off spans from a scope while retaining downstream spans and without breaking traces #3867

@jack-berg

Description

@jack-berg

Suppose you have a span hierarchy of ({tracerName}:{spanKind}:{spanName}):

tracerA:server:parent -> tracerB:internal:child -> tracerC:client:grandchild

And you decide that internal spans from tracerB are too noisy. You want to turn them off while:

  1. Not losing downstream spans. You want to keep tracerC:client:grandchild because it contains important information about calls to downstream systems.
  2. Not producing broken traces. tracerC:client:grandchild's parent should be tracerA:server:parent, not tracerB:internal:child.

How can you accomplish this? Well, you pretty much can't. A couple of problems prevent it:

  • Samplers dont have access to scope, so there's no SDK configuration mechanism which can do this.
  • Even if Samplers could make decisions based on scope, the sampling decision options don't make the desired behavior easy.
    • If a sampler returns DROP for tracerB and delegates to ParentBased{root=AlwaysOn} for other spans: Then all descendants of tracerB:internal:child will be dropped as well. Not what we want.
    • If a sampler returns DROP for tracerB and delegates to ParentBased{root=AlwaysOn,localParentNotSampled=AlwaysOn} for other spans: Then we end up with a broken trace since tracerC:client:grandchild will assign its parent to be tracerB:internal:child, which is dropped. Not what we want.
    • If a sampler returns DROP for tracerB and delegates to ParentBased{root=AlwaysOn,localParentNotSampled=AlwaysOn} and instrumentation is aware and only sets spans in context if Span.isRecording()=true: Then we achieve the desired affect. tracerB:internal:child is dropped and tracerC:client:grandchild will assign its parent to be tracerA:server:parent, so we have a complete trace.
  • Even if Samplers could make decisions based on scope, its cumbersome to configure a sampler to have special rules for a scope. It's simpler to think of a sampling policy as orthogonal to the decision of which scopes / instrumentation libraries should be enabled.

So there is no way to get the behavior we want today. And even if samplers were given access to scope, we'd still need: a complex / non-standard sampling policy and coordination with instrumentation (i.e. checking Span.isRecording()==true). Ouch.

This is related to #530 and #1588, but the problem framing here is a little different so I think it warrants a separate issue.

Originally opened as open-telemetry/opentelemetry-java#6197 but think we should provide a spec level solution for this.

Metadata

Metadata

Assignees

Labels

[label deprecated] triaged-accepted[label deprecated] Issue triaged and accepted by OTel community, can proceed with creating a PRspec:traceRelated to the specification/trace directory

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions