Skip to content

Ignored transactions and consistent probability sampling #3307

@PeterF778

Description

@PeterF778

Let's consider the following hypothetical example.

An OpenTelemetry user deploys a distributed application with tiers A, B, and C. Customer requests hit tier A, which makes calls to tier B, and tier B makes calls to C. The OTel user wants to trace some of the transactions only. Other transactions are to be ignored (for whatever reasons). The distinction between ignored and traced transaction is made by a custom sampler at tier A, based on the URL of the incoming requests.
When tiers B and C use ParentBasedSamplers with the default configuration, things work as expected, but the user may have performance issues because the sampling decisions made by tier A propagate to tiers B and C without any adjustment (fan out problem).

Employing Consistent Probability Samplers may fix the performance issue because sampling rates at tiers B and C can now be different. However, now calls to tier B and C may be traced even if they belong to transactions which the user wanted to ignore. This is because Consistent Probability Samplers ignore the sampled flag received from the parent. This behavior is not what the OTel user may want. For the ignored transactions the traces will always be incomplete, missing the root span.

But is it a valid use case?

If it is, a possible fix could be removing the requirement that a Consistent Probability Sampler always generates a new r-value for non-root spans (think tiers B and C), if it is missing. It should do it only when the parent's sampled flag is set. If the flag is false and there is no r-value, it should simply decide not to sample and leave the TraceState as is.

Metadata

Metadata

Assignees

No one assigned

    Labels

    spec:traceRelated to the specification/trace directorytriage:deciding:community-feedbackOpen to community discussion. If the community can provide sufficient reasoning, it may be accepted

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions