Let's consider the following hypothetical example.
An OpenTelemetry user deploys a distributed application with tiers A, B, and C. Customer requests hit tier A, which makes calls to tier B, and tier B makes calls to C. The OTel user wants to trace some of the transactions only. Other transactions are to be ignored (for whatever reasons). The distinction between ignored and traced transaction is made by a custom sampler at tier A, based on the URL of the incoming requests.
When tiers B and C use ParentBasedSamplers with the default configuration, things work as expected, but the user may have performance issues because the sampling decisions made by tier A propagate to tiers B and C without any adjustment (fan out problem).
Employing Consistent Probability Samplers may fix the performance issue because sampling rates at tiers B and C can now be different. However, now calls to tier B and C may be traced even if they belong to transactions which the user wanted to ignore. This is because Consistent Probability Samplers ignore the sampled flag received from the parent. This behavior is not what the OTel user may want. For the ignored transactions the traces will always be incomplete, missing the root span.
But is it a valid use case?
If it is, a possible fix could be removing the requirement that a Consistent Probability Sampler always generates a new r-value for non-root spans (think tiers B and C), if it is missing. It should do it only when the parent's sampled flag is set. If the flag is false and there is no r-value, it should simply decide not to sample and leave the TraceState as is.
Let's consider the following hypothetical example.
An OpenTelemetry user deploys a distributed application with tiers A, B, and C. Customer requests hit tier A, which makes calls to tier B, and tier B makes calls to C. The OTel user wants to trace some of the transactions only. Other transactions are to be ignored (for whatever reasons). The distinction between ignored and traced transaction is made by a custom sampler at tier A, based on the URL of the incoming requests.
When tiers B and C use ParentBasedSamplers with the default configuration, things work as expected, but the user may have performance issues because the sampling decisions made by tier A propagate to tiers B and C without any adjustment (fan out problem).
Employing Consistent Probability Samplers may fix the performance issue because sampling rates at tiers B and C can now be different. However, now calls to tier B and C may be traced even if they belong to transactions which the user wanted to ignore. This is because Consistent Probability Samplers ignore the
sampledflag received from the parent. This behavior is not what the OTel user may want. For the ignored transactions the traces will always be incomplete, missing the root span.But is it a valid use case?
If it is, a possible fix could be removing the requirement that a Consistent Probability Sampler always generates a new r-value for non-root spans (think tiers B and C), if it is missing. It should do it only when the parent's sampled flag is set. If the flag is false and there is no r-value, it should simply decide not to sample and leave the
TraceStateas is.