Description:

The OpenTelemetry project defines the concept of a Sampler API. A sampler allows controlling the spans you send to your observability backend, resulting in many benefits, such as lowering costs.

A sampler is associated with a TracerProvider in OpenTelemetry and there's a few built-in samplers.

In Envoy, we have the OpenTelemetry tracer, which can be seen as similar to the TracerProvider in OTel. The OTel tracer in Envoy is not flexible in regards to sampling. It only "forwards" the tracing decision that is passed down from Envoy via the Tracing::Decision tracing_decision. From my understanding that's a "custom" thing, that is also based on an "request id" https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/request_id/uuid/v3/uuid.proto#envoy-v3-api-field-extensions-request-id-uuid-v3-uuidrequestidconfig-use-request-id-for-trace-sampling. Meaning, this is not really "compliant" with what the OpenTelemetry project defines.

Users of OTel might want to control sampling in different ways, following the samplers that the OTel project defines. Moreover, observability vendors might also want to offer their own samplers which aligns with the decisions done by monitoring Agents for example, resulting in a higher chance of having complete traces, and be able to achieve things such as extracting metrics from spans.

By having the flexibility to configure a custom sampler in the OpenTelemetry tracer in Envoy, users have then full control on the volume of data it produces.

Feature request

1. Introduce a new interface inside the opentelemetry folder for a sampler. The interface would follow the Sampler interface as defined in the OpenTelemetry specification. It would leverage the Envoy extension framework.

2. When startSpan is called, the OpenTelemetry tracer in Envoy would then check if it was configured with a custom sampler. If it was, it defers the sampling decision to it, otherwise it falls back to the current behavior of using the Tracing::Decision tracing_decision.

3. This then allows anyone (Cloud vendors, Observability back-ends etc) to define their own samplers in a new folder tracers/opentelemetry/samplers

4. Allow the sampler to be configured by the user: Something like this:

tracing:
  provider:
    name: envoy.tracers.opentelemetry
    typed_config:
      "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
      sampler:
        name: envoy.opentelemetry.samplers.ParentBasedSampler

I believe the addition of this interface will benefit everyone and will open up for a lot of great opportunities in the observability space for Envoy.