Skip to content

datadog tracer samples heathchecks #5504

@cgilmour

Description

@cgilmour

Description:
Reported directly by a user, the datadog tracer is sampling HTTP requests that are for healthchecks.
This shouldn't occur when using the envoy.health_check http filter, because it sets sampling to false.

The datadog tracer ignores that call, because the headers for tracing have already been "injected" at an earlier step, and become immutable.

Currently, envoy creates the span for tracing, then immediately injects the headers.
This doesn't give an opportunity for extensions to change some of the tracing state before the injection occurs.

A simple fix for this specific issue is to delay the injection until after the healthcheck filter has been applied.
(Right after this call to decodeHeaders.)
However, ideally the injection would occur after all extension processing is done for a request. I'm not sure the precise point in the code for that.

I can submit a PR for the simple fix.

Repro steps:
Configure envoy with datadog tracing and the health_check filter. Send a request to the healthcheck endpoint. It is sampled.

Config:

          http_filters:
          - name: envoy.health_check
            config:
              pass_through_mode: false
              headers:
              - name: ":path"
                exact_match: "/healthcheck"
...
tracing:
  http:
    name: envoy.tracers.datadog
    config:
      collector_cluster: datadog_agent
      service_name: envoy-service1

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions