-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Description
Description:
Reported directly by a user, the datadog tracer is sampling HTTP requests that are for healthchecks.
This shouldn't occur when using the envoy.health_check http filter, because it sets sampling to false.
The datadog tracer ignores that call, because the headers for tracing have already been "injected" at an earlier step, and become immutable.
Currently, envoy creates the span for tracing, then immediately injects the headers.
This doesn't give an opportunity for extensions to change some of the tracing state before the injection occurs.
A simple fix for this specific issue is to delay the injection until after the healthcheck filter has been applied.
(Right after this call to decodeHeaders.)
However, ideally the injection would occur after all extension processing is done for a request. I'm not sure the precise point in the code for that.
I can submit a PR for the simple fix.
Repro steps:
Configure envoy with datadog tracing and the health_check filter. Send a request to the healthcheck endpoint. It is sampled.
Config:
http_filters:
- name: envoy.health_check
config:
pass_through_mode: false
headers:
- name: ":path"
exact_match: "/healthcheck"
...
tracing:
http:
name: envoy.tracers.datadog
config:
collector_cluster: datadog_agent
service_name: envoy-service1