Description
The Prometheus metrics endpoint does not set a Content-Type header when returning metrics. This causes the DataDog Agent ("DD Agent") we use to error out. Unfortunately we have no way of forcing the DD Agent to ignore http headers. The Geth metrics endpoint, which does set a Content-Type header, works fine.
Version
Latest, cookie cutter binary.
Present Behaviour
Error in DD Agent:
Instance ID: (......) [ERROR]
Configuration Source: file:/etc/datadog-agent/conf.d/prometheus.d/conf.yaml
Total Runs: 1
Metric Samples: Last Run: 0, Total: 0
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 1, Total: 1
Average Execution Time : 97ms
Last Execution Date : 2022-08-08 15:20:23 UTC (1659972023000)
Last Successful Execution Date : Never
Error: 'content-type'
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 1120, in run
self.check(instance)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/prometheus/base_check.py", line 106, in check
scraper.process(
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/prometheus/mixins.py", line 408, in process
for metric in self.scrape_metrics(endpoint, instance=instance):
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/prometheus/mixins.py", line 380, in scrape_metrics
for metric in self.parse_metric_family(response):
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/prometheus/mixins.py", line 190, in parse_metric_family
if 'application/vnd.google.protobuf' in response.headers['Content-Type']:
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/structures.py", line 54, in __getitem__
return self._store[key.lower()][1]
KeyError: 'content-type'
No header set as per curl verbose output:
:~$ curl -v http://localhost:5054/metrics > lighthouse.txt
* Trying localhost:5054...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (...) port 5054 (#0)
> GET /metrics HTTP/1.1
> Host: localhost:5054
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: Lighthouse/v2.5.1-df51a73/x86_64-linux
< content-length: 159978
< date: Mon, 08 Aug 2022 15:32:20 GMT
<
{ [65408 bytes data]
100 156k 100 156k 0 0 1679k 0 --:--:-- --:--:-- --:--:-- 1679k
* Connection #0 to host localhost left intact
Expected Behaviour
The Metrics endpoint/webserver should set a Content-Type: text/plain header and follow decent W3 specs.
Steps to resolve
The Metrics endpoint/webserver should set a Content-Type: text/plain header and follow decent W3 specs.
Description
The Prometheus metrics endpoint does not set a
Content-Typeheader when returning metrics. This causes the DataDog Agent ("DD Agent") we use to error out. Unfortunately we have no way of forcing the DD Agent to ignore http headers. The Geth metrics endpoint, which does set aContent-Typeheader, works fine.Version
Latest, cookie cutter binary.
Present Behaviour
Error in DD Agent:
No header set as per
curlverbose output:Expected Behaviour
The Metrics endpoint/webserver should set a
Content-Type: text/plainheader and follow decent W3 specs.Steps to resolve
The Metrics endpoint/webserver should set a
Content-Type: text/plainheader and follow decent W3 specs.