Prometheus does not recognize HELP and TYPE for OpenMetrics counters #13944
Description
What did you do?
I pointed Prometheus to OpenMetrics endpoint which outputs data that looks like this:
$ curl http://192.168.10.2:9001/metrics -v
* Trying 192.168.10.2:9001...
* Connected to 192.168.10.2 (192.168.10.2) port 9001 (#0)
> GET /metrics HTTP/1.1
> Host: 192.168.10.2:9001
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 561
< content-type: application/openmetrics-text;charset=utf-8;version=1.0.0
< date: Wed, 17 Apr 2024 10:56:28 GMT
<
# HELP direct_notifications Number of direct notifications.
# TYPE direct_notifications counter
direct_notifications_total 1
# HELP heartbeat_notifications Number of heartbeat notifications.
# TYPE heartbeat_notifications counter
heartbeat_notifications_total 123
# HELP heartbeat_registrations Number of heartbeat registrations.
# TYPE heartbeat_registrations counter
heartbeat_registrations_total 456
# HELP heartbeat_token_count Number of tokens registered for heartbeat notifications.
# TYPE heartbeat_token_count gauge
heartbeat_token_count 789
# EOF
* Connection #0 to host 192.168.10.2 left intact
Endpoint is provided by https://github.com/deltachat/notifiers which uses https://github.com/prometheus/client_rust
What did you expect to see?
I expected that Prometheus would display "counter" hint and help line "Number of direct notifications." However, Prometheus ignores HELP and TYPE for direct_notifications.
Apparently it expected that response would contain:
# HELP direct_notifications_total Number of direct notifications.
# TYPE direct_notifications_total counter
direct_notifications_total 1
but this is not correct according to OpenMetrics specification. This is the difference documented in https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/
What did you see instead? Under which circumstances?
I expected that OpenMetrics parser would be used according to the content-type and recognize HELP and TYPE for direct_notifications_total counter.
System information
Linux 5.15.0-102-generic x86_64
Prometheus version
prometheus, version 2.51.2 (branch: HEAD, revision: b4c0ab52c3e9b940ab803581ddae9b3d9a452337)
build user: root@b63f02a423d9
build date: 20240410-14:05:54
go version: go1.22.2
platform: linux/amd64
tags: netgo,builtinassets,stringlabels
Prometheus configuration file
scrape_configs:
...
- job_name: "notifications"
scrape_interval: 30s
static_configs:
- targets: ['192.168.10.2:9001']