Credits and notes: The below has mostly been written by @gotjosh. He's currently on vacation. That's why I (@beorn7) am filing this now. I helped with the draft, and we also got input from @davkal (Grafana), @slrtbtfs (LSP implementation for PromQL) and @jkohen and @StevenYCChou (Stackdriver Prometheus sidecar). Note, however, that the below mostly caters for the Grafana and LSP use case, while the Stackdriver use case needed more invasive changes to provide metadata from targets that are not active anymore. This more or less historical data would also help Grafana, but is out of scope for this proposal. This proposal is a fairly lightweight (and hopefully uncontroversial) addition to serve as an intermediate solution until we get full metadata support, which realistically I only expect in Prometheus v3.

Context and motivation

At the moment of writing, Prometheus users and integrations (e.g. Grafana) make use of /api/v1/label/__name__/values as a way to expose all the metric names that Prometheus has scraped.

Prometheus currently offers a way to obtain metadata (type, help, etc) for a given metric via the /api/v1/targets/metadata endpoint. This metadata helps power features such as query hinting, where users are suggested to use rate on counters.

Often, integrations are unhappy with the current state of the API endpoint. Both Grafana and the PromQL LSP would like to obtain metadata in a situation where the target is not relevant or even unknown – in which case the endpoint has to be queried for all targets, resulting in a possibly very large payload, blown up by a factor equal to the number of targets exposing the requested metric name (possibly thousands). Without the metadata, integrations have to fall back to heuristics to determine it e.g. detect monotonicity to infer it’s a counter.

Challenges

/api/v1/targets/metadata is per target

The current metadata API responds with the information per target. Certainly, the current API was implemented with a lot of discussion and scrutiny applied to it. In fact, it can be seen used as part of Stackdriver’s Prometheus sidecar with the target information.

However, Grafana and the PromQL LSP tend to follow different semantics than what the endpoint was originally conceived for, by querying metadata globally (independent of the target) as opposed to per target. As described above, querying the data for all targets creates a huge blow-up of redundant data. The client then has to collapse the data between all the targets, where most of the metadata is the same, but occasionally, the metadata becomes ambiguous when clients try to collapse the information from different targets e.g. a metric having a type of “gauge” in one instance and “counter” in another, or (more commonly) the help strings differ.

The previous examples serve as an indication that there are different use cases the work proposed could serve, namely Grafana tool tips and the PromQL LSP implementation.

/api/v1/labels/{name}/values responds with primitives instead of objects

A possible way to serve metadata for metric names would be to enrich the response of the label values endpoint with the metadata but the fact that it currently only returns primitives (strings) in the response would make it a breaking change of the current stable API. Also, Prometheus currently doesn’t persist metadata, so we could only serve the metadata for metrics still exposed by active targets, while the label values endpoint takes into account all data in the TSDB.

Proposal

We propose the introduction of two new endpoints to obtain metadata by series name as opposed to targets. Safe to say, that we’re not proposing the introduction of any additional in-RAM data structures. The response to these endpoints can be assembled by sweeping the existing metadata cache in the scrape-pool. (Essentially, we are moving the collapsing of the redundant metadata from the client into the Prometheus server, thereby saving marshalling and unmarshalling of the blown-up data and its transfer over the wire. Even for the Prometheus server, it will probably be a net cost-saving compared to calling the existing targets/metadata endpoint for all targets.)

The output we propose is best illustrated with examples:

For the envelope, we still follow the same Prometheus format. The objects embedded within data follow this spec:

api/v1/metadata

curl -G http://localhost:9090/api/v1/metadata --data-urlencode ‘limit=10’ | jq
{
  “status”: “success”,
  “data”: [
    {
      “promhttp_metric_handler_requests_in_flight”: [{
        “type”: ”gauge”,
        “help”: ”Current number of scrapes being served.”,
        “unit”: ”,”
      }],
    },
    {
      “go_goroutines”: [
        {
          “type”: ”gauge”,
          “help”: ”Number of bytes used for mcache structures obtained from system.”,
          “unit”: ””,
        },
        {
          “type”: ”gauge”,
          “help”: ”A great number of of bytes used for mcache structures obtained from system.”,
          “unit”: ””,
        }
      ],
    },
    {
      “go_memstats_mcache_sys_bytes”: [{
        “type”: ”gauge”,
        “help”: ”Number of bytes used for mcache structures obtained from system.”,
        “unit”: ””,
      }]
    }
  ]
}

api/v1/metadata/{name}

curl -G http://localhost:9090/api/v1/metadata/go_goroutines |  jq
{
  “status”: “success”,
  “data”: {
    “go_goroutines”: [
      {
        “type”: ”gauge”,
        “help”: ”Number of bytes used for mcache structures obtained from system.”,
        “unit”: ””,
      },
      {
        “type”: ”gauge”,
        “help”: ”A great number of of bytes used for mcache structures obtained from system.”,
        “unit”: ””,
      }
    ]
  }
}

We believe that this proposed format is better than the alternative of fetching metadata for all targets from the existing API and let the client collapse it.

In the response, if different targets provide different versions of metadata for the same metric name, they will be listed as additional objects inside the array of metadata. If the metadata is the same for a given metric name across all targets, the array will only have a single element.

We believe this to be sufficient for the use-case(s) described above. The exposure of the data in this format will make it feasible for integrations to consume all metrics when a target is not known (or relevant) and not have the payload linearly increase with each target added.

At this point, we would love to hear more opinions on it.