Skip to content

Use Elasticsearch histogram type to store Prometheus histograms #32006

@MichaelKatsoulis

Description

@MichaelKatsoulis

Allow to store Prometheus histograms types as Elasticsearch histograms using metricbeat.

For example prometheus histograms metrics like

# TYPE apiserver_request_duration_seconds histogram
apiserver_request_duration_seconds_bucket{component="",dry_run="",group="",resource="",scope="",subresource="/healthz",verb="GET",version="",le="0.05"}

are currently stored in elasticsearch as

"request": {
  "component": "apiserver",
  "duration": {
    "us": {
     "bucket": {
        "+Inf": 2,
        "100000": 0,
	"1000000": 0,
	"10000000": 0,
	"1250000": 0,
	"150000": 0,
	"1500000": 0,
	"15000000": 0,
	"9000000": 0
   },
   "count": 2,
   "sum": 1081003139.2180002
  }
},			

Instead according to elastic/kibana#59387 they could be stored as Histogram Data Type like

"request.duration.us": {
   "histogram": {
       "counts": [
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        2
        ],
        "values": [
                        100000,
                        1000000,
                        10000000,
                        1250000,
                        150000,
                        1500000,
                        15000000,
                        9000000,
                        +Inf
        ]}},

This can then help in better visualisations of those data in Kibana using the percentiles aggregation.
An implementation of this data transformation has already been done in #17061 for prometheus collector.

We should consider doing the same in prometheus helper.

We should proceed with caution as hard updating this can lead to breaking changes.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions