Background

Today, when the xpack.monitoring.* settings are configured in logstash.yml, Logstash performs internal collection of it's own monitoring metrics and ships them to .monitoring-logstash-*. The Monitoring UI in Kibana then reads documents from these indices to create visualizations of a user's Logstash instances and pipelines.

In the future, we're aiming to have Metricbeat externally collect Logstash metrics and ship them to .monitoring-logstash-* indices. This will mean that Metricbeat has to poll various Logstash HTTP APIs to collect said metrics.

The Request

Currently the GET _node/stats/pipelines API returns much of the information that Metricbeat will need in order to build the documents to be indexed into .monitoring-logstash-*. However, there are a few additional pieces of information that this API — for each pipeline object under the pipelines array in the response — would need to return to complete the picture:

• ephemeral_id: Ephemeral ID of pipeline
• hash: Hash of pipeline representation
• queue: Pipeline queue type + metrics
• reloads: Pipeline reload metrics
• vertices: LIR vertices of the pipeline + their metrics

Here is an example JSON document showing just these additional fields:

{
  "ephemeral_id":"6d69db08-f2a5-411f-81cf-0f395d5de358",
  "hash":"4a5913a37e548b5921a3a7aec761cc0e59f0ce988c44bca8bcf14163a0cfc550",
  "queue":{
    "type":"memory",
    "queue_size_in_bytes":0,
    "max_queue_size_in_bytes":0,
    "events_count":0
  },
  "reloads":{
    "successes":0,
    "failures":0
  },
  "vertices":[
    {
      "pipeline_ephemeral_id":"6d69db08-f2a5-411f-81cf-0f395d5de358",
      "queue_push_duration_in_millis":0,
      "events_out":0,
      "id":"a6288a968df1bdcfe6433498f8bbabd15e6be53134f3594128330fd37b9ca837"
    },
    {
      "duration_in_millis":16,
      "pipeline_ephemeral_id":"6d69db08-f2a5-411f-81cf-0f395d5de358",
      "events_in":0,
      "events_out":0,
      "id":"991a78728a853623523a25457cf2932267f11f34b0c8af6310c48e44a6d69c43"
    },
    {
      "duration_in_millis":21,
      "pipeline_ephemeral_id":"6d69db08-f2a5-411f-81cf-0f395d5de358",
      "events_in":0,
      "events_out":0,
      "id":"42811067c43d0beafc02fa83221107895efc12eb37f9c4d92b94758bdf1dd3e8"
    }
  ]
}