Background

Today, when the xpack.monitoring.* settings are configured in logstash.yml, Logstash performs internal collection of it's own monitoring metrics and ships them to .monitoring-logstash-*. Note that Logstash ships these metrics to the production Elasticsearch cluster, which then enriches them with it's cluster_uuid and forwards them on to the monitoring Elasticsearch cluster for indexing. The Monitoring UI in Kibana then reads documents from these indices to create visualizations of a user's Logstash instances and pipelines.

In the future, we're aiming to have Metricbeat externally collect Logstash metrics and ship them to .monitoring-logstash-* indices. This will mean that Metricbeat has to poll various Logstash HTTP APIs to collect said metrics.

The Request

This request is a follow up to #10119 and #10120. Once those issues are implemented, the corresponding APIs will return almost all the information needed by Metricbeat. However, there is still one piece of information needed, which is currently being injected into Logstash monitoring data by the production Elasticsearch cluster (see bolded sentence in the Background section above): the cluster_uuid of the Elasticsearch production cluster.

Taking a step back and looking at this cluster_uuid from a conceptual perspective, it really is meant to be the cluster_uuid of any Elasticsearch cluster that a Logstash pipeline is sending its data to. That way the Stack Monitoring UI can show this Logstash pipeline (and it's node) as part of the correct stack, which is built on top of the correct Elasticsearch cluster.

Of course, it's possible that a particular Logstash pipeline is not using the Elasticsearch output at all or is using multiple Elasticsearch outputs. If a Logstash pipeline is using one or more Elasticsearch outputs, would it be possible to return these Elasticsearch clusters' cluster_uuids in the API responses, perhaps as a cluster_uuids or elasticsearch_cluster_uuids property, as a sibling to the ephemeral_id property? If a Logstash pipeline is not using an Elasticsearch output, this property could be omitted or returned as null, whatever is easier.