What does this PR do?

This PR introduces a new scope setting for the elasticsearch Metricbeat module. This setting can take one of two values:

• node (default): indicates that each item in the hosts list points to a distinct Elasticsearch node in a cluster, or
• cluster: indicates that each item in the hosts lists points to a single endpoint for a distinct Elasticsearch cluster (e.g. a load-balancing proxy fronting the cluster).

Why is it important?

Sometimes it may not be possible for Metricbeat to reach individual Elasticsearch nodes. It might only have access to a single endpoint that fronts the entire Elasticsearch cluster.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Manual testing

Testing the new scope: cluster functionality introduced in this PR requires an Elasticsearch cluster with multiple nodes. The easiest way is probably to spin up an Elastic Cloud cluster.

1. Enable the elasticsearch-xpack module.

   ./metricbeat modules enable elasticsearch-xpack
   

2. Configure the module (./modules.d/elasticsearch-xpack.yml) like so:

   - module: elasticsearch
     xpack.enabled: true
     period: 10s
     hosts:
       - XXXXX
     scope: cluster
     username: XXXXX
     password: XXXXX
   

   Obviously, replace the XXXXX placeholders with your own.

   The key is that there should be exactly one item under hosts, pointing to single endpoint for your cluster. This could be the Elasticsearch endpoint obtained from Elastic Cloud or the address of a single node in your on-prem/local Elasticsearch cluster.

3. Configure Metricbeat to send the collected stats to an Elasticsearch cluster. This will act as your Monitoring Cluster. It could be the same cluster you're using to collect stats from (as configured in your elasticsearch-xpack module configuration above) or it could be an entirely separate cluster.

   metricbeat.config.modules:
   path: ${path.config}/modules.d/*.yml
   reload.enabled: false
   
   output.elasticsearch:
     enabled: true
     hosts:
       - XXXXXX
     username: XXXXXX
     password: XXXXXX
   

4. Start Metricbeat.

   ./metricbeat -e
   

5. Let Metricbeat run for ~30 seconds. Make sure there are no errors in the Metricbeat logs.

6. Perform the following query against your Monitoring Cluster.

   POST .monitoring-es-7-mb-*/_search
   {
     "size": 0,
     "aggs": {
       "by_cluster_uuid": {
         "terms": {
           "field": "cluster_uuid"
         }
       },
       "cluster_stats": {
         "filter": {
           "term": {
             "type": "cluster_stats"
           }
         },
         "aggs": {
           "by_period": {
             "date_histogram": {
               "field": "timestamp",
               "interval": "10s"
             }
           }
         }
       },
       "node_stats": {
         "filter": {
           "term": {
             "type": "node_stats"
           }
         },
         "aggs": {
           "by_period": {
             "date_histogram": {
               "field": "timestamp",
               "interval": "10s"
             }
           }
         }
       }
     }
   }
   

   This query checks 3 things:

   1. The aggs.by_cluster_uuid aggregation checks that we are only seeing data for a single cluster and that all documents contain that single cluster UUID.
      • Verify that aggregations.by_cluster_uuid.buckets only contains a single bucket, for the single cluster UUID of the Elasticsearch cluster you are monitoring.
      • Verify that aggregations.by_cluster_uuid.buckets[0].doc_count is the same as the hits.total.value.
   2. The aggs.cluster_stats aggregation checks that type: cluster_stats documents are only indexed once every collection period (10 seconds) and that there is at most one document per collection period. We are using type: cluster_stats here as an example; the same should be true for any types other than type: node_stats.
      • Verify that aggregations.cluster_stats.by_period.buckets have several buckets, each corresponding to a time period. Each buckets should be 10 seconds "wide". Within each bucket, verify that doc_count is <= 1.
   3. The aggs.node_stats aggregation checks that type: node_stats documents are only indexed once every collection period (10 seconds) and that there are at most N documents per collection period, where N is the number of nodes in the cluster you are monitoring.
      • Verify that aggregations.node_stats.by_period.buckets have several buckets, each corresponding to a time period. Each buckets should be 10 seconds "wide". Within each bucket, verify that doc_count is <= N, where N is the number of nodes in the cluster you are monitoring.

Related issues

• Resolves Metricbeat elasticsearch module should be able to collect from a single cluster endpoint #18539.