This PR adds a new beat module to Metricbeat. It also implements the first (of two) metricsets for this module, stats.

More specifically, this PR:

• Introduces the beat module.
• Implements the stats metricset for this module.
• Implements the non-xpack code path for this metricset.
• Implements the xpack code path for this metricset.
• Introduces the modules.d/beat.yml and modules.d/beat-xpack.yml module configuration files.
• Implements integration tests for the module.
• Implements system tests for the module.

Testing this PR

X-Pack code path (for Stack Monitoring UI)

1. Build metricbeat with this PR.

   cd $GOPATH/src/github.com/elastic/beats/metricbeat
   mage build
   

2. Using Kibana > Console, clear out existing Beats monitoring data.

   DELETE .monitoring-beats-*
   

3. Start Metricbeat with internal collection.

   ./metricbeat -e -E monitoring.enabled=true
   

4. Wait 10 seconds or so for monitoring data to be collected.

5. Using Kibana > Console, retrieve the latest document from the .monitoring-beats-* index and save it.

   POST .monitoring-beats-*/_search?filter_path=hits.hits._source
   {
     "query": {
       "bool": {
         "filter": {
           "term": {
             "type": "beats_stats"
           }
         }
       }
     },
     "sort": [
       {
         "timestamp": {
           "order": "desc"
         }
       }
     ]
   }
   

6. Stop Metricbeat.

7. Using Kibana > Console, clear out existing Beats monitoring data.

   DELETE .monitoring-beats-*
   

8. Start Metricbeat with HTTP API enabled for external collection.

   ./metricbeat -e -E http.enabled=true
   

9. In a new window enable the beats-xpack Metricbeat module for external collection and start this Metricbeat instance.

   ./metricbeat modules enable beat-xpack
   ./metricbeat -e
   

10. Wait 10 seconds or so for monitoring data to be collected.

11. Using Kibana > Console, retrieve the latest document from the .monitoring-beats-* index and save it.

    POST .monitoring-beats-*/_search?filter_path=hits.hits._source
    {
      "query": {
        "bool": {
          "filter": {
            "term": {
              "type": "beats_stats"
            }
          }
        }
      },
      "sort": [
        {
          "timestamp": {
            "order": "desc"
          }
        }
      ]
    }
    

12. Compare the JSON document saved from internal collection (step 5) with the JSON document saved from external collection (step 11). You might want to use a tool like jsondiff.com for this purpose.

    The only structural difference between the two documents should be that the externally-collected JSON document should have 7 extra top-level fields: @timestamp, event, ecs, agent, metricset, service, host. These are expected extra fields that are always inserted by Metricbeat. Other than these expected structural differences, the rest of the structure should be identical; however, values of corresponding fields may be different.