💚 Build Succeeded

Script used for automatically generating field aliases YAML, to be added to module/elasticsearch/_meta/fields.yml: https://gist.github.com/ycombinator/c48af6ea6f121851c30b21d3de98c6c0

Pinging @elastic/integrations-services (Team:Services)

I tried to run this locally and I'm seeing this error in the metricbeat log:

2020-07-20T11:23:43.341-0400	INFO	module/wrapper.go:259	Error fetching data for metricset elasticsearch.node_stats: 1 error: failure to apply node schema: 3 errors: key `os.cgroup` not found; key `os.cpu.load_average.5m` not found; key `os.cpu.load_average.15m` not found
2020-07-20T11:23:53.343-0400	INFO	module/wrapper.go:259	Error fetching data for metricset elasticsearch.node_stats: 1 error: failure to apply node schema: 3 errors: key `os.cpu.load_average.5m` not found; key `os.cpu.load_average.15m` not found; key `os.cgroup` not found

@chrisronline Thanks for reporting the errors. The latest two commits should take care of them.

I'm running this successfully now (no errors in the metricbeat log) in a fresh environment and I'm not seeing node_stats documents.

POST .monitoring-es-*/_search
{
  "size": 0,
  "aggs": {
    "types": {
      "terms": {
        "field": "type",
        "size": 10
      }
    }
  }
}

->

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 376,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "types" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "index_stats",
          "doc_count" : 280
        },
        {
          "key" : "cluster_stats",
          "doc_count" : 20
        },
        {
          "key" : "enrich_coordinator_stats",
          "doc_count" : 20
        },
        {
          "key" : "index_recovery",
          "doc_count" : 20
        },
        {
          "key" : "indices_stats",
          "doc_count" : 20
        },
        {
          "key" : "shards",
          "doc_count" : 16
        }
      ]
    }
  }
}

elasticsearch-xpack module:

- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - enrich
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard

@chrisronline Awesome! And, just to confirm, you're seeing the equivalent of node_stats documents in metricbeat-*, yes?

Where did we land on the type field? I don't see an alias for it. Can we alias it to metricset.name?

Also, for source_node, can we map the following two:

source_node.uuid -> elasticsearch.node.id
source_node.name -> elasticsearch.node.name

Where did we land on the type field? I don't see an alias for it. Can we alias it to metricset.name?

I'm seeing a top-level type field already in the template generated by Metricbeat. So aliasing it to metricset.name might not be possible. I'm trying to figure out where this field is coming from.

Also, I just realized we're going to run into a blocker eventually. For this POC we can alias the top-level node_stats.* fields to various elasticsearch.node.stats.* fields. This will work for Elasticsearch monitoring data.

However, once we try to handle Logstash monitoring data of type: node_stats, we'll be stuck because we won't be able to also alias any Logstash node_stats.* fields with the same names as Elasticsearch node_stats.* fields to logstash.node.stats.* fields.

@chrisronline I added the aliases you requested for type, source_node.uuid, and source_node.name. However, please also see my previous comment about a potential blocker: #19747 (comment).

However, once we try to handle Logstash monitoring data of type: node_stats, we'll be stuck because we won't be able to also alias any Logstash node_stats.* fields with the same names as Elasticsearch node_stats.* fields to logstash.node.stats.* fields.

I'm confused. Logstash only has logstash_stats and logstash_state?

I'm confused. Logstash only has logstash_stats and logstash_state?

Ah yes, I was mis-remembering 🤦. We should be good then. Sorry for causing confusion.

What about timestamp? Can we alias:

timestamp -> @timestamp

What about timestamp?

Yes, fixed now. Apparently I forgot to alias a bunch of top-level fields 🤦 🤦.

I added the aliases you requested for type, source_node.uuid, and source_node.name. However, please also see my previous comment about a potential blocker: #19747 (comment).

I'm confused about the type field. I also see it in the mappings. Does changing this affect folks when they upgrade to a version of Metricbeat containing the alias? Wouldn't that error out saying the field exists?

I'm confused about the type field. I also see it in the mappings. Does changing this affect folks when they upgrade to a version of Metricbeat containing the alias? Wouldn't that error out saying the field exists?

Yes, good point. I'm still trying to figure out where in the Beats codebase that field is coming from. It feels like a bug to me so perhaps we could have it removed, even in a minor. But let's assume we can't remove it. What would our alternatives be? How is the type field being used from Monitoring UI / Telemetry / Watcher queries?

I'm still trying to figure out where in the Beats codebase that field is coming from.

No action needed, just wanted to post an update about this. I've narrowed it down to Metricbeat:

$ ./filebeat export template | jq .mappings.properties.type
null
$ ./metricbeat export template | jq .mappings.properties.type
{
  "ignore_above": 1024,
  "type": "keyword"
}

Still figuring out exactly what in Metricbeat is injecting this top-level type key but at least we know it's something restricted to Metricbeat — I was worried it was being injected by the template management code in libbeat — if we decide it's a bug today and want to make a breaking change to remove it in a minor.

@chrisronline Thinking about our Zoom discussion a few minutes ago:

• I'm thinking of converting this PR from a POC to The Real Thing (TM) but maybe targeting a feature branch instead of master so all the metricsets' changes can land at once when they are all ready. The benefit of working in a feature branch would be that we don't inadvertently roll out changes to some metricsets that we later need to roll back because of some issue we find later. It also reduces the pressure to try and get all the changes shipped in a single minor release. WDYT about that + doing the same thing with your PRs on the UI side?

• In order to convert this PR to The Real Thing (TM) I'd need the list of fields actually being used by Stack Monitoring UI + Telemetry + Cluster Alerts. That way I can remove any fields and aliases we don't need to reduce the mapping size. Just post the list of fields here in this PR and I'll update it so you can re-test it to make sure things still work with your POC PR on the UI side.

I'd need the list of fields actually being used by Stack Monitoring UI + Telemetry + Cluster Alerts. That way I can remove any fields and aliases we don't need to reduce the mapping size.

BTW, just to stress the need to reduce the fields in the mapping to just the ones we actually need, here's a blocker we found on the Filebeat side with mappings being too large: #19965. So we'd want to try and avoid this with the Metricbeat mapping.

@ycombinator It'd be nice to have type as an alias but we can work around it. We mainly use it in terms filter in queries but we can easily change those to use bool and should across either type or metricset.name

It'd be nice to have type as an alias but we can work around it. We mainly use it in terms filter in queries but we can easily change those to use bool and should across either type or metricset.name

@chrisronline Then lets go ahead with changing the queries. Beats won't allow duplicate definitions of the same field in the mappings (makes sense!) and I don't feel comfortable redefining the existing type field to an alias as we don't know how it might be used so that could cause a breaking change.

@sayden Good catches re: data-xpack.json and data.json files! I've removed data-xpack.json in 61b1c81 as it no longer makes sense for the node_stats metricset. And I've updated data.json in d7066eb now that the metricset is collecting a lot more metrics.

Re: feature branch, yes, I still need to create it and re-target this PR to it. But first I want to get CI green on this PR.

Feature branch is created: https://github.com/elastic/beats/tree/feature-stack-monitoring-mb-ecs. Will re-target this PR to it as soon as CI goes green.

Travis CI failures look unrelated (generator failures) and Jenkins CI is green on this PR. Re-targeting to feature branch...

Didn't realize Jenkins CI would run again when I re-targeted the PR to the feature branch. I will let it run and, when it passes, merge this PR.