[Metricbeat] Remove xpack enabled flag on ES, Logstash, Beats and Kibana#24427
Merged
[Metricbeat] Remove xpack enabled flag on ES, Logstash, Beats and Kibana#24427
Conversation
…hout xpack.enabled flag (#19747) * Make node_stats ignore xpack.enabled: true for the POC * Index more fields * WIP: add fields to fields.yml * Fleshing out alias fields * Adding more fields to mapping * Marking optional fields * Fixing datatypes * Fixing formatting * Fixing alias field definitions * More field fixes * Making cgroups metrics collection optional * Marking os.load_avg as optional for Windows * Aliasing type => metricset.name * Aliasing source_node fields * Aliasing timestamp => @timestamp * Removing field alias mappings for unused fields * Removing unnecessary fields from mapping * Updating generated files * Reducing visibility of isMaster function * Adding methods to metricset * Refactoring for testability * Fixing formatting * Making indices.bulk and thread_pool.write fields optional for BC * Deleting sample file from unsupported ES version * Remove xpack.enabled code path!!! * Updating xpack unit test * Updating python system tests for xpack * Remove top-level type field * Removing data-xpack.json * Updating data.json * Not collecting unmapped fields * Fixing formatting
…0613) * Auto-include node_stats metricset when xpack.enabled: true is set * Fixing some tests * Try to fix python system test * Fixing len check * Fixing monitoring index type for node_stats metricset * Account for node_stats docs being indexed into metricbeat-* * Debugging * More debugging * Debugging * Updating integration test * Fixing test code
…without xpack.enabled flag (#20615)
… xpack.enabled flag (#20821)
…ack.enabled flag (#21348)
… without xpack.enabled flag (#21099)
…xpack.enabled flag (#21438)
… without xpack.enabled flag (#20810)
… xpack.enabled flag (#21043)
…xpack.enabled flag (#21546)
…xpack.enabled flag (#21389)
…eature/mb/elasticsearch/beats_state_xpack_flag
…csearch/beats_state_xpack_flag
…stack-monitoring-mb-ecs # Conflicts: # metricbeat/docs/fields.asciidoc
Contributor
Author
|
/test |
1 similar comment
Contributor
Author
|
/test |
Contributor
Author
|
/test |
Contributor
|
Here are the changes I needed to make to get beats/apm working: https://gist.github.com/chrisronline/f18740f00601251f979cec3dccbac320 At a high level, the I also noticed the |
…stack-monitoring-mb-ecs # Conflicts: # metricbeat/module/logstash/logstash_integration_test.go
Contributor
|
This pull request is now in conflicts. Could you fix it? 🙏 |
…stack-monitoring-mb-ecs # Conflicts: # metricbeat/module/elasticsearch/index/data_xpack.go
…ta on cluster/stats
ycombinator
approved these changes
Apr 26, 2021
Contributor
ycombinator
left a comment
There was a problem hiding this comment.
I skimmed through the changes in this PR as it's quite large. In any case, most of these changes were reviewed in smaller PRs that targeted the feature-stack-monitoring-mb-ecs branch.
3 tasks
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Elasticsearch, Kibana, Logstash and Beat modules has a flag called
xpack.enabled: true/falsethat can be activated to ensure that all metrics required on Stack Monitoring UI in Kibana are included in events from those modules. Before that, only a subset of the metrics were included.At the same time, all metrics have been "refactored" to be fully compatible with ECS and the common structure of Metricbeat events.
Finally, metrics will appear in indices of Metricbeat (the good ol'
metricbeat-*indices) instead of.monitoringindices. This way users will also be able to build their own dashboards around those modules more easily.xpack.enabled: trueflag will exist for some time more to help users activate the Metricbeat modules required by SM UI in Kibana.Why is it important?
ECS is a benefit for all users and Elasticsearch, Kibana, Logstash and Beat modules weren´t publishing events following the schema yet. OSS users will also benefit from this change, as the metrics will now be available in
metricbeat-*indices.Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
To test this PR locally, a SNAPSHOT version of Metricbeat and Kibana is necessary. Launch Elasticsearch, Kibana and Logstash and setup the same Metricbeat modules to monitor them. Metrics from those modules should appear in
metricbeat-*indices and the Stack Monitoring UI in Kibana should work without issues.To test Beats, you need to setup another Metricbeat or use this same one to monitor itself.
Related issues
metricbeat-*indices kibana#96205