Skip to content

metricbeat monitoring of Elasticsearch and Kibana broke on recent 7.17.0-SNAPSHOT rpm #29920

@LeeDr

Description

@LeeDr

Please include configurations and logs if available.

For confirmed bugs, please report:

  • Version: 7.17.0-SNAPSHOT
  • Operating System: CentOS 7
  • Discuss Forum URL:
  • Steps to Reproduce:
  1. Install metricbeat and enable modules for monitoring Elasticsearch, Kibana, Logstash, and Beats

Monitoring worked on the 7.17.0-766e7ca2 SNAPSHOT build from Jan 18, 2022 10:02 AM
but fails on the 7.17.0-68da5d12 SNAPSHOT build from Jan 18, 2022 5:38 PM
and also fails on the 7.17.0-1bd53ff7 BC2 build.

I don't find anything of interest in the metricbeat log. I don't see anything about monitoring in the log with either the passing or failing tests.

All of the configuration is the same between the build that passes and the builds that fail. I've run the passing and failing builds locally with the same results as on Jenkins.

/etc/metricbeat/metricbeat.yml:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.period: 10s
  reload.enabled: false
metricbeat.max_start_delay: 10s
metricbeat.modules:
- module: system
  enabled: true
  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
  processes: ['.*']
- module: system
  enabled: true
  period: 1m
  metricsets:
    - filesystem
    - fsstat
- module: kibana
  metricsets: ["status"]
  enabled: false
  period: 10s
  hosts: ["localhost:5601"]
- module: elasticsearch
  metricsets: ["node", "node_stats"]
  enabled: false
  period: 10s
  hosts: ["localhost:9200"]


output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: https
  username: "beats_internal"
  password: "changeme"
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/metricbeat/ca.crt"]
  ssl.certificate: /etc/metricbeat/elasticsearch.crt
  ssl.key: /etc/metricbeat/elasticsearch.key

#setup.dashboards.snapshot: true

setup.kibana:
  protocol: https
  host: localhost:5601
  username: "beats_internal"
  password: "changeme"
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/metricbeat/ca.crt"]
  ssl.certificate: /etc/metricbeat/elasticsearch.crt
  ssl.key: /etc/metricbeat/elasticsearch.key

xpack.monitoring:
  enabled: false
  elasticsearch:
    url: "https://localhost:9200"
    username: elastic
    password: changeit
http.enabled: true
http.port: 5068
[root@localhost vagrant]# ls -l /etc/metricbeat/modules.d/ | grep -v disabled
total 272
-rw-r--r-- 1 root root  295 Jan 19 22:31 beat-xpack.yml
-rw-r--r-- 1 root root  456 Jan 19 22:31 elasticsearch-xpack.yml
-rw-r--r-- 1 root root  461 Jan 19 22:31 kibana-xpack.yml
-rw-r--r-- 1 root root  251 Jan 19 22:31 logstash-xpack.yml
-rw-r--r-- 1 root root  956 Jan 19 14:31 system.yml

The beats_internal user in the config has;

curl -k https://elastic:changeit@localhost:9200/_security/user/beats_internal
{"beats_internal":{"username":"beats_internal","roles":["beats_writer","beats_reader","kibana_admin","machine_learning_admin","remote_monitoring_agent"],"full_name":"beats_internal","email":"beats_internal@test.co","metadata":{},"enabled":true}}

And beats_reader role is;

[root@localhost vagrant]# curl -k https://elastic:changeit@localhost:9200/_security/role/beats_reader
{"beats_reader":{"cluster":[],"indices":[{"names":["*beat-*"],"privileges":["read","view_index_metadata"],"allow_restricted_indices":false}],"applications":[],"run_as":[],"metadata":{},"transient_metadata":{"enabled":true}}}

And beats_writer role is;

[root@localhost vagrant]# curl -k https://elastic:changeit@localhost:9200/_security/role/beats_writer
{"beats_writer":{"cluster":["manage_index_templates","monitor","manage_ilm"],"indices":[{"names":["filebeat-*","metricbeat-*","heartbeat-*","packetbeat-*","auditbeat-*","winlogbeat-*",".kibana*"],"privileges":["write","delete","create_index","manage","manage_ilm"],"allow_restricted_indices":false}],"applications":[],"run_as":[],"metadata":{},"transient_metadata":{"enabled":true}}}

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions