Rel: elastic/kibana#120825
I’m trying to use filebeat (master, mage build) to collect ES logs (master, ./gradlew localDistro) for use in stack monitoring. Here’s the config.
http.enabled: true
filebeat.modules:
- module: elasticsearch
server:
enabled: true
var.paths:
- ../../elasticsearch/build/distribution/local/elasticsearch-8.1.0-SNAPSHOT/logs/*.log
- ../../elasticsearch/build/distribution/local/elasticsearch-8.1.0-SNAPSHOT/logs/*_server.json
output.elasticsearch:
hosts: [ "http://localhost:9200" ]
username: "elastic"
password: "changeme"It seems like it’s only able to output the logs/*.log files not the logs/*_server.json so I’m getting only plaintext documents which lack a cluster ID.
If I use this config:
http.enabled: true
filebeat.modules:
- module: elasticsearch
server:
enabled: true
var.paths:
- ../../elasticsearch/build/distribution/local/elasticsearch-8.1.0-SNAPSHOT/logs/*_server.json
output.console:
pretty: true
I get documents in the output:
{
"@timestamp": "2022-01-18T04:56:48.666Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.1.0",
"pipeline": "filebeat-8.1.0-elasticsearch-server-pipeline"
},
"log": {
"offset": 339955,
"file": {
"path": "/Users/matschaffer/elastic/elasticsearch/build/distribution/local/elasticsearch-8.1.0-SNAPSHOT/logs/main_server.json"
}
},
"message": "{\"@timestamp\":\"2022-01-18T04:28:53.149Z\", \"log.level\": \"INFO\", \"message\":\"[.kibana_8.1.0_001/ausLgRqnR-SDU35g1hgOaw] update_mapping [_doc]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[matschaffer-mbp2019.lan][masterService#updateTask][T#1]\",\"log.logger\":\"org.elasticsearch.cluster.metadata.MetadataMappingService\",\"elasticsearch.cluster.uuid\":\"_NFvREx5RfeRW-X2HgEzrg\",\"elasticsearch.node.id\":\"hatOG6LsTYe1L7tKLlm2Lg\",\"elasticsearch.node.name\":\"matschaffer-mbp2019.lan\",\"elasticsearch.cluster.name\":\"main\"}",
"fileset": {
"name": "server"
},
"service": {
"type": "elasticsearch"
},
"input": {
"type": "log"
},
"ecs": {
"version": "1.12.0"
},
"event": {
"module": "elasticsearch",
"dataset": "elasticsearch.server"
},
"host": {
"name": "matschaffer-mbp2019.lan"
},
"agent": {
"name": "matschaffer-mbp2019.lan",
"type": "filebeat",
"version": "8.1.0",
"ephemeral_id": "2256c651-c048-4083-8e39-fe6f76d131f4",
"id": "4cda5c97-4168-4090-911e-3c34c1ae00aa"
}
}
But if I switch the output to
output.elasticsearch:
hosts: [ "http://localhost:9200" ]
username: "elastic"
password: "changeme"
No indices appear in the output cluster.
Rel: elastic/kibana#120825
I’m trying to use filebeat (master,
mage build) to collect ES logs (master,./gradlew localDistro) for use in stack monitoring. Here’s the config.It seems like it’s only able to output the
logs/*.logfiles not thelogs/*_server.jsonso I’m getting only plaintext documents which lack a cluster ID.If I use this config:
I get documents in the output:
But if I switch the output to
No indices appear in the output cluster.