I am using one instance of filebeat to store docker logs in local log file, each line is json. Another filebeat is reading this file and sending the logs to elasticsearch. This second filebeat does not work if I enable json.overwrite_keys: true.
My config:
name: "myname"
path:
home: /opt/filebeat
data: "/app/volumes/data/logforwarder"
filebeat.prospectors:
- type: log
paths:
- "/app/volumes/logs/docker-json.log*"
json:
keys_under_root: true
add_error_key: true
message_key: message
overwrite_keys: true
close_inactive: 1440h
fields:
machine_group: "mg1"
setup.template.enabled: false
output.elasticsearch:
hosts: ["eshost:9210"]
index: "mg1-dockercontainer-%{+yyyy.MM}"
protocol: https
ssl:
certificate_authorities: ["/app/volumes/config/certs/elasticsearch-ca.crt"]
enabled: true
verification_mode: full
supported_protocols: [ "TLSv1.2" ]
username: sl
password: sl
Filebeat log:
log/harvester.go:216 Harvester started for file: /app/volumes/logs/docker-json.log
panic: assignment to entry in nil map
goroutine 73 [running]:
github.com/elastic/beats/libbeat/common.MapStr.Update(0x0, 0xc4204af0b0)
/go/src/github.com/elastic/beats/libbeat/common/mapstr.go:43 +0xd7
github.com/elastic/beats/libbeat/common/jsontransform.WriteJSONKeys(0xc420053bb8, 0xc4204af080, 0x16ca001)
/go/src/github.com/elastic/beats/libbeat/common/jsontransform/jsonhelper.go:50 +0xa9b
github.com/elastic/beats/filebeat/harvester/reader.MergeJSONFields(0xc4204af2c0, 0xc4204af080, 0xc420053ce0, 0x0, 0x0, 0x10101, 0xc420017260, 0x21, 0xfd65)
/go/src/github.com/elastic/beats/filebeat/harvester/reader/json.go:120 +0x1c3
github.com/elastic/beats/filebeat/prospector/log.(*Harvester).Run(0xc42009b900, 0x0, 0x0)
/go/src/github.com/elastic/beats/filebeat/prospector/log/harvester.go:287 +0x98a
github.com/elastic/beats/filebeat/harvester.(*Registry).Start.func1(0xc42005ccc0, 0x1f3c240, 0xc42009b900)
/go/src/github.com/elastic/beats/filebeat/harvester/registry.go:81 +0x95
created by github.com/elastic/beats/filebeat/harvester.(*Registry).Start
/go/src/github.com/elastic/beats/filebeat/harvester/registry.go:74 +0x148
Example entry from /app/volumes/logs/docker-json.log:
{"@timestamp":"2018-02-14T08:58:13.484Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.1"},"stream":"stdout","message":"[2018-02-14T09:58:13,483][INFO ][logstash.agent ] Pipelines running {:count=\u003e2, :pipelines=\u003e[".monitoring-logstash", "beatsmain"]}","prospector":{"type":"docker"},"docker":{"container":{"id":"48425cf7a58af112ff950ddcebe06968aed2d589bd9729422e98a4c9f7bd2a56","name":"els","image":"localhost/elk/logstash:latest","labels":{"com":{"docker":{"compose":{"version":"1.18.0","config-hash":"0993dddd4bf7d23bb20d04392b9fcb853da11d00a4b5d615ecf31af2be02f26b","container-number":"1","oneoff":"False","project":"elspo","service":"logstash"}}}}}},"beat":{"version":"6.2.1","name":"myname","hostname":"7a0006d02784"},"source":"/var/lib/docker/containers/48425cf7a58af112ff950ddcebe06968aed2d589bd9729422e98a4c9f7bd2a56/48425cf7a58af112ff950ddcebe06968aed2d589bd9729422e98a4c9f7bd2a56-json.log","offset":21400}
Affected filebeat versions: at least 6.1.x, 6.2.x
I am using one instance of filebeat to store docker logs in local log file, each line is json. Another filebeat is reading this file and sending the logs to elasticsearch. This second filebeat does not work if I enable json.overwrite_keys: true.
My config:
Filebeat log:
Example entry from /app/volumes/logs/docker-json.log:
{"@timestamp":"2018-02-14T08:58:13.484Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.1"},"stream":"stdout","message":"[2018-02-14T09:58:13,483][INFO ][logstash.agent ] Pipelines running {:count=\u003e2, :pipelines=\u003e[".monitoring-logstash", "beatsmain"]}","prospector":{"type":"docker"},"docker":{"container":{"id":"48425cf7a58af112ff950ddcebe06968aed2d589bd9729422e98a4c9f7bd2a56","name":"els","image":"localhost/elk/logstash:latest","labels":{"com":{"docker":{"compose":{"version":"1.18.0","config-hash":"0993dddd4bf7d23bb20d04392b9fcb853da11d00a4b5d615ecf31af2be02f26b","container-number":"1","oneoff":"False","project":"elspo","service":"logstash"}}}}}},"beat":{"version":"6.2.1","name":"myname","hostname":"7a0006d02784"},"source":"/var/lib/docker/containers/48425cf7a58af112ff950ddcebe06968aed2d589bd9729422e98a4c9f7bd2a56/48425cf7a58af112ff950ddcebe06968aed2d589bd9729422e98a4c9f7bd2a56-json.log","offset":21400}
Affected filebeat versions: at least 6.1.x, 6.2.x