ES returns a mapping error when trying to ingest Logstash logs (in json format) via logstash module
FB mapping:
"logstash": {
"properties": {
"log": {
"properties": {
"log_event": {
"type": "object"
},
"module": {
"type": "keyword",
"ignore_above": 1024
},
"pipeline_id": {
"type": "keyword",
"ignore_above": 1024
},
"thread": {
"type": "keyword",
"fields": {
"text": {
"type": "text",
"norms": false
}
},
"ignore_above": 1024
}
}
},
Event in Filebeat:
2020-08-14T17:52:51.518+0100 DEBUG [processors] processing/processors.go:186 Publish event: {
"@timestamp": "2020-08-14T16:52:51.518Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.5.2",
"pipeline": "filebeat-7.5.2-logstash-log-pipeline-json"
},
"input": {
"type": "log"
},
"event": {
"module": "logstash",
"dataset": "logstash.log"
},
"ecs": {
"version": "1.1.0"
},
"host": {
"containerized": false,
"hostname": "indexXXXX",
"architecture": "x86_64",
"os": {
"kernel": "3.10.0-1062.7.1.el7.x86_64",
"codename": "Maipo",
"platform": "rhel",
"version": "7.7 (Maipo)",
"family": "redhat",
"name": "Red Hat Enterprise Linux Server"
},
"name": "indexXXXX",
"id": "2d268dd2fe52485fbac908a2d7eb4ec7"
},
"message": "{\"level\":\"WARN\",\"loggerName\":\"logstash.outputs.elasticsearch\",\"timeMillis\":1597423971116,\"thread\":\"[XXXXX-logs]>worker0\",\"logEvent\":{\"message\":\"Could not index event to Elasticsearch.\",\"status\":400,\"action\":[\"index\",{\"_index\":\"indexXXXX\",\"_type\":\"_doc\"},{\"metaClass\":{\"metaClass\":{\"metaClass\":{\"action\":\"[\\\"index\\\", {:_id=>nil, :_index=>\\\"indexXXXX\\\", :routing=>nil, :_type=>\\\"_doc\\\"}, #<LogStash::Event:0x2ee7e327>]\",\"response\":{\"index\":{\"_index\":\"indexXXXX-2020.08.13-000001\",\"_type\":\"_doc\",\"_id\":\"dNXj7XMB12EK1s8HakX7\",\"status\":400,\"error\":{\"type\":\"strict_dynamic_mapping_exception\",\"reason\":\"mapping set to strict, dynamic introduction of [guitester] within [_doc] is not allowed\"}}}}}}}]}}",
"fileset": {
"name": "log"
},
"agent": {
"ephemeral_id": "8c03cbf1-4e9e-4d49-8929-a35e34313565",
"hostname": "XXXXXXX",
"id": "133b094d-2331-4159-908e-f2b9287a5fa7",
"version": "7.5.2",
"type": "filebeat"
},
"log": {
"offset": 2179,
"file": {
"path": "/var/logs/logs/logstash-json.log"
}
},
"service": {
"type": "logstash"
}
}
And it is rejected in ES due the mapping not being aligned:
2020-08-14T17:52:52.525+0100 WARN elasticsearch/client.go:511 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfc5ce38dee4ad4e, ext:101083374145, loc:(*time.Location)(0x4e5d700)}, Meta:common.MapStr{"pipeline":"filebeat-7.5.2-logstash-log-pipeline-json"}, Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"8c03cbf1-4e9e-4d49-8929-a35e34313565", "hostname":"XXXXXXX", "id":"133b094d-2331-4159-908e-f2b9287a5fa7", "type":"filebeat", "version":"7.5.2"}, "ecs":common.MapStr{"version":"1.1.0"}, "event":common.MapStr{"dataset":"logstash.log", "module":"logstash"}, "fileset":common.MapStr{"name":"log"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"XXXXXXX", "id":"2d268dd2fe52485fbac908a2d7eb4ec7", "name":"XXXXXXX", "os":common.MapStr{"codename":"Maipo", "family":"redhat", "kernel":"3.10.0-1062.7.1.el7.x86_64", "name":"Red Hat Enterprise Linux Server", "platform":"rhel", "version":"7.7 (Maipo)"}}, "input":common.MapStr{"type":"log"}, "log":common.MapStr{"file":common.MapStr{"path":"/var/logs/logs/logstash-json.log"}, "offset":2179}, "message":"{\"level\":\"WARN\",\"loggerName\":\"logstash.outputs.elasticsearch\",\"timeMillis\":1597423971116,\"thread\":\"[indexXXXX]>worker0\",\"logEvent\":{\"message\":\"Could not index event to Elasticsearch.\",\"status\":400,\"action\":[\"index\",{\"_index\":\"indexXXXX\",\"_type\":\"_doc\"},{\"metaClass\":{\"metaClass\":{\"metaClass\":{\"action\":\"[\\\"index\\\", {:_id=>nil, :_index=>\\\"indexXXXX\\\", :routing=>nil, :_type=>\\\"_doc\\\"}, #<LogStash::Event:0x2ee7e327>]\",\"response\":{\"index\":{\"_index\":\"indexXXXX-2020.08.13-000001\",\"_type\":\"_doc\",\"_id\":\"dNXj7XMB12EK1s8HakX7\",\"status\":400,\"error\":{\"type\":\"strict_dynamic_mapping_exception\",\"reason\":\"mapping set to strict, dynamic introduction of [guitester] within [_doc] is not allowed\"}}}}}}}]}}", "service":common.MapStr{"type":"logstash"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000443a00), Source:"/var/logs/log/logstash-json.log", Offset:2904, Timestamp:time.Time{wall:0xbfc5ce389ed00996, ext:100082021520, loc:(*time.Location)(0x4e5d700)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x4f, Device:0xfd06}}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"mapper [logstash.log.log_event.action] of different type, current_type [keyword], merged_type [ObjectMapper]"}
ES returns a mapping error when trying to ingest Logstash logs (in json format) via logstash module
FB mapping:
Event in Filebeat:
And it is rejected in ES due the mapping not being aligned: