Skip to content

Mapping error ingesting logstash loggin #20709

@psanz-estc

Description

@psanz-estc

ES returns a mapping error when trying to ingest Logstash logs (in json format) via logstash module

FB mapping:

 "logstash": {
          "properties": {
            "log": {
              "properties": {
                "log_event": {
                  "type": "object"
                },
                "module": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "pipeline_id": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "thread": {
                  "type": "keyword",
                  "fields": {
                    "text": {
                      "type": "text",
                      "norms": false
                    }
                  },
                  "ignore_above": 1024
                }
              }
            },

Event in Filebeat:

2020-08-14T17:52:51.518+0100	DEBUG	[processors]	processing/processors.go:186	Publish event: {
  "@timestamp": "2020-08-14T16:52:51.518Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.2",
    "pipeline": "filebeat-7.5.2-logstash-log-pipeline-json"
  },
  "input": {
    "type": "log"
  },
  "event": {
    "module": "logstash",
    "dataset": "logstash.log"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "host": {
    "containerized": false,
    "hostname": "indexXXXX",
    "architecture": "x86_64",
    "os": {
      "kernel": "3.10.0-1062.7.1.el7.x86_64",
      "codename": "Maipo",
      "platform": "rhel",
      "version": "7.7 (Maipo)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux Server"
    },
    "name": "indexXXXX",
    "id": "2d268dd2fe52485fbac908a2d7eb4ec7"
  },
  "message": "{\"level\":\"WARN\",\"loggerName\":\"logstash.outputs.elasticsearch\",\"timeMillis\":1597423971116,\"thread\":\"[XXXXX-logs]>worker0\",\"logEvent\":{\"message\":\"Could not index event to Elasticsearch.\",\"status\":400,\"action\":[\"index\",{\"_index\":\"indexXXXX\",\"_type\":\"_doc\"},{\"metaClass\":{\"metaClass\":{\"metaClass\":{\"action\":\"[\\\"index\\\", {:_id=>nil, :_index=>\\\"indexXXXX\\\", :routing=>nil, :_type=>\\\"_doc\\\"}, #<LogStash::Event:0x2ee7e327>]\",\"response\":{\"index\":{\"_index\":\"indexXXXX-2020.08.13-000001\",\"_type\":\"_doc\",\"_id\":\"dNXj7XMB12EK1s8HakX7\",\"status\":400,\"error\":{\"type\":\"strict_dynamic_mapping_exception\",\"reason\":\"mapping set to strict, dynamic introduction of [guitester] within [_doc] is not allowed\"}}}}}}}]}}",
  "fileset": {
    "name": "log"
  },
  "agent": {
    "ephemeral_id": "8c03cbf1-4e9e-4d49-8929-a35e34313565",
    "hostname": "XXXXXXX",
    "id": "133b094d-2331-4159-908e-f2b9287a5fa7",
    "version": "7.5.2",
    "type": "filebeat"
  },
  "log": {
    "offset": 2179,
    "file": {
      "path": "/var/logs/logs/logstash-json.log"
    }
  },
  "service": {
    "type": "logstash"
  }
}

And it is rejected in ES due the mapping not being aligned:

2020-08-14T17:52:52.525+0100	WARN	elasticsearch/client.go:511	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfc5ce38dee4ad4e, ext:101083374145, loc:(*time.Location)(0x4e5d700)}, Meta:common.MapStr{"pipeline":"filebeat-7.5.2-logstash-log-pipeline-json"}, Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"8c03cbf1-4e9e-4d49-8929-a35e34313565", "hostname":"XXXXXXX", "id":"133b094d-2331-4159-908e-f2b9287a5fa7", "type":"filebeat", "version":"7.5.2"}, "ecs":common.MapStr{"version":"1.1.0"}, "event":common.MapStr{"dataset":"logstash.log", "module":"logstash"}, "fileset":common.MapStr{"name":"log"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"XXXXXXX", "id":"2d268dd2fe52485fbac908a2d7eb4ec7", "name":"XXXXXXX", "os":common.MapStr{"codename":"Maipo", "family":"redhat", "kernel":"3.10.0-1062.7.1.el7.x86_64", "name":"Red Hat Enterprise Linux Server", "platform":"rhel", "version":"7.7 (Maipo)"}}, "input":common.MapStr{"type":"log"}, "log":common.MapStr{"file":common.MapStr{"path":"/var/logs/logs/logstash-json.log"}, "offset":2179}, "message":"{\"level\":\"WARN\",\"loggerName\":\"logstash.outputs.elasticsearch\",\"timeMillis\":1597423971116,\"thread\":\"[indexXXXX]>worker0\",\"logEvent\":{\"message\":\"Could not index event to Elasticsearch.\",\"status\":400,\"action\":[\"index\",{\"_index\":\"indexXXXX\",\"_type\":\"_doc\"},{\"metaClass\":{\"metaClass\":{\"metaClass\":{\"action\":\"[\\\"index\\\", {:_id=>nil, :_index=>\\\"indexXXXX\\\", :routing=>nil, :_type=>\\\"_doc\\\"}, #<LogStash::Event:0x2ee7e327>]\",\"response\":{\"index\":{\"_index\":\"indexXXXX-2020.08.13-000001\",\"_type\":\"_doc\",\"_id\":\"dNXj7XMB12EK1s8HakX7\",\"status\":400,\"error\":{\"type\":\"strict_dynamic_mapping_exception\",\"reason\":\"mapping set to strict, dynamic introduction of [guitester] within [_doc] is not allowed\"}}}}}}}]}}", "service":common.MapStr{"type":"logstash"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000443a00), Source:"/var/logs/log/logstash-json.log", Offset:2904, Timestamp:time.Time{wall:0xbfc5ce389ed00996, ext:100082021520, loc:(*time.Location)(0x4e5d700)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x4f, Device:0xfd06}}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"mapper [logstash.log.log_event.action] of different type, current_type [keyword], merged_type [ObjectMapper]"}

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions