When indexing the filebeat test data from the beats 6.7 branch into a 7.0.0-SNAPSHOT elasticsearch cluster, the access logs for the web servers (at least nginx, iis and traefik) fail to be indexed with errors messages akin to the following:
info [o.e.a.b.TransportShardBulkAction] [${HOSTNAME}] [filebeat-6.7.0-2019.02.08][1] failed to execute bulk item (index) index {[filebeat-6.7.0-2019.02.08][_doc][-v9vzWgBSKfxSV4q4CHr], source[{"offset":1204,"log":{"file":{"path":"${SOMEDIR}/beats/filebeat/module/iis/access/test/test.log"}},"prospector":{"type":"log"},"read_timestamp":"2019-02-08T14:08:07.032Z","source":"${SOMEDIR}/beats/filebeat/module/iis/access/test/test.log","fileset":{"module":"iis","name":"access"},"error":{"message":"field [iis.access.user_agent.original] already exists"},"input":{"type":"log"},"iis":{"access":{"server_name":"MACHINE-NAME","agent":"Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:57.0)+Gecko/20100101+Firefox/57.0","response_code":"200","cookie":"-","method":"GET","sub_status":"0","user_name":"-","http_version":"1.1","url":"/","site_name":"W3SVC1","referrer":"-","body_received":{"bytes":"456"},"hostname":"example.com","remote_ip":"85.181.35.98","port":"80","server_ip":"127.0.0.1","body_sent":{"bytes":"123"},"win32_status":"0","request_time_ms":"789","query_string":"-","user_agent":{"original":"Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:57.0)+Gecko/20100101+Firefox/57.0","os":{"name":"Windows"},"name":"Firefox","device":{"name":"Other"},"version":"57.0"}}},"@timestamp":"2018-01-01T10:11:12.000Z","beat":{"hostname":"${HOSTNAME}","name":"${HOSTNAME}","version":"6.7.0"},"host":{"os":{"build":"rolling","name":"Arch Linux","family":"","version":"","platform":"arch"},"containerized":false,"name":"${HOSTNAME}","id":"${HOSTID}","architecture":"x86_64"},"event":{"dataset":"iis.access"}}]}
│ org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [iis.access.user_agent.os] of type [keyword] in document with id '-v9vzWgBSKfxSV4q4CHr'
| ...SNIP...
│ Caused by: java.lang.IllegalStateException: Can't get text on a START_OBJECT at 1:419
I haven't tested all of them, but this probably happens for all filebeat web server modules that use the user_agent processor in the pipeline.
Versions:
Operating System:
Linux 4.20.6-arch1-1-ARCH elastic/beats#1 SMP PREEMPT Thu Jan 31 08:22:01 UTC 2019 x86_64 GNU/LinuxDescription:
When indexing the filebeat test data from the beats 6.7 branch into a 7.0.0-SNAPSHOT elasticsearch cluster, the access logs for the web servers (at least nginx, iis and traefik) fail to be indexed with errors messages akin to the following:
I would suspect that the
user_agent.originalfield, which is already populated byuser_agentingest processor in elasticsearch 7.0.0, causes therenameoperation in the version 6.7.0 pipeline to fail.I haven't tested all of them, but this probably happens for all filebeat web server modules that use the
user_agentprocessor in the pipeline.Steps to Reproduce:
nginxoriis