#30018 added support for ES 8.0's ECS formatted logs.
Since the logs are ECS format we opted for minimal processing and removed the drop found in the 7.x log processors.
Unfortunately in cases like kubernetes, the combination of multiple file sets (server, audit, etc...) and using a single mixed stream (stdout) means we end up double-ingesting the same messages across multiple pipelines and storing the duplicates.
To work around this we can add a drop to the 8.0 pipelines for the elasticsearch module.
#30018 added support for ES 8.0's ECS formatted logs.
Since the logs are ECS format we opted for minimal processing and removed the
dropfound in the 7.x log processors.Unfortunately in cases like kubernetes, the combination of multiple file sets (server, audit, etc...) and using a single mixed stream (stdout) means we end up double-ingesting the same messages across multiple pipelines and storing the duplicates.
To work around this we can add a
dropto the 8.0 pipelines for the elasticsearch module.