Skip to content

Drop non-matching logs inside elasticsearch filebeat module 8.0 pipelines #30428

@matschaffer

Description

@matschaffer

#30018 added support for ES 8.0's ECS formatted logs.

Since the logs are ECS format we opted for minimal processing and removed the drop found in the 7.x log processors.

Unfortunately in cases like kubernetes, the combination of multiple file sets (server, audit, etc...) and using a single mixed stream (stdout) means we end up double-ingesting the same messages across multiple pipelines and storing the duplicates.

To work around this we can add a drop to the 8.0 pipelines for the elasticsearch module.

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions