Skip to content

Filestream input duplicating events after every restart #30061

@eedugon

Description

@eedugon

Under some circumstances the filestream input is processing all the events after every restart.

For example the following configuration works fine in a Filebeat running on Kubernetes (static input, no autodiscover):

    - type: filestream
      enabled: true
      paths:
       - /var/log/k8sapps/myapp/*.log
      fields:
       app.name: "myapp"
      fields_under_root: true

But if we add a second input (actually from same disk) then Filebeat sends everything after every restart:

    - type: filestream
      enabled: true
      paths:
       - /var/log/k8sapps/myapp/*.log
      fields:
       app.name: "myapp"
      fields_under_root: true

    - type: filestream
      enabled: true
      paths:
       - /var/log/k8sapps/secondapp/*.log
      fields:
       app.name: "secondapp"
      fields_under_root: true

I've tried to apply file_identity.inode_marker.path: /var/log/.filebeat-marker but the result is the same, and with a single input all works as expected.
The inodes of the files do not change after every restart and the volume UUID i don't know because it's not reported by lsblk (checked from the filebeat container itself).

Doc reference: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filestream-file-identity

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions