Under some circumstances the filestream input is processing all the events after every restart.
For example the following configuration works fine in a Filebeat running on Kubernetes (static input, no autodiscover):
- type: filestream
enabled: true
paths:
- /var/log/k8sapps/myapp/*.log
fields:
app.name: "myapp"
fields_under_root: true
But if we add a second input (actually from same disk) then Filebeat sends everything after every restart:
- type: filestream
enabled: true
paths:
- /var/log/k8sapps/myapp/*.log
fields:
app.name: "myapp"
fields_under_root: true
- type: filestream
enabled: true
paths:
- /var/log/k8sapps/secondapp/*.log
fields:
app.name: "secondapp"
fields_under_root: true
I've tried to apply file_identity.inode_marker.path: /var/log/.filebeat-marker but the result is the same, and with a single input all works as expected.
The inodes of the files do not change after every restart and the volume UUID i don't know because it's not reported by lsblk (checked from the filebeat container itself).
Doc reference: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filestream-file-identity
Under some circumstances the filestream input is processing all the events after every restart.
For example the following configuration works fine in a Filebeat running on Kubernetes (static input, no autodiscover):
But if we add a second input (actually from same disk) then Filebeat sends everything after every restart:
I've tried to apply
file_identity.inode_marker.path: /var/log/.filebeat-markerbut the result is the same, and with a single input all works as expected.The inodes of the files do not change after every restart and the volume UUID i don't know because it's not reported by
lsblk(checked from the filebeat container itself).Doc reference: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filestream-file-identity