When using the disk queue, when Filebeat is restarted and a new event is sent, the disk queue creates a second segment file, if both are present, whenever Filebeat is restarted, the last event is always re-sent.
The original Discuss Topic contains the steps to reproduce using the file output, here I used the console output, either way works (aka, the bug is reproducible)
Steps to Reproduce:
1. Use the config below and start Filebeat.
filebeat.inputs:
- type: filestream
id: test-id
enabled: true
paths:
- /tmp/test.log
output.console:
enabled: true
pretty: true
logging.level: debug
logging.to_files: true
queue.disk:
max_size: 1GB
filebeat.shutdown_timeout: 30s2. Add a line to the log file.
Contents of the log file (/tmp/test.log):
2022-07-29 09:00:00.000 0000001
You will see a event in the console with the line that was added to the file.
The disk queue folder (./data/diskqueue) contains two files:
-rw------- 1 root root 424 Jul 29 16:43 0.seg
-rw------- 1 root root 28 Jul 29 16:43 state.dat
3. Enter a second line to the log file:
Contents of the log file (/tmp/test.log):
2022-07-29 09:00:00.000 0000001
2022-07-29 09:01:00.000 0000002
You will see a event in the console with the line that was added to the file.
4. Restart Filebeat
No changes in the files/folders
5. Add a new log line
Contents of the log file (/tmp/test.log):
2022-07-29 09:00:00.000 0000001
2022-07-29 09:01:00.000 0000002
2022-07-29 09:01:00.000 0000003
You will see a event in the console with the line that was added to the file.
The diskqueue creates a new segment file, now there are two:
-rw------- 1 root root 424 Jul 29 16:43 0.seg
-rw------- 1 root root 216 Jul 29 16:46 1.seg
-rw------- 1 root root 28 Jul 29 16:46 state.dat
5. Restart Filebeat
You will see the last event sent (2022-07-29 09:01:00.000 0000003).
Every time you restart Filebeat the same event will be sent.
6. Keep sending events and restarting filebeat
If we send more events, the current segment files are deleted, a new one created. Whenever there is only one segment file, no duplicated data is sent.
If Filebeat is restarted once more, a second segment file will be created, and while there are two segment files, the last segment file will always be sent when Filebeat restarts.
Logs and observations
When there are two segment files in the diskqueue and Filebeat starts, the "last segment" seems to be resent before the registry is started, the diskqueue also logs a negative number of events: Found -4 existing events on queue start.
Here are some debug logs being sent to stderr with the console output enabled: filebeat.log
When using the disk queue, when Filebeat is restarted and a new event is sent, the disk queue creates a second segment file, if both are present, whenever Filebeat is restarted, the last event is always re-sent.
7.17.5andmain(confirmed on those versions)The original Discuss Topic contains the steps to reproduce using the file output, here I used the console output, either way works (aka, the bug is reproducible)
Steps to Reproduce:
1. Use the config below and start Filebeat.
2. Add a line to the log file.
Contents of the log file (
/tmp/test.log):You will see a event in the console with the line that was added to the file.
The disk queue folder (
./data/diskqueue) contains two files:3. Enter a second line to the log file:
Contents of the log file (
/tmp/test.log):You will see a event in the console with the line that was added to the file.
4. Restart Filebeat
No changes in the files/folders
5. Add a new log line
Contents of the log file (
/tmp/test.log):You will see a event in the console with the line that was added to the file.
The diskqueue creates a new segment file, now there are two:
5. Restart Filebeat
You will see the last event sent (
2022-07-29 09:01:00.000 0000003).Every time you restart Filebeat the same event will be sent.
6. Keep sending events and restarting filebeat
If we send more events, the current segment files are deleted, a new one created. Whenever there is only one segment file, no duplicated data is sent.
If Filebeat is restarted once more, a second segment file will be created, and while there are two segment files, the last segment file will always be sent when Filebeat restarts.
Logs and observations
When there are two segment files in the diskqueue and Filebeat starts, the "last segment" seems to be resent before the registry is started, the diskqueue also logs a negative number of events:
Found -4 existing events on queue start.Here are some debug logs being sent to stderr with the console output enabled: filebeat.log