Description : The cloud foundry input is dropping logs, in a repeatable, reproducible manner.
- Version: 7.7
- Operating System: Linux / Darwin
Comparing Systems over 24 hours yields these results for a sample "ticker" app that produces a controlled cadence of log events.
syslog : 34,558 of 34,560 = 99.99421%
Firehose to syslog : 34,557 of 34,560 so 99.99131%
Filebeat : 34,319 of 34,560 so 99.30266%
FIlebeat needs to be as good or better than Firehose to syslog, i.e. at least 4 "9s"
Steps to reproduce:
-
Deploy app that emits a regular cadence of logs such as
https://github.com/bvader/scheduler
-
Deploy filebeat
-
Deploy firehose-to-syslog from:
https://github.com/cloudfoundry-community/firehose-to-syslog
-
Run and observe:
The black line is the correct / accurate number of logs via firehose-to-syslog the green line shows the dropped logs via filebeat.

Example in detail : scheduler-5s creates 2 logs every 5s = 24 events / Min
6 Hours = 8640 Events which can be observed below

However with filebeat only 8,582 events are capture during the exact same time frame.

NOTE : This same defect likely applies to metricbeat.
Description : The cloud foundry input is dropping logs, in a repeatable, reproducible manner.
Comparing Systems over 24 hours yields these results for a sample "ticker" app that produces a controlled cadence of log events.
syslog : 34,558 of 34,560 = 99.99421%
Firehose to syslog : 34,557 of 34,560 so 99.99131%
Filebeat : 34,319 of 34,560 so 99.30266%
FIlebeat needs to be as good or better than Firehose to syslog, i.e. at least 4 "9s"
Steps to reproduce:
Deploy app that emits a regular cadence of logs such as
https://github.com/bvader/scheduler
Deploy filebeat
Deploy firehose-to-syslog from:
https://github.com/cloudfoundry-community/firehose-to-syslog
Run and observe:
The black line is the correct / accurate number of logs via firehose-to-syslog the green line shows the dropped logs via filebeat.
Example in detail : scheduler-5s creates 2 logs every 5s = 24 events / Min

6 Hours = 8640 Events which can be observed below
However with filebeat only 8,582 events are capture during the exact same time frame.

NOTE : This same defect likely applies to metricbeat.