Skip to content

Filebeat : Cloud foundry input dropping logs #18202

@bvader

Description

@bvader

Description : The cloud foundry input is dropping logs, in a repeatable, reproducible manner.

  • Version: 7.7
  • Operating System: Linux / Darwin

Comparing Systems over 24 hours yields these results for a sample "ticker" app that produces a controlled cadence of log events.
syslog : 34,558 of 34,560 = 99.99421%
Firehose to syslog : 34,557 of 34,560 so 99.99131%
Filebeat : 34,319 of 34,560 so 99.30266%

FIlebeat needs to be as good or better than Firehose to syslog, i.e. at least 4 "9s"

Steps to reproduce:

  1. Deploy app that emits a regular cadence of logs such as
    https://github.com/bvader/scheduler

  2. Deploy filebeat

  3. Deploy firehose-to-syslog from:
    https://github.com/cloudfoundry-community/firehose-to-syslog

  4. Run and observe:
    The black line is the correct / accurate number of logs via firehose-to-syslog the green line shows the dropped logs via filebeat.

Screen Shot 2020-05-04 at 11 44 29 AM

Example in detail : scheduler-5s creates 2 logs every 5s = 24 events / Min
6 Hours = 8640 Events which can be observed below
Screen Shot 2020-05-04 at 11 50 36 AM

However with filebeat only 8,582 events are capture during the exact same time frame.
Screen Shot 2020-05-04 at 11 50 43 AM

NOTE : This same defect likely applies to metricbeat.

Metadata

Metadata

Labels

FilebeatFilebeatTeam:PlatformsLabel for the Integrations - Platforms teambug

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions