Skip to content

Cherry-pick #19125 to 7.x: Add support for cloudfoundry logs collection using consumer v1 API#19212

Merged
jsoriano merged 1 commit intoelastic:7.xfrom
jsoriano:backport_19125_7.x
Jun 16, 2020
Merged

Cherry-pick #19125 to 7.x: Add support for cloudfoundry logs collection using consumer v1 API#19212
jsoriano merged 1 commit intoelastic:7.xfrom
jsoriano:backport_19125_7.x

Conversation

@jsoriano
Copy link
Copy Markdown
Member

@jsoriano jsoriano commented Jun 16, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #19125 to 7.x branch. Original message:

What does this PR do?

Add support for collection of logs from cloud foundry using consumer v1 API. v1 API uses doppler/trafficcontrol to get events from the firehose, v2 API uses the reverse log proxy (RLP).

v2 API is still supported, and can be selected by adding version: v2 to the configuration, v1 is used by default as is in principle more reliable. version is ignored by now by Metricbeat, that continues using v2. Support for v1 in Metricbeat will be implemented in a future change.

This change adds some additional dependencies for the doppler consumer (noaa).

Why is it important?

We already support the v2 API, but we saw that this API is not reliable enough (see #18202).

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Review FIXMEs in code.
  • Test with PCF

How to test this PR locally

  • Run filebeat with the cloudfoundry input and check that it collects events. Try these combinations of the version option:
    • No version option (what would use v1)
    • version: v1.
    • version: v2.
    • For other values of version it should fail to start.

Related issues

@jsoriano
Add support for cloudfoundry logs collection using consumer v1 API (e…
d3bcf31 
…lastic#19125)

Add support for collection of logs from cloud foundry using consumer v1 API.
v1 API uses doppler/trafficcontrol to get events from the firehose, v2 API uses
the reverse log proxy (RLP).

v2 API is still supported, and can be selected by adding `version: v2` to the
configuration, v1 is used by default as is in principle more reliable. `version`
is ignored by now by Metricbeat.

(cherry picked from commit 3ebceec)
@jsoriano jsoriano added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Jun 16, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 16, 2020

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 16, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 16, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19212 opened]

  • Start Time: 2020-06-16T08:09:53.674+0000

  • Duration: 83 min 44 sec

Test stats 🧪

Test Results
Failed 0
Passed 6694
Skipped 1069
Total 7763

Steps errors

Expand to view the steps failures

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-06-16T08:45:25.718+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-06-16T09:29:42.790+0000

    • log

ChrsMark
ChrsMark reviewed Jun 16, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano how about fixing #19213 here to avoid having an extra backport PR?
nah, this PR seems to not have any lint issues? 🤔

@jsoriano
Copy link
Copy Markdown
Member Author

jsoriano commented Jun 16, 2020

@jsoriano how about fixing #19213 here to avoid having an extra backport PR?
nah, this PR seems to not have any lint issues? 🤔

#19213 seems to be needed in combination with #19135.

@jsoriano jsoriano merged commit e8aa16e into elastic:7.x Jun 16, 2020
@jsoriano jsoriano deleted the backport_19125_7.x branch June 16, 2020 10:41
@zube zube bot removed the [zube]: Done label Oct 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

Assignees

No one assigned

Labels

backport Team:Platforms Label for the Integrations - Platforms team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jsoriano @elasticmachine @ChrsMark