Skip to content

Fix metrics hints builder to avoid wrong container metadata usage when port is not exposed#18979

Merged
jsoriano merged 1 commit intoelastic:masterfrom
vjsamuel:fix_port_overlaps
Jun 16, 2020
Merged

Fix metrics hints builder to avoid wrong container metadata usage when port is not exposed#18979
jsoriano merged 1 commit intoelastic:masterfrom
vjsamuel:fix_port_overlaps

Conversation

@vjsamuel
Copy link
Copy Markdown
Contributor

@vjsamuel vjsamuel commented Jun 4, 2020 •
edited
Loading

Type of PR

  • Enhancement

What does this PR do?

I have enhanced the emitEvent to generate a pod level event with no port. The hints builder checks to see if there is a port or not. If not then it uses the pod level metadata to monitor data.host:<port> annotations. This ensures that we dont use the wrong container metadata to start up a metricbeat module or do it multiple times as well.

Why is it important?

The way that things exist today is that when ports arent exposed, the container meta is non-deterministic

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Fixes: #12011

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 4, 2020

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

1 similar comment
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 4, 2020

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 4, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 4, 2020 •
edited
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [jsoriano commented: jenkins, run the tests please]

  • Start Time: 2020-06-16T15:25:10.040+0000

  • Duration: 79 min 8 sec

Test stats 🧪

Test Results
Failed 0
Passed 9441
Skipped 1574
Total 11015

Steps errors

Expand to view the steps failures

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-06-16T16:01:05.557+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-06-16T16:03:27.465+0000

    • log

@vjsamuel vjsamuel force-pushed the fix_port_overlaps branch from 1d6c0c1 to 7083347 Compare June 4, 2020 17:34
@ChrsMark ChrsMark added [zube]: In Review review Team:Platforms Label for the Integrations - Platforms team labels Jun 5, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 5, 2020

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 5, 2020
jsoriano
jsoriano reviewed Jun 8, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @vjsamuel, thanks for this fix.

As you mention, this PR includes two things, one fix to avoid mapping incorrect ports, and an improvement on how beats generate autodiscover events for pods. Could you please separate the change in two PRs?
I think that the enhancement may require further discussion and I wouldn't like to block one thing with the other.

Changelogs will be needed.

@vjsamuel vjsamuel force-pushed the fix_port_overlaps branch 5 times, most recently from bd11535 to 5d1e81c Compare June 10, 2020 20:31
@jsoriano
Copy link
Copy Markdown
Member

jsoriano commented Jun 12, 2020

I think we can go on with this change. @vjsamuel could you take a look to the merge conflict?

@vjsamuel vjsamuel force-pushed the fix_port_overlaps branch from 5d1e81c to e0a6431 Compare June 12, 2020 15:46
@jsoriano
Copy link
Copy Markdown
Member

jsoriano commented Jun 12, 2020

jenkins, run the tests please

@vjsamuel vjsamuel force-pushed the fix_port_overlaps branch 2 times, most recently from ee02af1 to 08baa61 Compare June 15, 2020 06:01
@jsoriano
Copy link
Copy Markdown
Member

jsoriano commented Jun 15, 2020

jenkins, run the tests please

@vjsamuel vjsamuel force-pushed the fix_port_overlaps branch from 08baa61 to 1a5eac8 Compare June 15, 2020 23:30
@jsoriano jsoriano self-assigned this Jun 16, 2020
@jsoriano
Copy link
Copy Markdown
Member

jsoriano commented Jun 16, 2020

jenkins, run the tests please

@jsoriano jsoriano merged commit 2f7b501 into elastic:master Jun 16, 2020
@jsoriano jsoriano mentioned this pull request Jun 17, 2020
Merged
6 tasks
jsoriano pushed a commit to jsoriano/beats that referenced this pull request Jun 17, 2020
@vjsamuel @jsoriano
Fix metrics hints builder to avoid wrong container metadata usage whe…
3e7b675 
…n port is not exposed (elastic#18979)

(cherry picked from commit 2f7b501)
@jsoriano jsoriano added test-plan Add this PR to be manual test plan and removed test-plan Add this PR to be manual test plan labels Jun 17, 2020
jsoriano added a commit that referenced this pull request Jun 18, 2020
@jsoriano @vjsamuel
Fix metrics hints builder to avoid wrong container metadata usage whe…
f65b237 
…n port is not exposed (#18979) (#19233)

(cherry picked from commit 2f7b501)

Co-authored-by: Vijay Samuel <vjsamuel@ebay.com>
@vjsamuel vjsamuel deleted the fix_port_overlaps branch June 22, 2020 06:14
@vjsamuel vjsamuel mentioned this pull request Jun 22, 2020
Merged
6 tasks
@ChrsMark ChrsMark mentioned this pull request Jun 25, 2020
Closed
jsoriano pushed a commit that referenced this pull request Jul 2, 2020
@vjsamuel
Fix ordering and duplicate configs on autodiscover (#19317)
58edbb4 
#18979 introduced a pod level event which is generated after all container events.
The ordering is wrong in that pod events are sent last which would generate a valid
event similar to container events. The ordering needs to be pod first and container
events next so that pod events dont override valid container events. One other issue
was that the pod level hint generates a single config with all hosts and it wont get
over written by container hints causing more than one config to be spun up for the
same hint (one with a container meta and one without).
@jsoriano jsoriano mentioned this pull request Jul 2, 2020
Merged
6 tasks
jsoriano pushed a commit to jsoriano/beats that referenced this pull request Jul 2, 2020
@vjsamuel @jsoriano
Fix ordering and duplicate configs on autodiscover (elastic#19317)
7967db7 
elastic#18979 introduced a pod level event which is generated after all container events.
The ordering is wrong in that pod events are sent last which would generate a valid
event similar to container events. The ordering needs to be pod first and container
events next so that pod events dont override valid container events. One other issue
was that the pod level hint generates a single config with all hosts and it wont get
over written by container hints causing more than one config to be spun up for the
same hint (one with a container meta and one without).

(cherry picked from commit 58edbb4)
jsoriano added a commit that referenced this pull request Jul 6, 2020
@jsoriano @vjsamuel
Fix ordering and duplicate configs on autodiscover (#19317) (#19614)
24b1a0b 
#18979 introduced a pod level event which is generated after all container events.
The ordering is wrong in that pod events are sent last which would generate a valid
event similar to container events. The ordering needs to be pod first and container
events next so that pod events dont override valid container events. One other issue
was that the pod level hint generates a single config with all hosts and it wont get
over written by container hints causing more than one config to be spun up for the
same hint (one with a container meta and one without).

(cherry picked from commit 58edbb4)

Co-authored-by: Vijay Samuel <vjsamuel@ebay.com>
@jsoriano jsoriano mentioned this pull request Aug 21, 2020
Closed
This was referenced Sep 1, 2020
Merged
Merged
Merged
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
@vjsamuel @melchiormoulin
Fix metrics hints builder to avoid wrong container metadata usage whe…
603d344 
…n port is not exposed (elastic#18979)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
@vjsamuel @melchiormoulin
Fix ordering and duplicate configs on autodiscover (elastic#19317)
14c84a3 
elastic#18979 introduced a pod level event which is generated after all container events.
The ordering is wrong in that pod events are sent last which would generate a valid
event similar to container events. The ordering needs to be pod first and container
events next so that pod events dont override valid container events. One other issue
was that the pod level hint generates a single config with all hosts and it wont get
over written by container hints causing more than one config to be spun up for the
same hint (one with a container meta and one without).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChrsMark ChrsMark ChrsMark left review comments

@jsoriano jsoriano jsoriano approved these changes

Assignees

@jsoriano jsoriano

Labels

review Team:Platforms Label for the Integrations - Platforms team v7.9.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Metricbeat hints builder discovers same hosts multiple times

5 participants

@vjsamuel @elasticmachine @jsoriano @ChrsMark @andresrc