Skip to content

Cherry-pick #18979 to 7.x: Fix metrics hints builder to avoid wrong container metadata usage when port is not exposed#19233

Merged
jsoriano merged 1 commit intoelastic:7.xfrom
jsoriano:backport_18979_7.x
Jun 18, 2020
Merged

Cherry-pick #18979 to 7.x: Fix metrics hints builder to avoid wrong container metadata usage when port is not exposed#19233
jsoriano merged 1 commit intoelastic:7.xfrom
jsoriano:backport_18979_7.x

Conversation

@jsoriano
Copy link
Copy Markdown
Member

@jsoriano jsoriano commented Jun 17, 2020

Cherry-pick of PR #18979 to 7.x branch. Original message:

Type of PR

  • Enhancement

What does this PR do?

I have enhanced the emitEvent to generate a pod level event with no port. The hints builder checks to see if there is a port or not. If not then it uses the pod level metadata to monitor data.host:<port> annotations. This ensures that we dont use the wrong container metadata to start up a metricbeat module or do it multiple times as well.

Why is it important?

The way that things exist today is that when ports arent exposed, the container meta is non-deterministic

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Fixes: #12011

Co-Authored-By: Vijay Samuel vjsamuel@ebay.com

@jsoriano jsoriano added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Jun 17, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 17, 2020
@jsoriano jsoriano self-assigned this Jun 17, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 17, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [jsoriano commented: jenkins, run the tests please]

  • Start Time: 2020-06-17T09:40:12.589+0000

  • Duration: 80 min 3 sec

Test stats 🧪

Test Results
Failed 1
Passed 6777
Skipped 1069
Total 7847

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat oss / test_clean_removed_with_clean_inactive – test_registrar.Test
    • Age: 1
    • Duration: 0.584
    • Error Details:
      -------------------- >> begin captured stdout << ---------------------
      registry size: 2
      registry size after remove: 2

--------------------- >> end captured stdout << ----------------------

Steps errors

Expand to view the steps failures

  • Name: Make -C filebeat testsuite

    • Description: make -C filebeat testsuite

    • Duration: 29 min 13 sec

    • Start Time: 2020-06-17T10:03:06.214+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-06-17T10:05:59.701+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-17T10:59:48.769Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-06-17T10:59:48.769Z] + FILE=filebeat/build/coverage/full.cov
[2020-06-17T10:59:48.769Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-06-17T10:59:48.769Z] + FILE=heartbeat/build/coverage/full.cov
[2020-06-17T10:59:48.769Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-06-17T10:59:48.769Z] + FILE=libbeat/build/coverage/full.cov
[2020-06-17T10:59:48.769Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-06-17T10:59:48.769Z] + FILE=metricbeat/build/coverage/full.cov
[2020-06-17T10:59:48.769Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-06-17T10:59:48.769Z] + FILE=packetbeat/build/coverage/full.cov
[2020-06-17T10:59:48.769Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-06-17T10:59:48.769Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-06-17T10:59:48.769Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-06-17T10:59:48.769Z] + FILE=journalbeat/build/coverage/full.cov
[2020-06-17T10:59:48.769Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-06-17T10:59:49.353Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats
[2020-06-17T10:59:49.668Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-17T10:59:49.681Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Lint
[2020-06-17T10:59:49.774Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Winlogbeat-oss
[2020-06-17T10:59:49.884Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-06-17T10:59:49.978Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-06-17T10:59:50.071Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Dockerlogbeat
[2020-06-17T10:59:50.171Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Journalbeat-oss
[2020-06-17T10:59:50.263Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-06-17T10:59:50.352Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-06-17T10:59:50.445Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-06-17T10:59:50.538Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-06-17T10:59:50.633Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Heartbeat-oss
[2020-06-17T10:59:50.719Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-06-17T10:59:50.848Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-06-17T10:59:50.938Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-06-17T10:59:51.031Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-06-17T10:59:51.134Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-06-17T10:59:51.224Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-06-17T10:59:51.317Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Libbeat-x-pack
[2020-06-17T10:59:51.422Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Filebeat-Windows
[2020-06-17T10:59:51.505Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-06-17T10:59:51.591Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Packetbeat-oss
[2020-06-17T10:59:51.692Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-06-17T10:59:51.794Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-06-17T10:59:51.886Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Filebeat-x-pack
[2020-06-17T10:59:51.982Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-Windows
[2020-06-17T10:59:52.076Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Heartbeat-Windows
[2020-06-17T10:59:52.155Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-06-17T10:59:52.251Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Functionbeat-Windows
[2020-06-17T10:59:52.348Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Filebeat-oss
[2020-06-17T10:59:52.429Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-06-17T10:59:52.510Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Libbeat-oss
[2020-06-17T10:59:52.603Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-06-17T10:59:52.686Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-06-17T10:59:52.764Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-06-17T10:59:52.845Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-06-17T10:59:53.229Z] + cat
[2020-06-17T10:59:53.229Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-17T10:59:53.229Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-17T10:59:59.839Z] runbld>>> runbld started
[2020-06-17T10:59:59.839Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-06-17T11:00:01.753Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19233' in order of occurrence in the config (last value wins).
[2020-06-17T11:00:02.693Z] runbld>>> Debug logging enabled.
[2020-06-17T11:00:02.693Z] runbld>>> Storing result
[2020-06-17T11:00:02.960Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-06-17T11:00:02.960Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200617110002-AD9DD0DB
[2020-06-17T11:00:02.960Z] runbld>>> Adding system facts.
[2020-06-17T11:00:03.904Z] runbld>>> Adding vcs info for the latest commit:  3e7b675786246728abb9281ecc4a5dd2585e99bc
[2020-06-17T11:00:04.165Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-06-17T11:00:04.165Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-06-17T11:00:04.165Z] Processing JUnit reports with runbld...
[2020-06-17T11:00:04.165Z] + echo 'Processing JUnit reports with runbld...'
[2020-06-17T11:00:04.426Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-06-17T11:00:04.426Z] runbld>>> DURATION: 14ms
[2020-06-17T11:00:04.426Z] runbld>>> STDOUT: 40 bytes
[2020-06-17T11:00:04.426Z] runbld>>> STDERR: 49 bytes
[2020-06-17T11:00:04.426Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-06-17T11:00:04.426Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats
[2020-06-17T11:00:05.813Z] runbld>>> Storing build metadata: 
[2020-06-17T11:00:05.813Z] runbld>>> Adding test report.
[2020-06-17T11:00:05.813Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats
[2020-06-17T11:00:07.198Z] runbld>>> Found 99 test output files
[2020-06-17T11:00:07.773Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-06-17T11:00:07.773Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-06-17T11:00:07.773Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-06-17T11:00:07.773Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-06-17T11:00:07.773Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-06-17T11:00:07.773Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-06-17T11:00:08.716Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-06-17T11:00:08.716Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-06-17T11:00:08.716Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 7697 Skipped: 866
[2020-06-17T11:00:08.977Z] runbld>>> Storing result
[2020-06-17T11:00:08.977Z] runbld>>> FAILURES: 1
[2020-06-17T11:00:09.238Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-06-17T11:00:09.238Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200617110002-AD9DD0DB
[2020-06-17T11:00:09.238Z] runbld>>> Email notification disabled by environment variable.
[2020-06-17T11:00:09.238Z] runbld>>> Slack notification disabled by environment variable.
[2020-06-17T11:00:14.918Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19233
[2020-06-17T11:00:15.046Z] [INFO] getVaultSecret: Getting secrets
[2020-06-17T11:00:15.111Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-17T11:00:15.854Z] + chmod 755 generate-build-data.sh
[2020-06-17T11:00:15.855Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19233/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19233/runs/2 FAILURE 4803002
[2020-06-17T11:00:15.855Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19233/runs/2/steps/?limit=10000 -o steps-info.json
[2020-06-17T11:00:17.198Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19233/runs/2/tests/?status=FAILED -o tests-errors.json

@jsoriano
Copy link
Copy Markdown
Member Author

jenkins, run the tests please

@jsoriano jsoriano merged commit f65b237 into elastic:7.x Jun 18, 2020
@jsoriano jsoriano deleted the backport_18979_7.x branch June 18, 2020 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Platforms Label for the Integrations - Platforms team [zube]: Done

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants