Skip to content

Cherry-pick #19974 to 7.x: Remove unnecessary restarts of metricsets while using Node autodiscover#20310

Merged
ChrsMark merged 1 commit intoelastic:7.xfrom
ChrsMark:backport_19974_7.x
Jul 29, 2020
Merged

Cherry-pick #19974 to 7.x: Remove unnecessary restarts of metricsets while using Node autodiscover#20310
ChrsMark merged 1 commit intoelastic:7.xfrom
ChrsMark:backport_19974_7.x

Conversation

@ChrsMark
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark commented Jul 29, 2020

Cherry-pick of PR #19974 to 7.x branch. Original message:

Bug

What does this PR do?

Everytime the kubelet does a check for node health it posts an update to the node status. Each change even if overall status doesnt change then it causes the metricset to restart. This PR adds a watcher option to register a func that checks in detail if the update is valid or not.

Why is it important?

This makes sure that the kubelet status updates dont prompt a restart unless the overall status changes.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      in_cluster: false
      kube_config: ${HOME}/.kube/config
      scope: node
      node: ${NODENAME}
      resource: node
      templates:
        - config:
            - module: http
              period: 1m
              metricsets: ["json"]
              namespace: beats
              hosts: "${data.host}:5002"
              path: "/stats"

Related issues

@ChrsMark ChrsMark added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Jul 29, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 29, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 29, 2020
@ChrsMark ChrsMark self-assigned this Jul 29, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20310 opened]

  • Start Time: 2020-07-29T12:31:37.138+0000

  • Duration: 123 min 4 sec

Test stats 🧪

Test Results
Failed 18
Passed 17603
Skipped 1831
Total 19452

Test errors

Expand to view the tests failures

  • Name: Build and Test / Libbeat / Libbeat oss / TestDockerStart – docker

    • Age: 1
    • Duration: 12.12
    • Error Details: Failed
  • Name: Build and Test / Filebeat Mac OS X / test_debug_reader – test_harvester.Test

    • Age: 1
    • Duration: 5.277
    • Error Details: Timeout waiting for 'cond' to be true. Waited 5 seconds.
  • Name: Build and Test / Filebeat Mac OS X / test_not_ignore_old_files – test_input.Test

    • Age: 1
    • Duration: 10.231
    • Error Details: Timeout waiting for 'cond' to be true. Waited 10 seconds.
  • Name: Build and Test / Filebeat Mac OS X / test_shutdown – test_shutdown.Test

    • Age: 1
    • Duration: 1.894
    • Error Details: Expected exit code to be 0, but it was -15
  • Name: Build and Test / Metricbeat OSS Unit tests / test_process – test_system.Test

    • Age: 1
    • Duration: 0.987
    • Error Details: False is not true : fd not found in any process events
  • Name: Build and Test / Auditbeat x-pack / test_connected_udp_ipv4 – test_system_socket.Test

    • Age: 1
    • Duration: 60.112
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_connected_udp_ipv6 – test_system_socket.Test

    • Age: 1
    • Duration: 60.102
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_dns_enrichment – test_system_socket.Test

    • Age: 1
    • Duration: 60.115
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_dns_long_request – test_system_socket.Test

    • Age: 1
    • Duration: 60.113
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_dns_udp_ipv6 – test_system_socket.Test

    • Age: 1
    • Duration: 60.122
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_dns_unidirectional_udp – test_system_socket.Test

    • Age: 1
    • Duration: 60.117
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_multi_udp_upv4 – test_system_socket.Test

    • Age: 1
    • Duration: 60.107
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_no_dns_enrichment – test_system_socket.Test

    • Age: 1
    • Duration: 60.111
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_tcp_ipv4 – test_system_socket.Test

    • Age: 1
    • Duration: 60.121
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_tcp_ipv6_disabled – test_system_socket.Test

    • Age: 1
    • Duration: 60.121
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_udp_ipv4 – test_system_socket.Test

    • Age: 1
    • Duration: 60.118
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_udp_ipv6 – test_system_socket.Test

    • Age: 1
    • Duration: 60.114
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_udp_ipv6_disabled – test_system_socket.Test

    • Age: 1
    • Duration: 60.109
    • Error Details: Expected exit code to be 0, but it was 1

Steps errors

Expand to view the steps failures

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Duration: 9 min 51 sec

    • Start Time: 2020-07-29T13:21:15.044+0000

    • log

  • Name: Recursively delete the current directory from the workspace

    • Description: script returned exit code 1

    • Duration: 0 min 14 sec

    • Start Time: 2020-07-29T13:30:30.999+0000

    • log

  • Name: Mage update build test

    • Description: mage update build test

    • Duration: 22 min 12 sec

    • Start Time: 2020-07-29T12:57:39.069+0000

    • log

  • Name: Mage build test

    • Description: mage build test

    • Duration: 25 min 53 sec

    • Start Time: 2020-07-29T12:57:41.909+0000

    • log

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Duration: 9 min 5 sec

    • Start Time: 2020-07-29T12:57:43.146+0000

    • log

  • Name: Make test

    • Description: make -C generator/_templates/beat test

    • Duration: 7 min 59 sec

    • Start Time: 2020-07-29T14:26:46.653+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-29T14:34:02.077Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-29T14:34:02.077Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-07-29T14:34:02.077Z] 
[2020-07-29T14:34:02.524Z] Failed in branch Generators
[2020-07-29T14:34:04.465Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats
[2020-07-29T14:34:04.817Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-29T14:34:04.841Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Lint
[2020-07-29T14:34:05.003Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Winlogbeat-oss
[2020-07-29T14:34:05.159Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-07-29T14:34:05.321Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-07-29T14:34:05.480Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Dockerlogbeat
[2020-07-29T14:34:05.641Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Journalbeat-oss
[2020-07-29T14:34:05.800Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-07-29T14:34:05.961Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-07-29T14:34:06.118Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Packetbeat-Linux
[2020-07-29T14:34:06.275Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-07-29T14:34:06.432Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-07-29T14:34:06.590Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-29T14:34:06.754Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Heartbeat-oss
[2020-07-29T14:34:06.927Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-07-29T14:34:07.105Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-07-29T14:34:07.284Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-29T14:34:07.443Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-07-29T14:34:07.599Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-29T14:34:07.764Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-29T14:34:07.934Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Libbeat-x-pack
[2020-07-29T14:34:08.093Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-Windows
[2020-07-29T14:34:08.262Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-29T14:34:08.415Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-07-29T14:34:08.570Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-07-29T14:34:08.735Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-29T14:34:08.890Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Libbeat-oss
[2020-07-29T14:34:09.044Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-29T14:34:09.200Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-07-29T14:34:09.350Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Filebeat-oss
[2020-07-29T14:34:09.505Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-29T14:34:09.672Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-07-29T14:34:09.829Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-07-29T14:34:09.985Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-07-29T14:34:10.141Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-07-29T14:34:10.298Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-29T14:34:10.453Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-07-29T14:34:10.608Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-07-29T14:34:10.765Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-07-29T14:34:10.918Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Packetbeat-Mac-OS-X
[2020-07-29T14:34:11.069Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-07-29T14:34:11.224Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-29T14:34:11.402Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Heartbeat-Windows
[2020-07-29T14:34:11.558Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Functionbeat-Windows
[2020-07-29T14:34:11.719Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Packetbeat-Windows
[2020-07-29T14:34:11.877Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-07-29T14:34:12.031Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-07-29T14:34:12.480Z] + cat
[2020-07-29T14:34:12.480Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-29T14:34:12.481Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-29T14:34:19.071Z] runbld>>> runbld started
[2020-07-29T14:34:19.071Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-29T14:34:20.986Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-20310' in order of occurrence in the config (last value wins).
[2020-07-29T14:34:22.383Z] runbld>>> Debug logging enabled.
[2020-07-29T14:34:22.383Z] runbld>>> Storing result
[2020-07-29T14:34:22.383Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-29T14:34:22.383Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200729143422-9AF7374E
[2020-07-29T14:34:22.383Z] runbld>>> Adding system facts.
[2020-07-29T14:34:23.774Z] runbld>>> Adding vcs info for the latest commit:  55b6e23ac7aa19749eaba10a07cfa44f7c1bd300
[2020-07-29T14:34:23.774Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-29T14:34:23.774Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-29T14:34:23.774Z] Processing JUnit reports with runbld...
[2020-07-29T14:34:23.774Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-29T14:34:24.036Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-29T14:34:24.036Z] runbld>>> DURATION: 31ms
[2020-07-29T14:34:24.036Z] runbld>>> STDOUT: 40 bytes
[2020-07-29T14:34:24.036Z] runbld>>> STDERR: 49 bytes
[2020-07-29T14:34:24.036Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-29T14:34:24.036Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats
[2020-07-29T14:34:25.428Z] runbld>>> Storing build metadata: 
[2020-07-29T14:34:25.428Z] runbld>>> Adding test report.
[2020-07-29T14:34:25.428Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats
[2020-07-29T14:34:26.375Z] runbld>>> Found 135 test output files
[2020-07-29T14:34:27.323Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-07-29T14:34:27.323Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-07-29T14:34:27.323Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-07-29T14:34:27.323Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-07-29T14:34:27.323Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-07-29T14:34:27.584Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-07-29T14:34:27.584Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20310/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-07-29T14:34:30.139Z] runbld>>> Test output logs contained: Errors: 2 Failures: 16 Tests: 19306 Skipped: 1554
[2020-07-29T14:34:30.139Z] runbld>>> Storing result
[2020-07-29T14:34:30.139Z] runbld>>> FAILURES: 18
[2020-07-29T14:34:34.359Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-29T14:34:34.359Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200729143422-9AF7374E
[2020-07-29T14:34:34.359Z] runbld>>> Email notification disabled by environment variable.
[2020-07-29T14:34:34.359Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-29T14:34:39.985Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20310
[2020-07-29T14:34:40.200Z] [INFO] getVaultSecret: Getting secrets
[2020-07-29T14:34:40.306Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-29T14:34:41.406Z] + chmod 755 generate-build-data.sh
[2020-07-29T14:34:41.407Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20310/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20310/runs/1 FAILURE 7383999
[2020-07-29T14:34:41.407Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20310/runs/1/steps/?limit=10000 -o steps-info.json
[2020-07-29T14:34:44.635Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20310/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-07-29T14:34:44.635Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20310/runs/1/log/ -o pipeline-log.txt

@ChrsMark ChrsMark merged commit eef3581 into elastic:7.x Jul 29, 2020
@zube zube bot removed the [zube]: Done label Oct 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Platforms Label for the Integrations - Platforms team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants