Skip to content

Cherry-pick #21197 to 7.x: Fix libbeat.output.*.bytes metrics for ES#21210

Merged
kvch merged 1 commit intoelastic:7.xfrom
kvch:backport_21197_7.x
Sep 22, 2020
Merged

Cherry-pick #21197 to 7.x: Fix libbeat.output.*.bytes metrics for ES#21210
kvch merged 1 commit intoelastic:7.xfrom
kvch:backport_21197_7.x

Conversation

@kvch
Copy link
Copy Markdown
Contributor

@kvch kvch commented Sep 22, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #21197 to 7.x branch. Original message:

What does this PR do?

This PR passes a missing parameter to the constructor elasticsearch.Client.

Why is it important?

The missing parameter prevented Beats from reporting the metrics libbeat.output.write.bytes and libbeat.output.read.bytes in case of Elasticsearch output.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Example configuration:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - test.log

http.enabled: true
output.elasticsearch:
  hosts: ["localhost:9200"]

Read events from test.log. Then visit http://localhost:5066/stats?pretty and check that the aforementioned counters are greater than zero.

Related issues

Closes #20752

@kvch
Fix libbeat.output.*.bytes metrics of Elasticsearch output (elastic#2…
d929457 
…1197)

## What does this PR do?

This PR passes a missing parameter to the constructor `elasticsearch.Client`.

## Why is it important?

The missing parameter prevented Beats from reporting the metrics `libbeat.output.write.bytes` and `libbeat.output.read.bytes` in case of Elasticsearch output.

## Related issues

Closes elastic#20752

(cherry picked from commit 78dc0cf)
@kvch kvch added [zube]: In Review backport Team:Services (Deprecated) Label for the former Integrations-Services team labels Sep 22, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 22, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 22, 2020

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 22, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 22, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21210 opened]

  • Start Time: 2020-09-22T08:26:24.348+0000

  • Duration: 75 min 35 sec

Test stats 🧪

Test Results
Failed 3
Passed 18134
Skipped 1780
Total 19917

Test errors

Expand to view the tests failures

  • Name: Build and Test / Libbeat / Libbeat oss / TestOutputReload – pipeline

    • Age: 1
    • Duration: 35.19
    • Error Details: Failed

  • Name: Build and Test / Libbeat / Libbeat oss / TestOutputReload/network_client – pipeline

    • Age: 1
    • Duration: 18.63
    • Error Details: Failed

  • Name: Build and Test / Filebeat oss / test_default_settings – filebeat.tests.system.test_autodiscover.TestAutodiscover

    • Age: 1
    • Duration: 90.004
    • Error Details: Failed: Timeout >90.0s

Steps errors

Expand to view the steps failures

  • Name: Mage build test

    • Description: mage build test

    • Duration: 33 min 24 sec

    • Start Time: 2020-09-22T08:51:56.682+0000

    • log

  • Name: Mage build test

    • Description: mage build test

    • Duration: 5 min 50 sec

    • Start Time: 2020-09-22T08:51:58.956+0000

    • log

  • Name: Make test

    • Description: make -C generator/_templates/beat test

    • Duration: 8 min 41 sec

    • Start Time: 2020-09-22T09:12:04.435+0000

    • log

  • Name: Recursively delete the current directory from the workspace

    • Description: script returned exit code 2

    • Duration: 2 min 31 sec

    • Start Time: 2020-09-22T09:20:03.316+0000

    • log

Log output

Expand to view the last 100 lines of log output 

[2020-09-22T09:40:00.156Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-21210-1/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-21210/source.tgz
[2020-09-22T09:40:14.221Z] + tar --version
[2020-09-22T09:40:14.550Z] + tar -xpf source.tgz
[2020-09-22T09:40:27.162Z] + rm source.tgz
[2020-09-22T09:40:27.189Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats
[2020-09-22T09:40:27.241Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Lint
[2020-09-22T09:40:27.407Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Winlogbeat-oss
[2020-09-22T09:40:27.569Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-09-22T09:40:27.736Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-09-22T09:40:27.903Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/dockerlogbeat
[2020-09-22T09:40:28.067Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-09-22T09:40:28.230Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Journalbeat
[2020-09-22T09:40:28.391Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-09-22T09:40:28.560Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Libbeat-oss
[2020-09-22T09:40:28.722Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-09-22T09:40:28.885Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Packetbeat-Linux
[2020-09-22T09:40:29.046Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-09-22T09:40:29.208Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-09-22T09:40:29.375Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-09-22T09:40:29.536Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-09-22T09:40:29.703Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-09-22T09:40:29.867Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-09-22T09:40:30.028Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-09-22T09:40:30.191Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Heartbeat-Windows
[2020-09-22T09:40:30.357Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Heartbeat-oss
[2020-09-22T09:40:30.519Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-09-22T09:40:30.679Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-09-22T09:40:30.840Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-09-22T09:40:31.006Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-09-22T09:40:31.165Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Filebeat-Windows
[2020-09-22T09:40:31.325Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-09-22T09:40:31.487Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-09-22T09:40:31.646Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-09-22T09:40:31.814Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Packetbeat-Windows
[2020-09-22T09:40:31.986Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-09-22T09:40:32.145Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Packetbeat-Mac-OS-X
[2020-09-22T09:40:32.309Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-09-22T09:40:32.471Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Libbeat-x-pack
[2020-09-22T09:40:32.633Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-09-22T09:40:32.796Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-09-22T09:40:33.049Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-09-22T09:40:33.215Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Functionbeat-Windows
[2020-09-22T09:40:33.377Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-Windows
[2020-09-22T09:40:33.540Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-09-22T09:40:33.706Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-09-22T09:40:33.870Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-09-22T09:40:34.037Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-09-22T09:40:34.198Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Filebeat-oss
[2020-09-22T09:40:34.359Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Filebeat-x-pack
[2020-09-22T09:40:34.526Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-09-22T09:40:34.704Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-09-22T09:40:35.230Z] + cat
[2020-09-22T09:40:35.230Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-22T09:40:35.230Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-22T09:40:41.846Z] runbld>>> runbld started
[2020-09-22T09:40:41.846Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-22T09:40:43.234Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-22T09:40:43.234Z] runbld>>> Matches in the system config:
[2020-09-22T09:40:43.234Z] runbld>>> - Matched ^elastic\+beats
[2020-09-22T09:40:43.234Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-22T09:40:44.631Z] runbld>>> Debug logging enabled.
[2020-09-22T09:40:44.631Z] runbld>>> Storing result
[2020-09-22T09:40:44.893Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-22T09:40:44.893Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200922094044-D637079A
[2020-09-22T09:40:44.893Z] runbld>>> Adding system facts.
[2020-09-22T09:40:45.837Z] runbld>>> Adding vcs info for the latest commit:  d929457018e330c4ddf7476aedce42aebb01b0d5
[2020-09-22T09:40:45.837Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-22T09:40:46.099Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-22T09:40:46.099Z] Processing JUnit reports with runbld...
[2020-09-22T09:40:46.099Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-22T09:40:46.361Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-22T09:40:46.361Z] runbld>>> DURATION: 44ms
[2020-09-22T09:40:46.361Z] runbld>>> STDOUT: 40 bytes
[2020-09-22T09:40:46.361Z] runbld>>> STDERR: 49 bytes
[2020-09-22T09:40:46.361Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-22T09:40:46.361Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21210
[2020-09-22T09:40:47.751Z] runbld>>> Storing build metadata: 
[2020-09-22T09:40:47.751Z] runbld>>> Adding test report.
[2020-09-22T09:40:47.751Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats
[2020-09-22T09:40:48.338Z] runbld>>> Found 137 test output files
[2020-09-22T09:40:48.922Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-09-22T09:40:48.922Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-09-22T09:40:48.922Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-09-22T09:40:48.922Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-09-22T09:40:48.922Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-09-22T09:40:50.322Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-09-22T09:40:50.322Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21210/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-09-22T09:40:51.719Z] runbld>>> Test output logs contained: Errors: 0 Failures: 3 Tests: 19762 Skipped: 1519
[2020-09-22T09:40:51.981Z] runbld>>> Storing result
[2020-09-22T09:40:51.981Z] runbld>>> FAILURES: 3
[2020-09-22T09:40:52.925Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-22T09:40:52.925Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200922094044-D637079A
[2020-09-22T09:40:52.925Z] runbld>>> Email notification disabled by environment variable.
[2020-09-22T09:40:52.925Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-22T09:40:58.508Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-21210
[2020-09-22T09:40:58.689Z] [INFO] getVaultSecret: Getting secrets
[2020-09-22T09:40:58.784Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-22T09:40:59.889Z] + chmod 755 generate-build-data.sh
[2020-09-22T09:40:59.890Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21210/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21210/runs/1 FAILURE 4475270
[2020-09-22T09:40:59.890Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21210/runs/1/steps/?limit=10000 -o steps-info.json

@kvch kvch merged commit a8017d9 into elastic:7.x Sep 22, 2020
@zube zube bot removed the [zube]: Done label Dec 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@sayden sayden sayden approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport Team:Services (Deprecated) Label for the former Integrations-Services team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kvch @elasticmachine @sayden