Skip to content

Cherry-pick #19581 to 7.x: introduce journalbeat/pkg in order to provide reusable shared code#19734

Merged
urso merged 2 commits intoelastic:7.xfrom
urso:backport_19581_7.x
Jul 8, 2020
Merged

Cherry-pick #19581 to 7.x: introduce journalbeat/pkg in order to provide reusable shared code#19734
urso merged 2 commits intoelastic:7.xfrom
urso:backport_19581_7.x

Conversation

@urso
Copy link
Copy Markdown

@urso urso commented Jul 8, 2020

Cherry-pick of PR #19581 to 7.x branch. Original message:

  • Refactoring

What does this PR do?

This PR moves some functionality into journalbeat/pkg for reuse:

  • field conversion
  • adding field filters
  • low-level reader

The change is mostly moving code around with minor cleanups (e.g. by introducing more dedicated types and tests). Functionality from original read is split into: reader, matchers, and conversion

Why is it important?

Improve code sharing between journalbeat and journald input for agent. We keep the main packages/interfaces in journalbeat intact.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

  • "$ go test"
  • setup journalbeat with and without filtering + validate events in Elasticsearch.

Related issues

…lastic#19581)

This PR moves some functionality into journalbeat/pkg for reuse:
- field conversion
- adding field filters
- low-level reader

The change is mostly moving code around with minor cleanups (e.g. by introducing more dedicated types and tests). Functionality from original read is split into: reader, matchers, and conversion

(cherry picked from commit da1e55f)
@urso urso added [zube]: In Review backport Team:Services (Deprecated) Label for the former Integrations-Services team labels Jul 8, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 8, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 8, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jul 8, 2020

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19734 updated]

  • Start Time: 2020-07-08T12:14:42.484+0000

  • Duration: 123 min 41 sec

Test stats 🧪

Test Results
Failed 0
Passed 9063
Skipped 1580
Total 10643

Steps errors

Expand to view the steps failures

  • Name: Report to Codecov
    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-07-08T12:44:35.664+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-08T14:16:55.151Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-08T14:16:55.151Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-08T14:16:55.151Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-08T14:16:55.151Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-08T14:16:56.479Z] Failed in branch Filebeat oss
[2020-07-08T14:16:56.632Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats
[2020-07-08T14:16:56.954Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-08T14:16:56.970Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Lint
[2020-07-08T14:16:57.059Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-07-08T14:16:57.137Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Winlogbeat-oss
[2020-07-08T14:16:57.221Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-07-08T14:16:57.299Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-07-08T14:16:57.373Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-07-08T14:16:57.463Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-07-08T14:16:57.547Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-08T14:16:57.623Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Dockerlogbeat
[2020-07-08T14:16:57.704Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Journalbeat-oss
[2020-07-08T14:16:57.781Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-07-08T14:16:57.860Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-07-08T14:16:57.960Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-07-08T14:16:58.047Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-07-08T14:16:58.123Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-07-08T14:16:58.199Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-08T14:16:58.279Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-07-08T14:16:58.358Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-07-08T14:16:58.439Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-07-08T14:16:58.517Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Heartbeat-oss
[2020-07-08T14:16:58.595Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-08T14:16:58.671Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-08T14:16:58.747Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-08T14:16:58.821Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-08T14:16:58.900Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Libbeat-x-pack
[2020-07-08T14:16:58.977Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-07-08T14:16:59.053Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-08T14:16:59.131Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-07-08T14:16:59.210Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-07-08T14:16:59.288Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Packetbeat-oss
[2020-07-08T14:16:59.364Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-08T14:16:59.447Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-07-08T14:16:59.535Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-Windows
[2020-07-08T14:16:59.614Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-07-08T14:16:59.692Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Heartbeat-Windows
[2020-07-08T14:16:59.770Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Functionbeat-Windows
[2020-07-08T14:16:59.848Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-07-08T14:16:59.925Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-07-08T14:17:00.003Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Libbeat-oss
[2020-07-08T14:17:00.082Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-07-08T14:17:00.157Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-07-08T14:17:00.237Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-07-08T14:17:00.317Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-07-08T14:17:00.397Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-08T14:17:00.473Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-08T14:17:00.552Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Filebeat-oss
[2020-07-08T14:17:00.908Z] + cat
[2020-07-08T14:17:00.908Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-08T14:17:00.908Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-08T14:17:07.502Z] runbld>>> runbld started
[2020-07-08T14:17:07.502Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-08T14:17:08.889Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19734' in order of occurrence in the config (last value wins).
[2020-07-08T14:17:10.272Z] runbld>>> Debug logging enabled.
[2020-07-08T14:17:10.272Z] runbld>>> Storing result
[2020-07-08T14:17:10.533Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-08T14:17:10.533Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200708141710-F3E70AB3
[2020-07-08T14:17:10.533Z] runbld>>> Adding system facts.
[2020-07-08T14:17:11.477Z] runbld>>> Adding vcs info for the latest commit:  0b3ff11929e4e0f9cb5da44a150b6d64d93c038d
[2020-07-08T14:17:11.477Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-08T14:17:11.477Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-08T14:17:11.477Z] Processing JUnit reports with runbld...
[2020-07-08T14:17:11.477Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-08T14:17:12.046Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-08T14:17:12.046Z] runbld>>> DURATION: 41ms
[2020-07-08T14:17:12.046Z] runbld>>> STDOUT: 40 bytes
[2020-07-08T14:17:12.047Z] runbld>>> STDERR: 49 bytes
[2020-07-08T14:17:12.047Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-08T14:17:12.047Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats
[2020-07-08T14:17:12.986Z] runbld>>> Storing build metadata: 
[2020-07-08T14:17:12.986Z] runbld>>> Adding test report.
[2020-07-08T14:17:12.986Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats
[2020-07-08T14:17:13.935Z] runbld>>> Found 113 test output files
[2020-07-08T14:17:14.197Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-07-08T14:17:14.197Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-07-08T14:17:14.197Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-07-08T14:17:14.197Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-07-08T14:17:14.197Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-07-08T14:17:15.600Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-07-08T14:17:15.860Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-07-08T14:17:16.122Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 10493 Skipped: 1343
[2020-07-08T14:17:16.403Z] runbld>>> Storing result
[2020-07-08T14:17:16.403Z] runbld>>> FAILURES: 0
[2020-07-08T14:17:16.403Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-08T14:17:16.403Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200708141710-F3E70AB3
[2020-07-08T14:17:16.665Z] runbld>>> Email notification disabled by environment variable.
[2020-07-08T14:17:16.665Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-08T14:17:22.558Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19734
[2020-07-08T14:17:22.917Z] [INFO] getVaultSecret: Getting secrets
[2020-07-08T14:17:22.991Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-08T14:17:23.786Z] + chmod 755 generate-build-data.sh
[2020-07-08T14:17:23.786Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19734/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19734/runs/2 ABORTED 7361041
[2020-07-08T14:17:23.786Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19734/runs/2/steps/?limit=10000 -o steps-info.json
[2020-07-08T14:17:25.648Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19734/runs/2/tests/?status=FAILED -o tests-errors.json

@urso urso merged commit be0aa74 into elastic:7.x Jul 8, 2020
@urso urso deleted the backport_19581_7.x branch July 8, 2020 14:31
@zube zube bot removed the [zube]: Done label Oct 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Services (Deprecated) Label for the former Integrations-Services team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants