Skip to content

introduce journalbeat/pkg in order to provide reusable shared code#19581

Merged
urso merged 4 commits intoelastic:masterfrom
urso:journalbeat-pkg
Jul 2, 2020
Merged

introduce journalbeat/pkg in order to provide reusable shared code#19581
urso merged 4 commits intoelastic:masterfrom
urso:journalbeat-pkg

Conversation

@urso
Copy link
Copy Markdown

@urso urso commented Jul 2, 2020

  • Refactoring

What does this PR do?

This PR moves some functionality into journalbeat/pkg for reuse:

  • field conversion
  • adding field filters
  • low-level reader

The change is mostly moving code around with minor cleanups (e.g. by introducing more dedicated types and tests). Functionality from original read is split into: reader, matchers, and conversion

Why is it important?

Improve code sharing between journalbeat and journald input for agent. We keep the main packages/interfaces in journalbeat intact.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

  • "$ go test"
  • setup journalbeat with and without filtering + validate events in Elasticsearch.

Related issues

@urso urso added review needs_backport PR is waiting to be backported to other branches. Journalbeat test-plan Add this PR to be manual test plan Team:Services (Deprecated) Label for the former Integrations-Services team v7.9.0 labels Jul 2, 2020
@urso urso assigned kvch Jul 2, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 2, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jul 2, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-07-02T09:27:25.514+0000

  • Duration: 13 min 47 sec

Steps errors

Expand to view the steps failures

  • Name: Make check
    • Description: make check

    • Duration: 4 min 56 sec

    • Start Time: 2020-07-02T09:35:50.637+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-02T09:39:49.777Z] Stage "Generators" skipped due to earlier failure(s)
[2020-07-02T09:39:49.778Z] Stage "Kubernetes" skipped due to earlier failure(s)
[2020-07-02T09:39:49.890Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:49.891Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:49.893Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-07-02T09:39:49.894Z] Stage "Packetbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:49.895Z] Stage "dockerlogbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:49.895Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:49.896Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:49.897Z] Stage "Journalbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:49.898Z] Stage "Generators" skipped due to earlier failure(s)
[2020-07-02T09:39:50.616Z] Failed in branch Elastic Agent x-pack
[2020-07-02T09:39:50.616Z] Failed in branch Elastic Agent x-pack Windows
[2020-07-02T09:39:50.617Z] Failed in branch Elastic Agent Mac OS X
[2020-07-02T09:39:50.618Z] Failed in branch Filebeat oss
[2020-07-02T09:39:50.619Z] Failed in branch Filebeat x-pack
[2020-07-02T09:39:50.619Z] Failed in branch Filebeat Mac OS X
[2020-07-02T09:39:50.620Z] Failed in branch Filebeat x-pack Mac OS X
[2020-07-02T09:39:50.621Z] Failed in branch Filebeat Windows
[2020-07-02T09:39:50.621Z] Failed in branch Filebeat x-pack Windows
[2020-07-02T09:39:50.622Z] Failed in branch Auditbeat oss Linux
[2020-07-02T09:39:50.623Z] Failed in branch Auditbeat crosscompile
[2020-07-02T09:39:50.623Z] Failed in branch Auditbeat oss Mac OS X
[2020-07-02T09:39:50.624Z] Failed in branch Auditbeat oss Windows
[2020-07-02T09:39:50.625Z] Failed in branch Auditbeat x-pack
[2020-07-02T09:39:50.625Z] Failed in branch Auditbeat x-pack Mac OS X
[2020-07-02T09:39:50.626Z] Failed in branch Auditbeat x-pack Windows
[2020-07-02T09:39:50.627Z] Failed in branch Libbeat x-pack
[2020-07-02T09:39:50.627Z] Failed in branch Metricbeat OSS Unit tests
[2020-07-02T09:39:50.628Z] Failed in branch Metricbeat OSS Integration tests
[2020-07-02T09:39:50.629Z] Failed in branch Metricbeat Python integration tests
[2020-07-02T09:39:50.629Z] Failed in branch Metricbeat crosscompile
[2020-07-02T09:39:50.630Z] Failed in branch Metricbeat Mac OS X
[2020-07-02T09:39:50.631Z] Failed in branch Metricbeat x-pack Mac OS X
[2020-07-02T09:39:50.632Z] Failed in branch Metricbeat Windows
[2020-07-02T09:39:50.632Z] Failed in branch Metricbeat x-pack Windows
[2020-07-02T09:39:50.633Z] Failed in branch Winlogbeat Windows x-pack
[2020-07-02T09:39:50.634Z] Failed in branch Kubernetes
[2020-07-02T09:39:50.972Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:50.973Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:50.974Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-07-02T09:39:50.975Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:50.977Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:50.978Z] Stage "Generators" skipped due to earlier failure(s)
[2020-07-02T09:39:51.069Z] Failed in branch Packetbeat
[2020-07-02T09:39:51.069Z] Failed in branch dockerlogbeat
[2020-07-02T09:39:51.070Z] Failed in branch Journalbeat
[2020-07-02T09:39:51.338Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:51.339Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:51.340Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-07-02T09:39:51.342Z] Stage "Generators" skipped due to earlier failure(s)
[2020-07-02T09:39:51.393Z] Failed in branch Metricbeat x-pack
[2020-07-02T09:39:51.394Z] Failed in branch Winlogbeat
[2020-07-02T09:39:51.637Z] Failed in branch Heartbeat
[2020-07-02T09:39:51.638Z] Failed in branch Libbeat
[2020-07-02T09:39:51.638Z] Failed in branch Functionbeat
[2020-07-02T09:39:51.639Z] Stage "Generators" skipped due to earlier failure(s)
[2020-07-02T09:39:51.727Z] Failed in branch Generators
[2020-07-02T09:39:51.926Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19581/src/github.com/elastic/beats
[2020-07-02T09:39:52.240Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-02T09:39:52.251Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19581/src/github.com/elastic/beats/Lint
[2020-07-02T09:39:52.748Z] + cat
[2020-07-02T09:39:52.748Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-02T09:39:52.748Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-02T09:39:58.216Z] runbld>>> runbld started
[2020-07-02T09:39:58.216Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-02T09:40:00.144Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19581' in order of occurrence in the config (last value wins).
[2020-07-02T09:40:01.562Z] runbld>>> Debug logging enabled.
[2020-07-02T09:40:01.562Z] runbld>>> Storing result
[2020-07-02T09:40:01.562Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-02T09:40:01.562Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200702094001-35BF2952
[2020-07-02T09:40:01.562Z] runbld>>> Adding system facts.
[2020-07-02T09:40:02.525Z] runbld>>> Adding vcs info for the latest commit:  780c9b2cb34bc251dc52ff9301ed972e24d487a7
[2020-07-02T09:40:02.525Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-02T09:40:02.525Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-02T09:40:02.525Z] Processing JUnit reports with runbld...
[2020-07-02T09:40:02.525Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-02T09:40:02.789Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-02T09:40:02.789Z] runbld>>> DURATION: 28ms
[2020-07-02T09:40:02.789Z] runbld>>> STDOUT: 40 bytes
[2020-07-02T09:40:02.789Z] runbld>>> STDERR: 49 bytes
[2020-07-02T09:40:02.789Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-02T09:40:02.789Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19581/src/github.com/elastic/beats
[2020-07-02T09:40:03.738Z] runbld>>> Storing build metadata: 
[2020-07-02T09:40:03.738Z] runbld>>> Adding test report.
[2020-07-02T09:40:03.738Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19581/src/github.com/elastic/beats
[2020-07-02T09:40:04.304Z] runbld>>> Found 0 test output files
[2020-07-02T09:40:04.304Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 0 Skipped: 0
[2020-07-02T09:40:04.568Z] runbld>>> Storing result
[2020-07-02T09:40:04.838Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-02T09:40:04.838Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200702094001-35BF2952
[2020-07-02T09:40:04.838Z] runbld>>> Email notification disabled by environment variable.
[2020-07-02T09:40:04.838Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-02T09:40:11.086Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19581
[2020-07-02T09:40:11.245Z] [INFO] getVaultSecret: Getting secrets
[2020-07-02T09:40:11.391Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-02T09:40:12.294Z] + chmod 755 generate-build-data.sh
[2020-07-02T09:40:12.294Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19581/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19581/runs/3 FAILURE 766515
[2020-07-02T09:40:12.294Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19581/runs/3/steps/?limit=10000 -o steps-info.json
[2020-07-02T09:40:12.844Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19581/runs/3/tests/?status=FAILED -o tests-errors.json

Copy link
Copy Markdown
Contributor

@kvch kvch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You made the code so much more readable! Thank you!

@urso urso merged commit da1e55f into elastic:master Jul 2, 2020
@urso urso deleted the journalbeat-pkg branch July 2, 2020 12:18
@zube zube bot added [zube]: Done and removed [zube]: Inbox labels Jul 2, 2020
v1v added a commit to v1v/beats that referenced this pull request Jul 3, 2020
…ne-beats

* upstream/master: (35 commits)
  [ci] fix env variable name for xpack filebeats (elastic#19617)
  Cache error responses for cloudfoundry apps metadata (elastic#19181)
  ci: user fixed type of agent (elastic#19625)
  Input v2 cursor testing (elastic#19573)
  Update Jenkinsfile to not inspect removed vendor (elastic#19610)
  Fix ordering and duplicate configs on autodiscover (elastic#19317)
  Prepare input/file for changes in the registrar (elastic#19516)
  Cursor input and manager implementation (elastic#19571)
  [Filebeat] Fix tls mapping in suricata module (elastic#19494)
  [Ingest Manager] Make Agent beta and Constraints experimental (elastic#19586)
  Accept prefix as metric_types for stackdriver metricset in GCP (elastic#19345)
  Implement memlog store operations (elastic#19533)
  introduce journalbeat/pkg in order to provide reusable shared code (elastic#19581)
  Add descriptions to HAProxy fields in Metricbeat (elastic#19561)
  ci: apm-server-update trigered only on upstream, comments, and manual triggered (elastic#19590)
  ci: enable upstream triggering on the packaging job (elastic#19589)
  ci: some jjbb improvements (elastic#19588)
  [MetricBeat] set tags correctly if the dimension value is ARN (elastic#19433)
  [Filebeat] Add default_fields: false to fields.yml in aws module (elastic#19568)
  Add publisher implementation for stateful inputs (elastic#19530)
  ...
urso pushed a commit to urso/beats that referenced this pull request Jul 8, 2020
…lastic#19581)

This PR moves some functionality into journalbeat/pkg for reuse:
- field conversion
- adding field filters
- low-level reader

The change is mostly moving code around with minor cleanups (e.g. by introducing more dedicated types and tests). Functionality from original read is split into: reader, matchers, and conversion

(cherry picked from commit da1e55f)
@urso urso removed the needs_backport PR is waiting to be backported to other branches. label Jul 8, 2020
urso pushed a commit that referenced this pull request Jul 8, 2020
@andresrc andresrc added test-plan-added This PR has been added to the test plan and removed [zube]: Done labels Jul 14, 2020
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
…lastic#19581)

This PR moves some functionality into journalbeat/pkg for reuse:
- field conversion
- adding field filters
- low-level reader

The change is mostly moving code around with minor cleanups (e.g. by introducing more dedicated types and tests). Functionality from original read is split into: reader, matchers, and conversion
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Journalbeat review Team:Services (Deprecated) Label for the former Integrations-Services team test-plan Add this PR to be manual test plan test-plan-added This PR has been added to the test plan v7.9.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants