Skip to content

Cherry-pick #21022 to 7.x: Sanitize event.host in Metricbeat#21030

Merged
mtojek merged 1 commit intoelastic:7.xfrom
mtojek:backport_21022_7.x
Sep 9, 2020
Merged

Cherry-pick #21022 to 7.x: Sanitize event.host in Metricbeat#21030
mtojek merged 1 commit intoelastic:7.xfrom
mtojek:backport_21022_7.x

Conversation

@mtojek
Copy link
Copy Markdown
Contributor

@mtojek mtojek commented Sep 9, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #21022 to 7.x branch. Original message:

What does this PR do?

This PR sanitizes the event.host not to leak any credentials. It fixes also a bug related to a defined but not used host parser.

Why is it important?

It's a security threat.

Checklist

  • My code follows the style guidelines of this project
  • [x I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@mtojek
Sanitize event.host in Metricbeat (elastic#21022)
189b73d 
* Sanitize event.host

* Update CHANGELOG

* Fix: enable host parser

(cherry picked from commit 3ecf7e6)
@mtojek mtojek added [zube]: In Review backport Team:Integrations Label for the Integrations team labels Sep 9, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 9, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 9, 2020

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 9, 2020
@mtojek mtojek requested review from a team and andrewkroh September 9, 2020 06:56
@mtojek mtojek merged commit 577f67d into elastic:7.x Sep 9, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 9, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21030 opened]

  • Start Time: 2020-09-09T06:56:56.432+0000

  • Duration: 61 min 42 sec

Test stats 🧪

Test Results
Failed 1
Passed 4165
Skipped 881
Total 5047

Test errors

Expand to view the tests failures

  • Name: Build and Test / Metricbeat OSS Go Integration tests / TestFetch – status

    • Age: 1
    • Duration: 135.09
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: Mage goIntegTest

    • Description: mage goIntegTest

    • Duration: 31 min 56 sec

    • Start Time: 2020-09-09T07:21:45.921+0000

    • log

Log output

Expand to view the last 100 lines of log output 

[2020-09-09T07:56:33.125Z] 	at hudson.remoting.Request$2.run(Request.java:369)
[2020-09-09T07:56:33.125Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
[2020-09-09T07:56:33.125Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-09-09T07:56:33.125Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-09-09T07:56:33.125Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-09-09T07:56:33.125Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
[2020-09-09T07:56:33.125Z] Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
[2020-09-09T07:56:33.125Z] 	at hudson.FilePath.act(FilePath.java:1072)
[2020-09-09T07:56:33.125Z] 	at hudson.FilePath.act(FilePath.java:1059)
[2020-09-09T07:56:33.125Z] 	at hudson.FilePath.validateAntFileMask(FilePath.java:2723)
[2020-09-09T07:56:33.125Z] 	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:270)
[2020-09-09T07:56:33.125Z] 	at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:91)
[2020-09-09T07:56:33.126Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-09-09T07:56:33.126Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-09-09T07:56:33.126Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-09-09T07:56:33.126Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-09-09T07:56:33.126Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-09-09T07:56:33.126Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-09-09T07:56:33.126Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-09-09T07:56:33.126Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-09-09T07:56:33.126Z] No artifacts found that match the file pattern "**/build/TEST*.out". Configuration error?
[2020-09-09T07:56:33.577Z] + python .ci/scripts/search_system_tests.py
[2020-09-09T07:56:33.609Z] [INFO] system-tests='build/metricbeat/build/system-tests'. If no empty then let's create a tarball
[2020-09-09T07:56:33.985Z] + tar --version
[2020-09-09T07:56:34.366Z] + tar --exclude=metricbeat--system-tests-linux.tgz -czf metricbeat--system-tests-linux.tgz build/metricbeat/build/system-tests
[2020-09-09T07:56:40.966Z] Archiving artifacts
[2020-09-09T07:56:56.088Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-09-09T07:56:56.129Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-09-09T07:56:56.206Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-21030-1/source/source.tgz
[2020-09-09T07:56:56.224Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-21030-1/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-21030/source.tgz
[2020-09-09T07:57:03.486Z] + tar --version
[2020-09-09T07:57:03.816Z] + tar -xpf source.tgz
[2020-09-09T07:57:14.185Z] + rm source.tgz
[2020-09-09T07:57:14.209Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats
[2020-09-09T07:57:14.255Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Lint
[2020-09-09T07:57:14.517Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-09-09T07:57:14.678Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-09-09T07:57:14.900Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-09-09T07:57:15.060Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-09-09T07:57:15.221Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-09-09T07:57:15.382Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-09-09T07:57:15.544Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-09-09T07:57:15.704Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-09-09T07:57:15.864Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-09-09T07:57:16.024Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-09-09T07:57:16.184Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-09-09T07:57:16.346Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-09-09T07:57:16.509Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-09-09T07:57:16.669Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Metricbeat-Windows
[2020-09-09T07:57:16.834Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-09-09T07:57:16.995Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-09-09T07:57:17.153Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-09-09T07:57:17.312Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-09-09T07:57:17.479Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-09-09T07:57:17.642Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-09-09T07:57:18.146Z] + cat
[2020-09-09T07:57:18.146Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-09T07:57:18.146Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-09T07:57:24.719Z] runbld>>> runbld started
[2020-09-09T07:57:24.719Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-09T07:57:25.655Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-09T07:57:25.655Z] runbld>>> Matches in the system config:
[2020-09-09T07:57:25.655Z] runbld>>> - Matched ^elastic\+beats
[2020-09-09T07:57:25.655Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-09T07:57:27.031Z] runbld>>> Debug logging enabled.
[2020-09-09T07:57:27.031Z] runbld>>> Storing result
[2020-09-09T07:57:27.031Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-09T07:57:27.031Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200909075726-F2792A8D
[2020-09-09T07:57:27.031Z] runbld>>> Adding system facts.
[2020-09-09T07:57:27.971Z] runbld>>> Adding vcs info for the latest commit:  189b73df2b9e5fb31fa362ccaf2457e4b73628ac
[2020-09-09T07:57:27.971Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-09T07:57:27.971Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-09T07:57:28.229Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-09T07:57:28.229Z] Processing JUnit reports with runbld...
[2020-09-09T07:57:28.489Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-09T07:57:28.489Z] runbld>>> DURATION: 27ms
[2020-09-09T07:57:28.489Z] runbld>>> STDOUT: 40 bytes
[2020-09-09T07:57:28.489Z] runbld>>> STDERR: 49 bytes
[2020-09-09T07:57:28.489Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-09T07:57:28.489Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21030
[2020-09-09T07:57:29.423Z] runbld>>> Storing build metadata: 
[2020-09-09T07:57:29.423Z] runbld>>> Adding test report.
[2020-09-09T07:57:29.424Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats
[2020-09-09T07:57:29.991Z] runbld>>> Found 70 test output files
[2020-09-09T07:57:31.367Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-09-09T07:57:31.368Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21030/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-09-09T07:57:31.368Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 5047 Skipped: 676
[2020-09-09T07:57:31.627Z] runbld>>> Storing result
[2020-09-09T07:57:31.627Z] runbld>>> FAILURES: 1
[2020-09-09T07:57:31.885Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-09T07:57:31.885Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200909075726-F2792A8D
[2020-09-09T07:57:32.144Z] runbld>>> Email notification disabled by environment variable.
[2020-09-09T07:57:32.144Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-09T07:57:37.648Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-21030
[2020-09-09T07:57:37.827Z] [INFO] getVaultSecret: Getting secrets
[2020-09-09T07:57:37.918Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-09T07:57:39.086Z] + chmod 755 generate-build-data.sh
[2020-09-09T07:57:39.086Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21030/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21030/runs/1 FAILURE 3642381
[2020-09-09T07:57:39.086Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21030/runs/1/steps/?limit=10000 -o steps-info.json
[2020-09-09T07:57:40.429Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21030/runs/1/tests/?status=FAILED -o tests-errors.json

@zube zube bot removed the [zube]: Done label Dec 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

@andrewkroh andrewkroh Awaiting requested review from andrewkroh

Assignees

No one assigned

Labels

backport Team:Integrations Label for the Integrations team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mtojek @elasticmachine @ChrsMark