Skip to content

Heartbeat: move configuration of common input setting to the runner factory#20610

Merged
urso merged 7 commits intoelastic:masterfrom
urso:missig-index-setting
Aug 20, 2020
Merged

Heartbeat: move configuration of common input setting to the runner factory#20610
urso merged 7 commits intoelastic:masterfrom
urso:missig-index-setting

Conversation

@urso
Copy link
Copy Markdown

@urso urso commented Aug 14, 2020

  • Bug

What does this PR do?

This changes moves the configuration of common input settings like
processors, fields, tags to the runner factory (similar to filebeat).

The settings index, pipeline, and dataset have been added as generic
settings to all monitors.

Why is it important?

Add missing settings index and pipeline to heartbeat monitors.

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Add custom index/pipeline setting and send event to console output. If index/pipeline are configured, they can be read from the @metadata object.

Related issues

@urso urso added bug Team:obs-ds-hosted-services Label for the Observability Hosted Services team labels Aug 14, 2020
@urso urso requested a review from andrewvc August 14, 2020 16:09
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 14, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 14, 2020 •
edited by jenkins-beats-ci bot
Loading

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20610 updated]

  • Start Time: 2020-08-17T16:50:57.522+0000

  • Duration: 33 min 4 sec

Test stats 🧪

Test Results
Failed 3
Passed 906
Skipped 15
Total 924

Test errors

Expand to view the tests failures

  • Name: Build and Test / Heartbeat Mac OS X / test_dataset – heartbeat.tests.system.test_base.Test

    • Age: 1
    • Duration: 0.312
    • Error Details: KeyError: 'event.dataset'

  • Name: Build and Test / Heartbeat oss / test_dataset – heartbeat.tests.system.test_base.Test

    • Age: 1
    • Duration: 0.295
    • Error Details: KeyError: 'event.dataset'

  • Name: Build and Test / Heartbeat Windows / test_dataset – heartbeat.tests.system.test_base.Test

    • Age: 1
    • Duration: 1.322
    • Error Details: KeyError: 'event.dataset'

Steps errors

Expand to view the steps failures

  • Name: Mage build test

    • Description: mage build test

    • Duration: 3 min 40 sec

    • Start Time: 2020-08-17T17:15:37.811+0000

    • log

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Duration: 2 min 15 sec

    • Start Time: 2020-08-17T17:13:53.076+0000

    • log

  • Name: Recursively delete the current directory from the workspace

    • Description: script returned exit code 1

    • Duration: 0 min 9 sec

    • Start Time: 2020-08-17T17:16:16.473+0000

    • log

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Duration: 5 min 40 sec

    • Start Time: 2020-08-17T17:18:25.854+0000

    • log

Log output

Expand to view the last 100 lines of log output 

[2020-08-17T17:23:05.898Z]     warnings.warn('assertDictContainsSubset is deprecated',
[2020-08-17T17:23:05.898Z] 
[2020-08-17T17:23:05.898Z] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[2020-08-17T17:23:05.898Z] - generated xml file: C:\Users\jenkins\workspace\Beats_beats_PR-20610\src\github.com\elastic\beats\heartbeat\build\TEST-python-unit.xml -
[2020-08-17T17:23:05.898Z] ============================ slowest 20 durations =============================
[2020-08-17T17:23:05.898Z] 6.63s call     heartbeat/tests/system/test_reload.py::Test::test_config_reload
[2020-08-17T17:23:05.898Z] 5.06s call     heartbeat/tests/system/test_icmp.py::Test::test_base
[2020-08-17T17:23:05.898Z] 3.63s call     heartbeat/tests/system/test_reload.py::Test::test_config_remove
[2020-08-17T17:23:05.898Z] 3.51s call     heartbeat/tests/system/test_monitor.py::Test::test_tcp_1
[2020-08-17T17:23:05.898Z] 2.63s call     heartbeat/tests/system/test_reload.py::Test::test_config_add
[2020-08-17T17:23:05.898Z] 2.13s call     heartbeat/tests/system/test_monitor.py::Test::test_http_delayed
[2020-08-17T17:23:05.898Z] 1.33s call     heartbeat/tests/system/test_base.py::Test::test_fields_under_root
[2020-08-17T17:23:05.898Z] 1.32s call     heartbeat/tests/system/test_base.py::Test::test_fields_not_under_root
[2020-08-17T17:23:05.898Z] 1.32s call     heartbeat/tests/system/test_base.py::Test::test_dataset
[2020-08-17T17:23:05.898Z] 1.31s call     heartbeat/tests/system/test_base.py::Test::test_host_fields_not_present
[2020-08-17T17:23:05.898Z] 0.63s call     heartbeat/tests/system/test_monitor.py::Test::test_http_0_200
[2020-08-17T17:23:05.898Z] 0.63s call     heartbeat/tests/system/test_monitor.py::Test::test_json_simple_comparisons_0__foo_bar_
[2020-08-17T17:23:05.898Z] 0.63s call     heartbeat/tests/system/test_monitor.py::Test::test_http_json_0_up
[2020-08-17T17:23:05.898Z] 0.63s call     heartbeat/tests/system/test_monitor.py::Test::test_http_with_hosts_config_1_404
[2020-08-17T17:23:05.898Z] 0.63s call     heartbeat/tests/system/test_monitor.py::Test::test_tcp_0
[2020-08-17T17:23:05.898Z] 0.62s call     heartbeat/tests/system/test_monitor.py::Test::test_http_json_1_down
[2020-08-17T17:23:05.898Z] 0.62s call     heartbeat/tests/system/test_monitor.py::Test::test_json_simple_comparisons_2__foo_3_
[2020-08-17T17:23:05.898Z] 0.62s call     heartbeat/tests/system/test_monitor.py::Test::test_http_with_hosts_config_0_200
[2020-08-17T17:23:05.898Z] 0.62s call     heartbeat/tests/system/test_monitor.py::Test::test_http_1_404
[2020-08-17T17:23:05.898Z] 0.62s call     heartbeat/tests/system/test_monitor.py::Test::test_http_json_2_down
[2020-08-17T17:23:05.899Z] =========================== short test summary info ===========================
[2020-08-17T17:23:05.899Z] FAILED tests\system\test_base.py::Test::test_dataset - KeyError: 'event.dataset'
[2020-08-17T17:23:05.899Z] ============ 1 failed, 20 passed, 9 skipped, 3 warnings in 38.26s =============
[2020-08-17T17:23:05.899Z] >> python test: Unit Testing Complete
[2020-08-17T17:23:05.899Z] Error: running "C:\Users\jenkins\workspace\Beats_beats_PR-20610\src\github.com\elastic\beats\build\ve\windows\Scripts\pytest.exe --timeout=90 --durations=20 --junit-xml=build/TEST-python-unit.xml tests\system\test_autodiscovery.py tests\system\test_base.py tests\system\test_icmp.py tests\system\test_monitor.py tests\system\test_reload.py tests\system\test_telemetry.py" failed with exit code 1
[2020-08-17T17:23:06.273Z] 
[2020-08-17T17:23:06.274Z] C:\Users\jenkins\workspace\Beats_beats_PR-20610\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2020-08-17T17:23:07.207Z] Copy .\heartbeat\build into build\heartbeat\build
[2020-08-17T17:23:07.216Z] Running in C:\Users\jenkins\workspace\Beats_beats_PR-20610\src\github.com\elastic\beats\build
[2020-08-17T17:23:07.239Z] Recording test results
[2020-08-17T17:23:08.986Z] Stashed 2 file(s)
[2020-08-17T17:23:08.996Z] Archiving artifacts
[2020-08-17T17:23:09.675Z] [INFO] system-tests='build\heartbeat\build\system-tests'. If no empty then let's create a tarball
[2020-08-17T17:23:10.010Z] 
[2020-08-17T17:23:10.010Z] C:\Users\jenkins\workspace\Beats_beats_PR-20610\src\github.com\elastic\beats>tar --version  1>NUL 
[2020-08-17T17:23:10.345Z] 
[2020-08-17T17:23:10.345Z] C:\Users\jenkins\workspace\Beats_beats_PR-20610\src\github.com\elastic\beats>tar --exclude=heartbeat--system-tests-windows.tgz -czf heartbeat--system-tests-windows.tgz build\heartbeat\build\system-tests 
[2020-08-17T17:23:10.651Z] Archiving artifacts
[2020-08-17T17:23:12.073Z] Failed in branch Heartbeat Windows
[2020-08-17T17:23:19.348Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-08-17T17:23:19.371Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-08-17T17:23:19.455Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-20610-5/source/source.tgz
[2020-08-17T17:23:19.473Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-20610-5/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-20610/source.tgz
[2020-08-17T17:23:29.677Z] + tar --version
[2020-08-17T17:23:29.970Z] + tar -xpf source.tgz
[2020-08-17T17:23:40.387Z] + rm source.tgz
[2020-08-17T17:23:40.398Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20610/src/github.com/elastic/beats
[2020-08-17T17:23:40.418Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20610/src/github.com/elastic/beats/Lint
[2020-08-17T17:23:40.508Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20610/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-08-17T17:23:40.585Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20610/src/github.com/elastic/beats/Heartbeat-oss
[2020-08-17T17:23:40.656Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20610/src/github.com/elastic/beats/Heartbeat-Windows
[2020-08-17T17:23:41.048Z] + cat
[2020-08-17T17:23:41.048Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-08-17T17:23:41.048Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-08-17T17:23:47.638Z] runbld>>> runbld started
[2020-08-17T17:23:47.638Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-08-17T17:23:49.032Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-08-17T17:23:49.032Z] runbld>>> Matches in the system config:
[2020-08-17T17:23:49.032Z] runbld>>> - Matched ^elastic\+beats
[2020-08-17T17:23:49.032Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-08-17T17:23:50.412Z] runbld>>> Debug logging enabled.
[2020-08-17T17:23:50.412Z] runbld>>> Storing result
[2020-08-17T17:23:50.412Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-08-17T17:23:50.412Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200817172350-91EFBA48
[2020-08-17T17:23:50.412Z] runbld>>> Adding system facts.
[2020-08-17T17:23:51.352Z] runbld>>> Adding vcs info for the latest commit:  ae117799885dbab83dfe686ce7bb97232977a8d6
[2020-08-17T17:23:51.352Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-08-17T17:23:51.352Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-08-17T17:23:51.352Z] Processing JUnit reports with runbld...
[2020-08-17T17:23:51.352Z] + echo 'Processing JUnit reports with runbld...'
[2020-08-17T17:23:51.613Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-08-17T17:23:51.613Z] runbld>>> DURATION: 28ms
[2020-08-17T17:23:51.613Z] runbld>>> STDOUT: 40 bytes
[2020-08-17T17:23:51.613Z] runbld>>> STDERR: 49 bytes
[2020-08-17T17:23:51.613Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-08-17T17:23:51.613Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-20610
[2020-08-17T17:23:52.553Z] runbld>>> Storing build metadata: 
[2020-08-17T17:23:52.553Z] runbld>>> Adding test report.
[2020-08-17T17:23:52.553Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-20610/src/github.com/elastic/beats
[2020-08-17T17:23:53.492Z] runbld>>> Found 6 test output files
[2020-08-17T17:23:53.753Z] runbld>>> Test output logs contained: Errors: 0 Failures: 3 Tests: 924 Skipped: 15
[2020-08-17T17:23:53.753Z] runbld>>> Storing result
[2020-08-17T17:23:53.753Z] runbld>>> FAILURES: 3
[2020-08-17T17:23:54.691Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-08-17T17:23:54.691Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200817172350-91EFBA48
[2020-08-17T17:23:54.691Z] runbld>>> Email notification disabled by environment variable.
[2020-08-17T17:23:54.691Z] runbld>>> Slack notification disabled by environment variable.
[2020-08-17T17:24:00.481Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20610
[2020-08-17T17:24:00.688Z] [INFO] getVaultSecret: Getting secrets
[2020-08-17T17:24:00.770Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-08-17T17:24:01.566Z] + chmod 755 generate-build-data.sh
[2020-08-17T17:24:01.566Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20610/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20610/runs/5 FAILURE 1983785
[2020-08-17T17:24:01.566Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20610/runs/5/steps/?limit=10000 -o steps-info.json
[2020-08-17T17:24:02.477Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20610/runs/5/tests/?status=FAILED -o tests-errors.json
[2020-08-17T17:24:02.477Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20610/runs/5/log/ -o pipeline-log.txt

@urso urso added needs_backport PR is waiting to be backported to other branches. review labels Aug 14, 2020
@urso urso marked this pull request as ready for review August 14, 2020 16:29
@urso urso requested a review from a team as a code owner August 14, 2020 16:29
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 14, 2020

Pinging @elastic/uptime (Team:Uptime)

andrewvc
andrewvc approved these changes Aug 14, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@andrewvc andrewvc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM aside from the failing lint

@urso urso force-pushed the missig-index-setting branch from a360f8d to ae11779 Compare August 17, 2020 16:49
urso added 7 commits August 20, 2020 17:19
Heartbeat: move configuration of common input setting to the runner f…
1850264 
…actory

This changes moves the configuration of common input settings like
processors, fields, tags to the runner factory (similar to filebeat).

The settings index, pipeline, and dataset have been added as generic
settings to all monitors.
remove unused function
0bb3106
Add index and pipeline setting to docs
e883c62
add index/ipeline setting to reference config
3ea607e
Add changelog
3d9a3e9
fix lint
4c5d741
Fix field not being added
748739b
@urso urso force-pushed the missig-index-setting branch from ae11779 to 748739b Compare August 20, 2020 15:30
andrewvc
andrewvc approved these changes Aug 20, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@andrewvc andrewvc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@urso urso merged commit 61eaf3b into elastic:master Aug 20, 2020
@urso urso deleted the missig-index-setting branch August 20, 2020 18:24
@urso urso mentioned this pull request Aug 20, 2020
Merged
4 tasks
@urso urso added v7.10.0 and removed needs_backport PR is waiting to be backported to other branches. labels Aug 20, 2020
urso pushed a commit to urso/beats that referenced this pull request Aug 20, 2020
Heartbeat: move configuration of common input setting to the runner f…
9b947ec 
…actory (elastic#20610)

(cherry picked from commit 61eaf3b)
urso pushed a commit that referenced this pull request Aug 21, 2020
Cherry-pick #20610 to 7.x: Heartbeat: move configuration of common in…
dcc316a 
…put setting to the runner factory (#20718)

(cherry picked from commit 61eaf3b)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
@melchiormoulin
Heartbeat: move configuration of common input setting to the runner f…
05325b6 
…actory (elastic#20610)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewvc andrewvc andrewvc approved these changes

Assignees

No one assigned

Labels

bug review Team:obs-ds-hosted-services Label for the Observability Hosted Services team v7.10.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Heartbeat] Add support for per monitor instance index

3 participants

@urso @elasticmachine @andrewvc