You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Cherry-pick of PR #21493 to 7.x branch. Original message:
What does this PR do?
This PR adds user-configurable fields retry_interval and max_retry_interval to the disk queue, and uses them to perform exponential backoff when encountering fatal errors writing to disk.
I'm aware that there are some existing helper wrappers for this functionality, e.g. ExpBackoff in libbeat/common/backoff. Unfortunately they didn't fit the cancellation or error handling model in the queue, so the backoff here is done "by hand." I've tried to restrict the moving parts to self-contained helper functions.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.
How to test this PR locally
Enable the disk queue (with e.g. queue.disk.max_size: 1GB in the beat config) and start the beat. While it's running, remove write permissions to data/diskqueue. This should log errors for the writer and deleter (if applicable), e.g.:
By default, any such errors should start 1 second apart and grow by powers of 2 up to 30 seconds. This default can be changed by setting queue.disk.{retry_interval, max_retry_interval}.
[2020-10-02T21:55:05.333Z] runc:
[2020-10-02T21:55:05.333Z] Version: 1.0.0-rc10
[2020-10-02T21:55:05.333Z] GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-10-02T21:55:05.333Z] docker-init:
[2020-10-02T21:55:05.333Z] Version: 0.18.0
[2020-10-02T21:55:05.333Z] GitCommit: fec3683
[2020-10-02T21:55:13.126Z] Failed in branch x-pack/filebeat-build
[2020-10-02T21:55:17.715Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-10-02T21:55:17.738Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-10-02T21:55:17.811Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-21497-1/source/source.tgz
[2020-10-02T21:55:17.828Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-21497-1/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-21497/source.tgz
[2020-10-02T21:55:32.101Z] + tar --version
[2020-10-02T21:55:32.428Z] + tar -xpf source.tgz
[2020-10-02T21:55:45.016Z] + rm source.tgz
[2020-10-02T21:55:45.157Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats
[2020-10-02T21:55:45.187Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/uncategorized-1601672104488
[2020-10-02T21:55:45.282Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/libbeat-stress-tests-1601672338714
[2020-10-02T21:55:45.375Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/winlogbeat-crosscompile-1601672434787
[2020-10-02T21:55:45.544Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/auditbeat-crosscompile-1601672452477
[2020-10-02T21:55:45.641Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-elastic-agent-build-1601672473638
[2020-10-02T21:55:45.739Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-dockerlogbeat-build-1601672511297
[2020-10-02T21:55:45.835Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/journalbeat-unitTest-1601672527604
[2020-10-02T21:55:45.932Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/libbeat-crosscompile-1601672623780
[2020-10-02T21:55:46.025Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-functionbeat-build-1601672632049
[2020-10-02T21:55:46.120Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/packetbeat-build-1601672709747
[2020-10-02T21:55:46.211Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-elastic-agent-windows-windows-2019-1601672821643
[2020-10-02T21:55:46.308Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/metricbeat-unitTest-1601672850812
[2020-10-02T21:55:46.398Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/auditbeat-windows-windows-2019-1601672912956
[2020-10-02T21:55:46.487Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/heartbeat-build-1601672917480
[2020-10-02T21:55:46.597Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/heartbeat-windows-windows-2019-1601672953396
[2020-10-02T21:55:46.702Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/metricbeat-crosscompile-1601673001480
[2020-10-02T21:55:46.803Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/auditbeat-build-1601673017781
[2020-10-02T21:55:46.901Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-auditbeat-build-1601673020335
[2020-10-02T21:55:46.991Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-auditbeat-windows-windows-2019-1601673029605
[2020-10-02T21:55:47.091Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/generator-metricbeat-test-1601673051905
[2020-10-02T21:55:47.184Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/generator-beat-test-1601673056656
[2020-10-02T21:55:47.277Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-winlogbeat-build-windows-2019-1601673060757
[2020-10-02T21:55:47.376Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/packetbeat-windows-windows-2019-1601673128832
[2020-10-02T21:55:47.478Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/filebeat-windows-windows-2019-1601673159493
[2020-10-02T21:55:47.571Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-libbeat-build-1601673163980
[2020-10-02T21:55:47.665Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/winlogbeat-windows-windows-2019-1601673166367
[2020-10-02T21:55:47.761Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019-1601673227592
[2020-10-02T21:55:47.866Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019-1601673309697
[2020-10-02T21:55:47.959Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/metricbeat-windows-windows-2019-1601673469362
[2020-10-02T21:55:48.057Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-metricbeat-windows-windows-2019-1601673545264
[2020-10-02T21:55:48.153Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/metricbeat-pythonIntegTest-1601674301488
[2020-10-02T21:55:48.250Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/metricbeat-goIntegTest-1601674386592
[2020-10-02T21:55:48.346Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/libbeat-build-1601674723823
[2020-10-02T21:55:48.443Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/filebeat-build-1601674826709
[2020-10-02T21:55:48.538Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-metricbeat-build-1601675129306
[2020-10-02T21:55:48.633Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-filebeat-build-1601675670406
[2020-10-02T21:55:49.053Z] + cat
[2020-10-02T21:55:49.053Z] + /usr/local/bin/runbld ./runbld-test-reports --job-name elastic+beats+pull-request
[2020-10-02T21:55:49.053Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-10-02T21:55:55.680Z] runbld>>> runbld started
[2020-10-02T21:55:55.680Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-10-02T21:55:58.015Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-10-02T21:55:58.015Z] runbld>>> Matches in the system config:
[2020-10-02T21:55:58.015Z] runbld>>> - Matched ^elastic\+beats
[2020-10-02T21:55:58.015Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-10-02T21:55:58.668Z] runbld>>> Debug logging enabled.
[2020-10-02T21:55:58.668Z] runbld>>> Storing result
[2020-10-02T21:55:58.979Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-10-02T21:55:58.979Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20201002215558-35202C88
[2020-10-02T21:55:58.979Z] runbld>>> Adding system facts.
[2020-10-02T21:55:59.969Z] runbld>>> Adding vcs info for the latest commit: 1468d104298a57decf0bf5713e25f62e13a9a2f6
[2020-10-02T21:55:59.969Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-10-02T21:55:59.969Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-10-02T21:55:59.969Z] Processing JUnit reports with runbld...
[2020-10-02T21:55:59.969Z] + echo 'Processing JUnit reports with runbld...'
[2020-10-02T21:56:00.589Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-10-02T21:56:00.590Z] runbld>>> DURATION: 25ms
[2020-10-02T21:56:00.590Z] runbld>>> STDOUT: 40 bytes
[2020-10-02T21:56:00.590Z] runbld>>> STDERR: 49 bytes
[2020-10-02T21:56:00.590Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-10-02T21:56:00.590Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21497
[2020-10-02T21:56:01.566Z] runbld>>> Storing build metadata:
[2020-10-02T21:56:01.566Z] runbld>>> Adding test report.
[2020-10-02T21:56:01.566Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats
[2020-10-02T21:56:02.163Z] runbld>>> Found 123 test output files
[2020-10-02T21:56:02.436Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/metricbeat-goIntegTest-1601674386592/metricbeat/build/TEST-go-integration-graphite.xml
[2020-10-02T21:56:02.436Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/metricbeat-goIntegTest-1601674386592/metricbeat/build/TEST-go-integration-windows.xml
[2020-10-02T21:56:04.389Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-metricbeat-build-1601675129306/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-10-02T21:56:04.389Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-metricbeat-build-1601675129306/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-10-02T21:56:04.389Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-metricbeat-build-1601675129306/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-10-02T21:56:04.389Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-metricbeat-build-1601675129306/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-10-02T21:56:04.389Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21497/src/github.com/elastic/beats/x-pack-metricbeat-build-1601675129306/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-10-02T21:56:05.355Z] runbld>>> Test output logs contained: Errors: 0 Failures: 327 Tests: 17625 Skipped: 1088
[2020-10-02T21:56:05.355Z] runbld>>> Storing result
[2020-10-02T21:56:05.355Z] runbld>>> FAILURES: 327
[2020-10-02T21:57:13.161Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-10-02T21:57:13.161Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20201002215558-35202C88
[2020-10-02T21:57:13.161Z] runbld>>> Email notification disabled by environment variable.
[2020-10-02T21:57:13.161Z] runbld>>> Slack notification disabled by environment variable.
[2020-10-02T21:57:13.461Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-21497
[2020-10-02T21:57:13.548Z] [INFO] getVaultSecret: Getting secrets
[2020-10-02T21:57:13.642Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-10-02T21:57:14.563Z] + chmod 755 generate-build-data.sh
[2020-10-02T21:57:14.563Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21497/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21497/runs/1 FAILURE 5071539
[2020-10-02T21:57:14.563Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21497/runs/1/steps/?limit=10000 -o steps-info.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of PR #21493 to 7.x branch. Original message:
What does this PR do?
This PR adds user-configurable fields
retry_intervalandmax_retry_intervalto the disk queue, and uses them to perform exponential backoff when encountering fatal errors writing to disk.I'm aware that there are some existing helper wrappers for this functionality, e.g.
ExpBackoffinlibbeat/common/backoff. Unfortunately they didn't fit the cancellation or error handling model in the queue, so the backoff here is done "by hand." I've tried to restrict the moving parts to self-contained helper functions.I have made corresponding changes to the documentationI have made corresponding change to the default configuration filesI have added tests that prove my fix is effective or that my feature worksI have added an entry inCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
Enable the disk queue (with e.g.
queue.disk.max_size: 1GBin the beat config) and start the beat. While it's running, remove write permissions todata/diskqueue. This should log errors for the writer and deleter (if applicable), e.g.:By default, any such errors should start 1 second apart and grow by powers of 2 up to 30 seconds. This default can be changed by setting
queue.disk.{retry_interval, max_retry_interval}.