Cherry-pick to 7.10: ci: configure retries on tests (#20891) by kuisathaverat · Pull Request #22023 · elastic/beats

kuisathaverat · 2020-10-20T14:03:46Z

Backports the following commits to 7.10:

ci: configure retries on tests (ci: configure retries on tests #20891)

* feat: configure retries on tests * Update Jenkinsfile Co-authored-by: cachedout <mike.place@elastic.co> * Update Jenkinsfile Co-authored-by: cachedout <mike.place@elastic.co> # Conflicts: # Jenkinsfile

elasticmachine · 2020-10-20T15:37:54Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: [Pull request #22023 opened]
Start Time: 2020-10-20T14:04:40.351+0000
Duration: 93 min 58 sec

Test stats 🧪

Test	Results
Failed	2
Passed	16333
Skipped	1367
Total	17702

Test errors

Expand to view the tests failures

Name: Build&Test / x-pack/elastic-agent-build / TestRetriesOnFailures – application
- Age: 1
- Duration: 0
- Error Details: Failed
Name: Build&Test / x-pack/elastic-agent-build / TestRetriesOnFailures/When_the_gateway_fails_to_communicate_with_the_checkin_API_we_will_retry – application
- Age: 1
- Duration: 0
- Error Details: Failed

Steps errors

Expand to view the steps failures

Name: x-pack/elastic-agent-build - mage build test
- Description: mage build test
- Duration: 12 min 2 sec
- Start Time: 2020-10-20T14:31:31.180+0000
- log

Log output

Expand to view the last 100 lines of log output

[2020-10-20T15:36:33.129Z]  runc:
[2020-10-20T15:36:33.129Z]   Version:          1.0.0-rc10
[2020-10-20T15:36:33.129Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-10-20T15:36:33.129Z]  docker-init:
[2020-10-20T15:36:33.129Z]   Version:          0.18.0
[2020-10-20T15:36:33.129Z]   GitCommit:        fec3683
[2020-10-20T15:36:47.108Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-10-20T15:36:47.131Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-10-20T15:36:47.192Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-22023-1/source/source.tgz
[2020-10-20T15:36:47.210Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-22023-1/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-22023/source.tgz
[2020-10-20T15:37:00.024Z] + tar --version
[2020-10-20T15:37:00.374Z] + tar -xpf source.tgz
[2020-10-20T15:37:10.902Z] + rm source.tgz
[2020-10-20T15:37:10.999Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats
[2020-10-20T15:37:11.030Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/libbeat-stress-tests-1603204290116
[2020-10-20T15:37:11.130Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/winlogbeat-crosscompile-1603204358492
[2020-10-20T15:37:11.229Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/auditbeat-crosscompile-1603204397259
[2020-10-20T15:37:11.326Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-dockerlogbeat-build-1603204442994
[2020-10-20T15:37:11.419Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/journalbeat-unitTest-1603204471276
[2020-10-20T15:37:11.517Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/libbeat-crosscompile-1603204531116
[2020-10-20T15:37:11.613Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-functionbeat-build-1603204586381
[2020-10-20T15:37:11.710Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/packetbeat-build-1603204658989
[2020-10-20T15:37:11.805Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/heartbeat-windows-windows-2019-1603204775815
[2020-10-20T15:37:11.906Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/heartbeat-build-1603204838540
[2020-10-20T15:37:12.000Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/metricbeat-unitTest-1603204846903
[2020-10-20T15:37:12.095Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/metricbeat-crosscompile-1603204870765
[2020-10-20T15:37:12.189Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-elastic-agent-windows-windows-2019-1603204872222
[2020-10-20T15:37:12.290Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/auditbeat-windows-windows-2019-1603204880163
[2020-10-20T15:37:12.395Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/auditbeat-build-1603204887226
[2020-10-20T15:37:12.497Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/generator-metricbeat-test-1603204971380
[2020-10-20T15:37:12.593Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-elastic-agent-build-1603205020852
[2020-10-20T15:37:12.688Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-auditbeat-windows-windows-2019-1603205023573
[2020-10-20T15:37:12.787Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/generator-beat-test-1603205030905
[2020-10-20T15:37:12.885Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-libbeat-build-1603205035656
[2020-10-20T15:37:12.985Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-auditbeat-build-1603205057992
[2020-10-20T15:37:13.084Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-winlogbeat-build-windows-2019-1603205129269
[2020-10-20T15:37:13.179Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/filebeat-windows-windows-2019-1603205132570
[2020-10-20T15:37:13.295Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019-1603205133383
[2020-10-20T15:37:13.388Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/winlogbeat-windows-windows-2019-1603205147087
[2020-10-20T15:37:13.484Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019-1603205157070
[2020-10-20T15:37:13.578Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/packetbeat-windows-windows-2019-1603205171605
[2020-10-20T15:37:13.680Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/metricbeat-windows-windows-2019-1603205355606
[2020-10-20T15:37:13.771Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-metricbeat-windows-windows-2019-1603205472733
[2020-10-20T15:37:13.864Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/filebeat-build-1603205775175
[2020-10-20T15:37:13.956Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/metricbeat-goIntegTest-1603206250276
[2020-10-20T15:37:14.052Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/metricbeat-pythonIntegTest-1603206332250
[2020-10-20T15:37:14.149Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-filebeat-build-1603206600511
[2020-10-20T15:37:14.239Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/libbeat-build-1603206819148
[2020-10-20T15:37:14.366Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-metricbeat-build-1603207809938
[2020-10-20T15:37:14.778Z] + cat
[2020-10-20T15:37:14.778Z] + /usr/local/bin/runbld ./runbld-test-reports --job-name elastic+beats+pull-request
[2020-10-20T15:37:14.778Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-10-20T15:37:21.367Z] runbld>>> runbld started
[2020-10-20T15:37:21.367Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-10-20T15:37:22.752Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-10-20T15:37:23.013Z] runbld>>> Matches in the system config:
[2020-10-20T15:37:23.013Z] runbld>>> - Matched ^elastic\+beats
[2020-10-20T15:37:23.013Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-10-20T15:37:24.395Z] runbld>>> Debug logging enabled.
[2020-10-20T15:37:24.395Z] runbld>>> Storing result
[2020-10-20T15:37:24.396Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-10-20T15:37:24.396Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20201020153724-23EFE63E
[2020-10-20T15:37:24.396Z] runbld>>> Adding system facts.
[2020-10-20T15:37:25.782Z] runbld>>> Adding vcs info for the latest commit:  6065e06b33875c994a4fd3a1dabf65d2087347a0
[2020-10-20T15:37:25.782Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-10-20T15:37:25.782Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-10-20T15:37:25.782Z] Processing JUnit reports with runbld...
[2020-10-20T15:37:25.782Z] + echo 'Processing JUnit reports with runbld...'
[2020-10-20T15:37:26.044Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-10-20T15:37:26.044Z] runbld>>> DURATION: 32ms
[2020-10-20T15:37:26.044Z] runbld>>> STDOUT: 40 bytes
[2020-10-20T15:37:26.044Z] runbld>>> STDERR: 49 bytes
[2020-10-20T15:37:26.044Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-10-20T15:37:26.044Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-22023
[2020-10-20T15:37:26.986Z] runbld>>> Storing build metadata: 
[2020-10-20T15:37:26.986Z] runbld>>> Adding test report.
[2020-10-20T15:37:26.986Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats
[2020-10-20T15:37:27.928Z] runbld>>> Found 123 test output files
[2020-10-20T15:37:28.873Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/metricbeat-goIntegTest-1603206250276/metricbeat/build/TEST-go-integration-windows.xml
[2020-10-20T15:37:28.873Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/metricbeat-goIntegTest-1603206250276/metricbeat/build/TEST-go-integration-graphite.xml
[2020-10-20T15:37:29.134Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-metricbeat-build-1603207809938/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-10-20T15:37:29.134Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-metricbeat-build-1603207809938/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-10-20T15:37:29.134Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-metricbeat-build-1603207809938/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-10-20T15:37:29.134Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-metricbeat-build-1603207809938/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-10-20T15:37:29.134Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22023/src/github.com/elastic/beats/x-pack-metricbeat-build-1603207809938/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-10-20T15:37:31.047Z] runbld>>> Test output logs contained: Errors: 0 Failures: 2 Tests: 17702 Skipped: 1088
[2020-10-20T15:37:31.047Z] runbld>>> Storing result
[2020-10-20T15:37:31.047Z] runbld>>> FAILURES: 2
[2020-10-20T15:37:31.618Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-10-20T15:37:31.618Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20201020153724-23EFE63E
[2020-10-20T15:37:31.618Z] runbld>>> Email notification disabled by environment variable.
[2020-10-20T15:37:31.618Z] runbld>>> Slack notification disabled by environment variable.
[2020-10-20T15:37:37.152Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-22023
[2020-10-20T15:37:37.248Z] [INFO] getVaultSecret: Getting secrets
[2020-10-20T15:37:37.313Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-10-20T15:37:38.209Z] + chmod 755 generate-build-data.sh
[2020-10-20T15:37:38.209Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22023/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22023/runs/1 FAILURE 5577587
[2020-10-20T15:37:38.209Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22023/runs/1/steps/?limit=10000 -o steps-info.json
[2020-10-20T15:37:41.437Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22023/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-10-20T15:37:41.688Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22023/runs/1/log/ -o pipeline-log.txt

ci: configure retries on tests (elastic#20891)

6065e06

* feat: configure retries on tests * Update Jenkinsfile Co-authored-by: cachedout <mike.place@elastic.co> * Update Jenkinsfile Co-authored-by: cachedout <mike.place@elastic.co> # Conflicts: # Jenkinsfile

kuisathaverat self-assigned this Oct 20, 2020

kuisathaverat added the backport label Oct 20, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 20, 2020

kuisathaverat added automation ci Team:Automation Label for the Observability productivity team labels Oct 20, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 20, 2020

kuisathaverat requested review from a team and jsoriano October 20, 2020 14:05

kuisathaverat mentioned this pull request Oct 20, 2020

Cherry-pick to 7.10: fix: mark flaky tests (#21300) #22007

Merged

v1v approved these changes Oct 20, 2020

View reviewed changes

kuisathaverat merged commit 6d0b2a6 into elastic:7.10 Oct 20, 2020

kuisathaverat deleted the backport/7.10/pr-20891 branch December 24, 2020 11:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick to 7.10: ci: configure retries on tests (#20891)#22023

Cherry-pick to 7.10: ci: configure retries on tests (#20891)#22023
kuisathaverat merged 1 commit intoelastic:7.10from
kuisathaverat:backport/7.10/pr-20891

kuisathaverat commented Oct 20, 2020

Uh oh!

elasticmachine commented Oct 20, 2020

Build stats

Test stats 🧪

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kuisathaverat commented Oct 20, 2020

Uh oh!

elasticmachine commented Oct 20, 2020

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Steps errors

Log output

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants