Skip to content

[CI] more parallel stages and less provisioned workers#20569

Merged
v1v merged 2 commits intoelastic:masterfrom
v1v:fix/workers-pipeline
Aug 13, 2020
Merged

[CI] more parallel stages and less provisioned workers#20569
v1v merged 2 commits intoelastic:masterfrom
v1v:fix/workers-pipeline

Conversation

@v1v
Copy link
Copy Markdown
Member

@v1v v1v commented Aug 12, 2020

What does this PR do?

  • More stages in parallel rather than sequentially when they are OS specific based.
  • Reuse worker in some of the stages.

Why is it important?

More parallel builds will ideally reduce the overall build time.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 12, 2020
@v1v v1v changed the title Fix/workers pipeline [CI] more parallel stages and less provisioned workers Aug 12, 2020
@v1v v1v self-assigned this Aug 12, 2020
@v1v v1v added automation ci Team:Automation Label for the Observability productivity team labels Aug 12, 2020
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 12, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20569 opened]

  • Start Time: 2020-08-12T11:08:39.495+0000

  • Duration: 75 min 24 sec

Test stats 🧪

Test Results
Failed 0
Passed 17834
Skipped 1935
Total 19769

Steps errors

Expand to view the steps failures

  • Name: Make test

    • Description: make -C generator/_templates/metricbeat test

    • Duration: 1 min 55 sec

    • Start Time: 2020-08-12T11:43:59.890+0000

    • log

  • Name: Make test

    • Description: make -C generator/_templates/metricbeat test

    • Duration: 3 min 37 sec

    • Start Time: 2020-08-12T11:39:50.780+0000

    • log

  • Name: Recursively delete the current directory from the workspace

    • Description: script returned exit code 2

    • Duration: 0 min 16 sec

    • Start Time: 2020-08-12T11:42:36.339+0000

    • log

  • Name: Make test

    • Description: make -C generator/_templates/beat test

    • Duration: 3 min 55 sec

    • Start Time: 2020-08-12T11:39:53.676+0000

    • log

  • Name: Recursively delete the current directory from the workspace

    • Description: script returned exit code 2

    • Duration: 0 min 18 sec

    • Start Time: 2020-08-12T11:42:56.458+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-08-12T12:23:22.071Z] + tar --version
[2020-08-12T12:23:22.387Z] + tar -xpf source.tgz
[2020-08-12T12:23:32.749Z] + rm source.tgz
[2020-08-12T12:23:32.773Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats
[2020-08-12T12:23:32.821Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Lint
[2020-08-12T12:23:32.976Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-08-12T12:23:33.135Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-08-12T12:23:33.289Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-08-12T12:23:33.441Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-08-12T12:23:33.595Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Winlogbeat-oss
[2020-08-12T12:23:33.749Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-08-12T12:23:33.905Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Journalbeat
[2020-08-12T12:23:34.060Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/dockerlogbeat
[2020-08-12T12:23:34.215Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-08-12T12:23:34.369Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-08-12T12:23:34.523Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-08-12T12:23:34.685Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Packetbeat-Linux
[2020-08-12T12:23:34.843Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-08-12T12:23:35.000Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-08-12T12:23:35.157Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Packetbeat-Mac-OS-X
[2020-08-12T12:23:35.309Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-08-12T12:23:35.464Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-08-12T12:23:35.620Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-08-12T12:23:35.784Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Heartbeat-oss
[2020-08-12T12:23:35.941Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-08-12T12:23:36.094Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-08-12T12:23:36.249Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-08-12T12:23:36.398Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-08-12T12:23:36.553Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-08-12T12:23:36.710Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Heartbeat-Windows
[2020-08-12T12:23:36.864Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-08-12T12:23:37.019Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-08-12T12:23:37.176Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-08-12T12:23:37.327Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-08-12T12:23:37.477Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Filebeat-Windows
[2020-08-12T12:23:37.633Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Libbeat-x-pack
[2020-08-12T12:23:37.790Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Packetbeat-Windows
[2020-08-12T12:23:37.952Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-08-12T12:23:38.107Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-08-12T12:23:38.262Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-08-12T12:23:38.416Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-Windows
[2020-08-12T12:23:38.570Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Functionbeat-Windows
[2020-08-12T12:23:38.728Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-08-12T12:23:38.883Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Filebeat-x-pack
[2020-08-12T12:23:39.036Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Filebeat-oss
[2020-08-12T12:23:39.188Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Libbeat-oss
[2020-08-12T12:23:39.402Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-08-12T12:23:39.560Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-08-12T12:23:39.718Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-08-12T12:23:39.878Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-08-12T12:23:40.030Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-08-12T12:23:40.533Z] + cat
[2020-08-12T12:23:40.533Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-08-12T12:23:40.533Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-08-12T12:23:47.134Z] runbld>>> runbld started
[2020-08-12T12:23:47.134Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-08-12T12:23:49.052Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-08-12T12:23:49.052Z] runbld>>> Matches in the system config:
[2020-08-12T12:23:49.052Z] runbld>>> - Matched ^elastic\+beats
[2020-08-12T12:23:49.052Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-08-12T12:23:50.441Z] runbld>>> Debug logging enabled.
[2020-08-12T12:23:50.441Z] runbld>>> Storing result
[2020-08-12T12:23:50.441Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-08-12T12:23:50.441Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200812122350-1DFD5AB2
[2020-08-12T12:23:50.441Z] runbld>>> Adding system facts.
[2020-08-12T12:23:51.843Z] runbld>>> Adding vcs info for the latest commit:  96ecf00d2e042d0d0f0b3ad683e3b5a100d6d159
[2020-08-12T12:23:51.843Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-08-12T12:23:51.843Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-08-12T12:23:51.843Z] + echo 'Processing JUnit reports with runbld...'
[2020-08-12T12:23:51.843Z] Processing JUnit reports with runbld...
[2020-08-12T12:23:52.103Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-08-12T12:23:52.103Z] runbld>>> DURATION: 34ms
[2020-08-12T12:23:52.103Z] runbld>>> STDOUT: 40 bytes
[2020-08-12T12:23:52.103Z] runbld>>> STDERR: 49 bytes
[2020-08-12T12:23:52.103Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-08-12T12:23:52.103Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-20569
[2020-08-12T12:23:53.047Z] runbld>>> Storing build metadata: 
[2020-08-12T12:23:53.047Z] runbld>>> Adding test report.
[2020-08-12T12:23:53.047Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats
[2020-08-12T12:23:53.619Z] runbld>>> Found 137 test output files
[2020-08-12T12:23:54.562Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-08-12T12:23:54.562Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-08-12T12:23:54.824Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-08-12T12:23:54.824Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-08-12T12:23:54.824Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-08-12T12:23:54.824Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-08-12T12:23:54.824Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20569/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-08-12T12:23:56.739Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 19619 Skipped: 1664
[2020-08-12T12:23:57.000Z] runbld>>> Storing result
[2020-08-12T12:23:57.000Z] runbld>>> FAILURES: 0
[2020-08-12T12:23:57.262Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-08-12T12:23:57.262Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200812122350-1DFD5AB2
[2020-08-12T12:23:57.262Z] runbld>>> Email notification disabled by environment variable.
[2020-08-12T12:23:57.262Z] runbld>>> Slack notification disabled by environment variable.
[2020-08-12T12:24:02.782Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20569
[2020-08-12T12:24:02.950Z] [INFO] getVaultSecret: Getting secrets
[2020-08-12T12:24:03.042Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-08-12T12:24:04.081Z] + chmod 755 generate-build-data.sh
[2020-08-12T12:24:04.082Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20569/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20569/runs/1 FAILURE 4524315
[2020-08-12T12:24:04.082Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20569/runs/1/steps/?limit=10000 -o steps-info.json

@v1v v1v requested a review from a team August 12, 2020 12:38
@v1v v1v marked this pull request as ready for review August 12, 2020 12:38
@v1v v1v merged commit dbd6e36 into elastic:master Aug 13, 2020
@v1v v1v deleted the fix/workers-pipeline branch August 13, 2020 10:31
v1v added a commit to v1v/beats that referenced this pull request Aug 13, 2020
v1v added a commit to v1v/beats that referenced this pull request Aug 13, 2020
v1v added a commit that referenced this pull request Aug 17, 2020
…rkers (#20569) (#20582)

* [CI] more parallel stages and less provisioned workers (#20569)
* Enable and fix Journalbeat integration tests (#20140)
* [CI] Fix missing when for Packetbeat (#20626)
v1v added a commit that referenced this pull request Aug 17, 2020
…rkers (#20569) (#20583)

* [CI] more parallel stages and less provisioned workers (#20569)
* Enable and fix Journalbeat integration tests (#20140)
* [CI] Fix missing when for Packetbeat (#20626)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…rkers (elastic#20569) (elastic#20582)

* [CI] more parallel stages and less provisioned workers (elastic#20569)
* Enable and fix Journalbeat integration tests (elastic#20140)
* [CI] Fix missing when for Packetbeat (elastic#20626)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automation ci Team:Automation Label for the Observability productivity team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants