Skip to content

Cherry-pick #20356 to 7.x: Prepare home directories for docker images in a different stage#20399

Merged
jsoriano merged 1 commit intoelastic:7.xfrom
jsoriano:backport_20356_7.x
Aug 3, 2020
Merged

Cherry-pick #20356 to 7.x: Prepare home directories for docker images in a different stage#20399
jsoriano merged 1 commit intoelastic:7.xfrom
jsoriano:backport_20356_7.x

Conversation

@jsoriano
Copy link
Copy Markdown
Member

@jsoriano jsoriano commented Aug 3, 2020

Cherry-pick of PR #20356 to 7.x branch. Original message:

Files included in the home directory of the docker images need some
changes on permissions and ownership after being copied. If this is
done as a COPY and a RUN, it creates two layers with all the files, that
are included in the final image, increasing its size.

Move the preparation of the home directory to a different stage, so in
the final image it is done as an only COPY operation.

This change reduces for example the Metricbeat image size a ~25%,
from 558MB to 418MB. Auditbeat image from 462MB to 370MB.

How to test

  • Docker images continue working.

…tic#20356)

Files included in the home directory of the docker images need some
changes on permissions and ownerships after being copied. If this is
done as a COPY and a RUN, it creates two layers with all the files, that
are included in the final image, increasing its size.

Move the preparation of the home directory to a different stage, so in
the final image it is done as an only COPY operation.

(cherry picked from commit aaf3d7e)
@jsoriano jsoriano added [zube]: In Review backport Team:Automation Label for the Observability productivity team labels Aug 3, 2020
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 3, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20399 opened]

  • Start Time: 2020-08-03T11:33:03.770+0000

  • Duration: 73 min 16 sec

Test stats 🧪

Test Results
Failed 1
Passed 17651
Skipped 1832
Total 19484

Test errors

Expand to view the tests failures

  • Name: Build and Test / Libbeat / Libbeat oss / TestDockerStart – docker

    • Age: 1
    • Duration: 12.52
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: Mage build test
    • Description: mage build test

    • Duration: 24 min 44 sec

    • Start Time: 2020-08-03T11:58:04.771+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-08-03T12:45:26.298Z]  Git commit:        e68fc7a
[2020-08-03T12:45:26.298Z]  Built:             Tue Aug 21 17:21:31 2018
[2020-08-03T12:45:26.298Z]  OS/Arch:           darwin/amd64
[2020-08-03T12:45:26.298Z]  Experimental:      false
[2020-08-03T12:45:26.298Z] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[2020-08-03T12:45:26.298Z] It requires Docker daemon to be installed and running
[2020-08-03T12:45:46.715Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats
[2020-08-03T12:45:47.069Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-08-03T12:45:47.099Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Lint
[2020-08-03T12:45:47.279Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Winlogbeat-oss
[2020-08-03T12:45:47.475Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-08-03T12:45:47.664Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-08-03T12:45:47.840Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-08-03T12:45:48.027Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Journalbeat-oss
[2020-08-03T12:45:48.204Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-08-03T12:45:48.380Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Dockerlogbeat
[2020-08-03T12:45:48.554Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-08-03T12:45:48.725Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-08-03T12:45:48.896Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Packetbeat-Linux
[2020-08-03T12:45:49.066Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-08-03T12:45:49.241Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-08-03T12:45:49.409Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-08-03T12:45:49.580Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-08-03T12:45:49.755Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-08-03T12:45:49.929Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-08-03T12:45:50.099Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Heartbeat-oss
[2020-08-03T12:45:50.399Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-08-03T12:45:50.574Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-08-03T12:45:50.774Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-08-03T12:45:50.946Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-08-03T12:45:51.104Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-08-03T12:45:51.276Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-08-03T12:45:51.439Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Libbeat-x-pack
[2020-08-03T12:45:51.599Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-08-03T12:45:51.762Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-08-03T12:45:51.921Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Filebeat-Windows
[2020-08-03T12:45:52.081Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-08-03T12:45:52.235Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Packetbeat-Mac-OS-X
[2020-08-03T12:45:52.393Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-08-03T12:45:52.553Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-08-03T12:45:52.709Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-Windows
[2020-08-03T12:45:52.869Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-08-03T12:45:53.025Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Libbeat-oss
[2020-08-03T12:45:53.182Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-08-03T12:45:53.342Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Heartbeat-Windows
[2020-08-03T12:45:53.494Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Filebeat-x-pack
[2020-08-03T12:45:53.657Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Filebeat-oss
[2020-08-03T12:45:53.819Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Packetbeat-Windows
[2020-08-03T12:45:53.986Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Functionbeat-Windows
[2020-08-03T12:45:54.152Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-08-03T12:45:54.319Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-08-03T12:45:54.495Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-08-03T12:45:54.676Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-08-03T12:45:54.839Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-08-03T12:45:55.305Z] + cat
[2020-08-03T12:45:55.306Z] + /usr/local/bin/runbld ./runbld-script
[2020-08-03T12:45:55.306Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-08-03T12:46:01.949Z] runbld>>> runbld started
[2020-08-03T12:46:01.949Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-08-03T12:46:03.866Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-20399' in order of occurrence in the config (last value wins).
[2020-08-03T12:46:05.255Z] runbld>>> Debug logging enabled.
[2020-08-03T12:46:05.255Z] runbld>>> Storing result
[2020-08-03T12:46:05.255Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-08-03T12:46:05.255Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200803124604-7F58CBFA
[2020-08-03T12:46:05.255Z] runbld>>> Adding system facts.
[2020-08-03T12:46:06.648Z] runbld>>> Adding vcs info for the latest commit:  97d44c0a782ab5a0a3a7abbeb124c6457593f672
[2020-08-03T12:46:06.648Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-08-03T12:46:06.648Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-08-03T12:46:06.648Z] + echo 'Processing JUnit reports with runbld...'
[2020-08-03T12:46:06.648Z] Processing JUnit reports with runbld...
[2020-08-03T12:46:06.911Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-08-03T12:46:06.911Z] runbld>>> DURATION: 60ms
[2020-08-03T12:46:06.911Z] runbld>>> STDOUT: 40 bytes
[2020-08-03T12:46:06.911Z] runbld>>> STDERR: 49 bytes
[2020-08-03T12:46:06.911Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-08-03T12:46:06.911Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats
[2020-08-03T12:46:07.854Z] runbld>>> Storing build metadata: 
[2020-08-03T12:46:07.854Z] runbld>>> Adding test report.
[2020-08-03T12:46:07.854Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats
[2020-08-03T12:46:08.799Z] runbld>>> Found 135 test output files
[2020-08-03T12:46:09.743Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-08-03T12:46:09.743Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-08-03T12:46:09.743Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-08-03T12:46:09.743Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-08-03T12:46:09.743Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-08-03T12:46:09.743Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-08-03T12:46:10.004Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20399/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-08-03T12:46:12.021Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 19338 Skipped: 1555
[2020-08-03T12:46:12.021Z] runbld>>> Storing result
[2020-08-03T12:46:12.021Z] runbld>>> FAILURES: 1
[2020-08-03T12:46:12.283Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-08-03T12:46:12.283Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200803124604-7F58CBFA
[2020-08-03T12:46:12.544Z] runbld>>> Email notification disabled by environment variable.
[2020-08-03T12:46:12.544Z] runbld>>> Slack notification disabled by environment variable.
[2020-08-03T12:46:18.420Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20399
[2020-08-03T12:46:18.721Z] [INFO] getVaultSecret: Getting secrets
[2020-08-03T12:46:18.809Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-08-03T12:46:19.921Z] + chmod 755 generate-build-data.sh
[2020-08-03T12:46:19.921Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20399/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20399/runs/1 FAILURE 4395879
[2020-08-03T12:46:19.921Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20399/runs/1/steps/?limit=10000 -o steps-info.json

@jsoriano jsoriano added the Team:Integrations Label for the Integrations team label Aug 3, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/integrations (Team:Integrations)

Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backport looks good. CI failed with TestDockerStart: docker_integration_test.go:125: Timeout waiting for provider events, doesn't seem to be related?!

@jsoriano
Copy link
Copy Markdown
Member Author

jsoriano commented Aug 3, 2020

Backport looks good. CI failed with TestDockerStart: docker_integration_test.go:125: Timeout waiting for provider events, doesn't seem to be related?!

This is a flaky test #20360, not related.

@jsoriano jsoriano merged commit 061a731 into elastic:7.x Aug 3, 2020
@jsoriano jsoriano deleted the backport_20356_7.x branch August 3, 2020 13:55
@kaiyan-sheng
Copy link
Copy Markdown
Contributor

Good to know! Thank you!!

@zube zube bot removed the [zube]: Done label Nov 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Automation Label for the Observability productivity team Team:Integrations Label for the Integrations team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants