Skip to content

Cherry-pick #19480 to 7.x: Agent includes pgp file #21222

Merged
michalpristas merged 3 commits intoelastic:7.xfrom
michalpristas:backport_19480_7.x
Sep 29, 2020
Merged

Cherry-pick #19480 to 7.x: Agent includes pgp file #21222
michalpristas merged 3 commits intoelastic:7.xfrom
michalpristas:backport_19480_7.x

Conversation

@michalpristas
Copy link
Copy Markdown
Contributor

@michalpristas michalpristas commented Sep 22, 2020

Cherry-pick of PR #19480 to 7.x branch. Original message:

What does this PR do?

This PR introduces baked in PGP file with a flag.
If DEV=true is specified during mage build PGP is not included and checks are omitted .
Otherwise PGP is provided and passing check is required.

The solution works well with connected agent to internet, but locally baked in packages are a bit tricky as we need to find a way of including asc files into agent package so they can be checked.

Why is it important?

More security running external binaries

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@michalpristas michalpristas self-assigned this Sep 22, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 22, 2020

Pinging @elastic/ingest-management (Team:Ingest Management)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 22, 2020
blakerouse
blakerouse approved these changes Sep 22, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backport looks good.

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 22, 2020 •
edited by jenkins-beats-ci bot
Loading

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21222 updated]

  • Start Time: 2020-09-25T06:12:41.265+0000

  • Duration: 77 min 27 sec

Test stats 🧪

Test Results
Failed 1
Passed 20255
Skipped 1862
Total 22118

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat Windows / test_close_renamed – filebeat.tests.system.test_harvester.Test

    • Age: 1
    • Duration: 10.552
    • Error Details: beat.beat.TimeoutError: Timeout waiting for 'cond' to be true. Waited 10 seconds.

Steps errors

Expand to view the steps failures

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Duration: 9 min 21 sec

    • Start Time: 2020-09-25T06:41:52.670+0000

    • log

  • Name: Notifies GitHub of the status of a Pull Request

    • Description: script returned exit code 1

    • Duration: 0 min 1 sec

    • Start Time: 2020-09-25T06:51:28.565+0000

    • log

Log output

Expand to view the last 100 lines of log output 

[2020-09-25T07:29:31.108Z] + tar -xpf source.tgz
[2020-09-25T07:29:41.407Z] + rm source.tgz
[2020-09-25T07:29:41.418Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats
[2020-09-25T07:29:41.440Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Lint
[2020-09-25T07:29:41.538Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-09-25T07:29:41.623Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Winlogbeat-oss
[2020-09-25T07:29:41.700Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-09-25T07:29:41.783Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-09-25T07:29:41.867Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-09-25T07:29:42.107Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-09-25T07:29:42.185Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/dockerlogbeat
[2020-09-25T07:29:42.268Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Journalbeat
[2020-09-25T07:29:42.345Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-09-25T07:29:42.421Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-09-25T07:29:42.498Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-09-25T07:29:42.576Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Packetbeat-Linux
[2020-09-25T07:29:42.654Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-09-25T07:29:42.731Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-09-25T07:29:42.807Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-09-25T07:29:42.894Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-09-25T07:29:42.972Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Packetbeat-Mac-OS-X
[2020-09-25T07:29:43.050Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-09-25T07:29:43.128Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-09-25T07:29:43.206Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-09-25T07:29:43.286Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Heartbeat-oss
[2020-09-25T07:29:43.363Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-09-25T07:29:43.440Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Heartbeat-Windows
[2020-09-25T07:29:43.517Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-09-25T07:29:43.600Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-09-25T07:29:43.679Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-09-25T07:29:43.757Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-09-25T07:29:43.838Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-09-25T07:29:43.918Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Libbeat-x-pack
[2020-09-25T07:29:43.999Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Filebeat-Windows
[2020-09-25T07:29:44.076Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-09-25T07:29:44.159Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-09-25T07:29:44.236Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Functionbeat-Windows
[2020-09-25T07:29:44.313Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-09-25T07:29:44.390Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Packetbeat-Windows
[2020-09-25T07:29:44.469Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-09-25T07:29:44.547Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-Windows
[2020-09-25T07:29:44.626Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-09-25T07:29:44.706Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-09-25T07:29:44.783Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Filebeat-oss
[2020-09-25T07:29:44.862Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-09-25T07:29:44.941Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Libbeat-oss
[2020-09-25T07:29:45.028Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Filebeat-x-pack
[2020-09-25T07:29:45.112Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-09-25T07:29:45.189Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-09-25T07:29:45.267Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-09-25T07:29:45.342Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-09-25T07:29:45.738Z] + cat
[2020-09-25T07:29:45.738Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-25T07:29:45.738Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-25T07:29:52.329Z] runbld>>> runbld started
[2020-09-25T07:29:52.329Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-25T07:29:53.269Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-25T07:29:53.269Z] runbld>>> Matches in the system config:
[2020-09-25T07:29:53.269Z] runbld>>> - Matched ^elastic\+beats
[2020-09-25T07:29:53.269Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-25T07:29:54.666Z] runbld>>> Debug logging enabled.
[2020-09-25T07:29:54.666Z] runbld>>> Storing result
[2020-09-25T07:29:54.666Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-25T07:29:54.666Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200925072954-8EC30F61
[2020-09-25T07:29:54.666Z] runbld>>> Adding system facts.
[2020-09-25T07:29:55.618Z] runbld>>> Adding vcs info for the latest commit:  7742949af25a1ae9b6c314ef5e61329d8f09ffa0
[2020-09-25T07:29:55.618Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-25T07:29:55.877Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-25T07:29:55.877Z] Processing JUnit reports with runbld...
[2020-09-25T07:29:55.877Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-25T07:29:56.137Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-25T07:29:56.137Z] runbld>>> DURATION: 16ms
[2020-09-25T07:29:56.137Z] runbld>>> STDOUT: 40 bytes
[2020-09-25T07:29:56.137Z] runbld>>> STDERR: 49 bytes
[2020-09-25T07:29:56.137Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-25T07:29:56.137Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21222
[2020-09-25T07:29:57.076Z] runbld>>> Storing build metadata: 
[2020-09-25T07:29:57.076Z] runbld>>> Adding test report.
[2020-09-25T07:29:57.076Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats
[2020-09-25T07:29:58.019Z] runbld>>> Found 140 test output files
[2020-09-25T07:29:58.285Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-09-25T07:29:58.285Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-09-25T07:29:58.285Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-09-25T07:29:58.285Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-09-25T07:29:58.285Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-09-25T07:29:59.675Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-09-25T07:29:59.675Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21222/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-09-25T07:30:01.584Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 21963 Skipped: 1578
[2020-09-25T07:30:01.584Z] runbld>>> Storing result
[2020-09-25T07:30:01.584Z] runbld>>> FAILURES: 1
[2020-09-25T07:30:01.843Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-25T07:30:01.843Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200925072954-8EC30F61
[2020-09-25T07:30:02.104Z] runbld>>> Email notification disabled by environment variable.
[2020-09-25T07:30:02.104Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-25T07:30:07.918Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-21222
[2020-09-25T07:30:08.161Z] [INFO] getVaultSecret: Getting secrets
[2020-09-25T07:30:08.241Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-25T07:30:09.012Z] + chmod 755 generate-build-data.sh
[2020-09-25T07:30:09.012Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21222/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21222/runs/2 FAILURE 4647487
[2020-09-25T07:30:09.012Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21222/runs/2/steps/?limit=10000 -o steps-info.json

@michalpristas michalpristas merged commit 284c8c1 into elastic:7.x Sep 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@blakerouse blakerouse blakerouse approved these changes

Assignees

@michalpristas michalpristas

Labels

backport enhancement Ingest Management:beta2 Group issues for ingest management beta2

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@michalpristas @elasticmachine @blakerouse