Skip to content

Cherry-pick #20842 to 7.x: [Elastic Agent] Add docker composable dynamic provider.#21117

Merged
blakerouse merged 2 commits intoelastic:7.xfrom
blakerouse:backport_20842_7.x
Sep 16, 2020
Merged

Cherry-pick #20842 to 7.x: [Elastic Agent] Add docker composable dynamic provider.#21117
blakerouse merged 2 commits intoelastic:7.xfrom
blakerouse:backport_20842_7.x

Conversation

@blakerouse
Copy link
Copy Markdown
Contributor

@blakerouse blakerouse commented Sep 16, 2020

Cherry-pick of PR #20842 to 7.x branch. Original message:

What does this PR do?

Adds a docker composable dynamic provider.

Why is it important?

Support generating inputs from discovered docker containers.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Using the following configuration results in the inputs results in a new input per container.

inputs:
  - type: logfile
     streams:
      - path: /var/lib/docker/containers/${docker.container.id}/${docker.container.id}-json.log

Related issues

@blakerouse
[Elastic Agent] Add docker composable dynamic provider. (elastic#20842)
cf3da16 
* Add docker provider.

* Add changelog.

* Update docker start message to info.

(cherry picked from commit f017e24)
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 16, 2020

Pinging @elastic/ingest-management (Team:Ingest Management)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 16, 2020
@blakerouse blakerouse self-assigned this Sep 16, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 16, 2020 •
edited by jenkins-beats-ci bot
Loading

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21117 updated]

  • Start Time: 2020-09-16T16:55:12.641+0000

  • Duration: 124 min 44 sec

Log output

Expand to view the last 100 lines of log output 

[2020-09-16T18:57:50.337Z] + '[' -z '' ']'
[2020-09-16T18:57:50.337Z] + git config user.email beatsmachine@users.noreply.github.com
[2020-09-16T18:57:50.337Z] + git config user.name beatsmachine
[2020-09-16T18:57:50.511Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21117/src/github.com/elastic/beats/x-pack/elastic-agent
[2020-09-16T18:57:51.213Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-09-16T18:57:51.235Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-09-16T18:57:51.291Z] + mage build unitTest
[2020-09-16T18:57:51.300Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-21117-2/source/source.tgz
[2020-09-16T18:57:52.981Z] Body did not finish within grace period; terminating with extreme prejudice
[2020-09-16T18:57:53.059Z] Failed in branch Elastic Agent x-pack Windows
[2020-09-16T18:57:52.979Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-21117-2/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-21117/source.tgz
[2020-09-16T18:57:53.981Z] Docker machine "default" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
[2020-09-16T18:57:53.982Z] Docker machine "default" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
[2020-09-16T18:57:53.982Z] Client: Docker Engine - Community
[2020-09-16T18:57:53.982Z]  Version:           19.03.1
[2020-09-16T18:57:53.982Z]  API version:       1.40
[2020-09-16T18:57:53.982Z]  Go version:        go1.12.5
[2020-09-16T18:57:53.982Z]  Git commit:        74b1e89
[2020-09-16T18:57:53.982Z]  Built:             Thu Jul 25 21:18:17 2019
[2020-09-16T18:57:53.982Z]  OS/Arch:           darwin/amd64
[2020-09-16T18:57:53.982Z]  Experimental:      false
[2020-09-16T18:57:53.982Z] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[2020-09-16T18:57:53.982Z] It requires Docker daemon to be installed and running
[2020-09-16T18:57:54.101Z] Failed in branch Elastic Agent x-pack
[2020-09-16T18:57:54.924Z] + python .ci/scripts/pre_archive_test.py
[2020-09-16T18:57:55.097Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21117/src/github.com/elastic/beats/build
[2020-09-16T18:57:55.176Z] Recording test results
[2020-09-16T18:57:55.235Z] None of the test reports contained any result
[2020-09-16T18:57:55.298Z] Stashed 0 file(s)
[2020-09-16T18:57:55.361Z] Archiving artifacts
[2020-09-16T18:57:56.253Z] + python .ci/scripts/search_system_tests.py
[2020-09-16T18:57:56.478Z] [INFO] system-tests=''. If no empty then let's create a tarball
[2020-09-16T18:57:57.710Z] Post stage
[2020-09-16T18:57:57.719Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21117/src/github.com/elastic/beats
[2020-09-16T18:57:58.665Z] Starting "default"...
[2020-09-16T18:58:01.631Z] (default) Check network to re-create if needed...
[2020-09-16T18:58:01.631Z] Error setting up host only network on machine start: VirtualBox is configured with multiple host-only adapters with the same IP "192.168.33.1". Please remove one
[2020-09-16T18:58:01.631Z] Error checking TLS connection: Host is not running
[2020-09-16T18:58:01.631Z] Client: Docker Engine - Community
[2020-09-16T18:58:01.631Z]  Version:           19.03.1
[2020-09-16T18:58:01.631Z]  API version:       1.40
[2020-09-16T18:58:01.631Z]  Go version:        go1.12.5
[2020-09-16T18:58:01.631Z]  Git commit:        74b1e89
[2020-09-16T18:58:01.631Z]  Built:             Thu Jul 25 21:18:17 2019
[2020-09-16T18:58:01.631Z]  OS/Arch:           darwin/amd64
[2020-09-16T18:58:01.631Z]  Experimental:      false
[2020-09-16T18:58:01.631Z] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[2020-09-16T18:58:01.631Z] It requires Docker daemon to be installed and running
[2020-09-16T18:58:10.711Z] Failed in branch Elastic Agent Mac OS X
[2020-09-16T18:58:19.464Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-09-16T18:58:19.488Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-09-16T18:58:19.546Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-21117-2/source/source.tgz
[2020-09-16T18:58:19.568Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-21117-2/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-21117/source.tgz
[2020-09-16T18:58:27.081Z] + tar --version
[2020-09-16T18:58:27.380Z] + tar -xpf source.tgz
[2020-09-16T18:58:37.683Z] + rm source.tgz
[2020-09-16T18:58:37.697Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21117/src/github.com/elastic/beats
[2020-09-16T18:58:37.721Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21117/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-09-16T18:58:38.146Z] + cat
[2020-09-16T18:58:38.146Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-16T18:58:38.146Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-16T18:58:44.720Z] runbld>>> runbld started
[2020-09-16T18:58:44.720Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-16T18:58:45.660Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-16T18:58:45.660Z] runbld>>> Matches in the system config:
[2020-09-16T18:58:45.660Z] runbld>>> - Matched ^elastic\+beats
[2020-09-16T18:58:45.660Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-16T18:58:47.040Z] runbld>>> Debug logging enabled.
[2020-09-16T18:58:47.040Z] runbld>>> Storing result
[2020-09-16T18:58:47.040Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-16T18:58:47.040Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200916185846-88D27CC8
[2020-09-16T18:58:47.040Z] runbld>>> Adding system facts.
[2020-09-16T18:58:47.981Z] runbld>>> Adding vcs info for the latest commit:  8a29ca654ff10be66fbb67a058e56564fea7764f
[2020-09-16T18:58:47.981Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-16T18:58:47.981Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-16T18:58:47.981Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-16T18:58:47.981Z] Processing JUnit reports with runbld...
[2020-09-16T18:58:48.551Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-16T18:58:48.551Z] runbld>>> DURATION: 33ms
[2020-09-16T18:58:48.551Z] runbld>>> STDOUT: 40 bytes
[2020-09-16T18:58:48.551Z] runbld>>> STDERR: 49 bytes
[2020-09-16T18:58:48.551Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-16T18:58:48.551Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21117
[2020-09-16T18:58:49.496Z] runbld>>> Storing build metadata: 
[2020-09-16T18:58:49.496Z] runbld>>> Adding test report.
[2020-09-16T18:58:49.496Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21117/src/github.com/elastic/beats
[2020-09-16T18:58:50.066Z] runbld>>> Found 0 test output files
[2020-09-16T18:58:50.066Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 0 Skipped: 0
[2020-09-16T18:58:50.327Z] runbld>>> Storing result
[2020-09-16T18:58:50.587Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-16T18:58:50.587Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200916185846-88D27CC8
[2020-09-16T18:58:50.587Z] runbld>>> Email notification disabled by environment variable.
[2020-09-16T18:58:50.587Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-16T18:58:56.043Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-21117
[2020-09-16T18:58:56.152Z] [INFO] getVaultSecret: Getting secrets
[2020-09-16T18:58:56.234Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-16T18:58:56.982Z] + chmod 755 generate-build-data.sh
[2020-09-16T18:58:56.982Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21117/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21117/runs/2 ABORTED 7424081
[2020-09-16T18:58:56.982Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21117/runs/2/steps/?limit=10000 -o steps-info.json
[2020-09-16T18:58:57.232Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21117/runs/2/tests/?status=FAILED -o tests-errors.json

@blakerouse
Copy link
Copy Markdown
Contributor Author

blakerouse commented Sep 16, 2020

Includes #21118

@blakerouse blakerouse merged commit c4121e5 into elastic:7.x Sep 16, 2020
@blakerouse blakerouse deleted the backport_20842_7.x branch September 16, 2020 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@blakerouse blakerouse

Labels

backport [zube]: In Review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@blakerouse @elasticmachine @kaiyan-sheng