[Elastic Agent] Add docker composable dynamic provider. by blakerouse · Pull Request #20842 · elastic/beats

blakerouse · 2020-08-28T01:21:14Z

What does this PR do?

Adds a docker composable dynamic provider.

Why is it important?

Support generating inputs from discovered docker containers.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Using the following configuration results in the inputs results in a new input per container.

inputs:
  - type: logfile
     streams:
      - path: /var/lib/docker/containers/${docker.container.id}/${docker.container.id}-json.log

Related issues

Relates Cherry-pick #16883 to 7.x: Use pytest instead of nosetests #20478

elasticmachine · 2020-08-28T01:22:59Z

Pinging @elastic/ingest-management (Team:Ingest Management)

elasticmachine · 2020-08-28T01:59:25Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request elastic/beats#20842 updated]
Start Time: 2020-09-08T15:07:43.574+0000
Duration: 34 min 56 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1691
Skipped	6
Total	1697

ruflin

Nice. How similar is this code from our Beats autodiscovery code?

I really like that this allows us to start playing around with it.

ruflin · 2020-08-31T08:14:54Z

x-pack/elastic-agent/pkg/composable/providers/docker/docker.go

+
+	if err := watcher.Start(); err != nil {
+		// warn only; return nil (do nothing)
+		c.logger.Warnf("Failed starting docker provider: %s", err)


On the Beats code based we tried no to use Warn as it is this strange middleground between info and error. Is user interaction needed or not? Is it a bug or not? I would suggest to move this to the Info level and phrase it more something like: Docker provider not started as docker not available (if we can make this distinction).

blakerouse · 2020-08-31T12:55:14Z

Nice. How similar is this code from our Beats autodiscovery code?

This is very, very similar. Because the docker code was so well encapsulated in libbeat/common/docker.NewWatcher it was very easy to add. That is also why this module can easily be implemented in less than 200 lines of code.

I really like that this allows us to start playing around with it.

ph · 2020-08-31T13:02:03Z

cc @exekias You might want to watch this :)

exekias · 2020-09-02T08:17:36Z

Nice job! Maybe this means we don't really need to refactor current providers #19271?

I see it's possible to pass the config for the provider, how would that look in the YAML?

blakerouse · 2020-09-02T12:30:08Z

@exekias I think creating the new providers in Agent will be the most amount of work, but if most of the watcher logic is in libbeat then Agent can easily pull it in and use it.

For configuring it looks like:

providers:
  docker:
    host: "unix:///var/run/docker.sock"
    tls:
      verification_mode: none

x-pack/elastic-agent/pkg/composable/providers/docker/docker.go

ruflin · 2020-09-16T14:12:17Z

x-pack/elastic-agent/pkg/composable/providers/docker/docker.go

+}
+
+// Run runs the environment context provider.
+func (c *dynamicProvider) Run(comm composable.DynamicProviderComm) error {


Nit: I would call it Start instead of Run as it is non blocking, but we change that later.

Its not called Start because there is no Stop. It is stopped only by the cancelling of the passed context. It is up to the provider on how it wants to implement. Example is the local_dynamic provider it just reads the config and returns, as it does need a goroutine to always be running.

ruflin · 2020-09-16T14:13:32Z

x-pack/elastic-agent/pkg/composable/providers/docker/docker.go

+						"id":     container.ID,
+						"name":   container.Name,
+						"image":  container.Image,
+						"labels": processorLabelMap,


We should revisit later on what the default info is that we ship, if we potentially should reduce it.

Works for me.

@blakerouse Could you follow up with an issue? I'm pretty sure we will forget otherwise :-D

Filed #21135

* Add docker provider. * Add changelog. * Update docker start message to info. (cherry picked from commit f017e24)

…mic provider. (#21117) * [Elastic Agent] Add docker composable dynamic provider. (#20842) * Add docker provider. * Add changelog. * Update docker start message to info. (cherry picked from commit f017e24) * Fix docker provider builder.

…ne-2.0 * upstream/master: (44 commits) Update users.asciidoc (elastic#20802) (elastic#21108) Fix docker provider builder. (elastic#21118) [Elastic Agent] Add docker composable dynamic provider. (elastic#20842) Add new modules/filesets from rsa2elk for 7.10 (elastic#20820) Fix broken links to external websites (elastic#21061) [docs] typo in the command line (elastic#20799) [Filebeat] add panos type and sub_type (elastic#20912) Move the `compute_vm_scalset` to a light metricset and map the cloud metadata (elastic#21038) [Filebeat] Add support for Cloudtrail digest files (elastic#21086) Add metrics collection from cost explorer into aws/billing metricset (elastic#20527) Add vendoring to Google Cloud Functions again (elastic#21070) [Elastic Agent] Add fleet.host.id for sending to endpoint. (elastic#21042) Do not need Google credentials before using it (elastic#21072) [Filebeat][New Module] Zoom webhook module (elastic#20414) Add support for GMT timezone offset in decode_cef (elastic#20993) Filebeat: Fix random error on harvester close (elastic#21048) Add ingress controller dashboards (elastic#21052) Fix loggers in composable module. (elastic#21047) [Ingest Manager] Increase kibana client timeout to 5 minutes (elastic#21037) Add changelog. (elastic#21041) ...

Add docker provider.

7382256

blakerouse added the Team:Ingest Management label Aug 28, 2020

blakerouse self-assigned this Aug 28, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 28, 2020

Add changelog.

f514945

blakerouse marked this pull request as ready for review August 28, 2020 01:22

ruflin reviewed Aug 31, 2020

View reviewed changes

Update docker start message to info.

e0a5ade

ChrsMark reviewed Sep 8, 2020

View reviewed changes

x-pack/elastic-agent/pkg/composable/providers/docker/docker.go Show resolved Hide resolved

Merge branch 'master' into agent-docker-provider

f8ff54f

ruflin approved these changes Sep 16, 2020

View reviewed changes

blakerouse merged commit f017e24 into elastic:master Sep 16, 2020

blakerouse deleted the agent-docker-provider branch September 16, 2020 16:34

blakerouse mentioned this pull request Sep 16, 2020

Cherry-pick #20842 to 7.x: [Elastic Agent] Add docker composable dynamic provider. #21117

Merged

4 tasks

blakerouse added the v7.10.0 label Sep 16, 2020

blakerouse added a commit to blakerouse/beats that referenced this pull request Sep 16, 2020

[Elastic Agent] Add docker composable dynamic provider. (elastic#20842)

cf3da16

* Add docker provider. * Add changelog. * Update docker start message to info. (cherry picked from commit f017e24)

This was referenced Mar 7, 2022

[Elastic Agent] Look into reducing the docker provider add_fields data elastic/elastic-agent#124

Open

[Elastic Agent] Add support for dynamics inputs #19225

Closed

EricDavisX mentioned this pull request Nov 5, 2020

[Ingest Manager] Add test for dynamic-input (conditionals) from 7.10 feature (for stand-alone Agent mode or Linux specific tests for Fleet mode) elastic/e2e-testing#424

Closed

Conversation

blakerouse commented Aug 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Aug 28, 2020

Uh oh!

elasticmachine commented Aug 28, 2020 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blakerouse commented Aug 31, 2020

Uh oh!

ph commented Aug 31, 2020

Uh oh!

exekias commented Sep 2, 2020

Uh oh!

blakerouse commented Sep 2, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

blakerouse commented Aug 28, 2020 •

edited

Loading

elasticmachine commented Aug 28, 2020 •

edited by jenkins-beats-ci bot

Loading