Overview

To help support dynamic inputs elastic/beats#19225 Elastic Agent needs to add the ability to debug the providers using for variable substitution. This issue is to track the debugging effort, for information about variable substitution review elastic/beats#20781

Debugging

This obviously adds a lot of confusion to what the resulting configuration that Elastic Agent will be running with. To ensure that the feature is deployed correctly and that providers are working as expected debugging needs to be a top priority in the implementation.

Debugging the running daemon

With the new ability to communicate with the running daemon the inspect command should be changed to talk to the running daemon and return the current configuration that is being used in memory. This will ensure that with running providers like Docker and Kubernetes it is easy to inspect what the resulting configuration is.

The current inspect and output commands can be combined and moved under the debug subcommand. (Note: This is not connecting to the currently running Elastic Agent)

$ ./elastic-agent debug config

Possible to watch the configuration as changes come in with --watch.

$ ./elastic-agent debug config --watch

Debugging a single provider

A new debug command should be implemented that runs a single provider and output what it's currently providing back to the Elastic Agent. (Note: This is not connecting to the currently running Elastic Agent)

Example outputting docker provider inventory key/value mappings:

$ ./elastic-agent debug provider docker 
{"id": "1",  "mapping": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}}, "processors": {"add_fields": {"container.name": "my-container"}},}
{"id": "2", "mapping": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}}, "processors": {"add_fields": {"container.name": "other-container"}},}
{"id": "2", "mapping": nil}

Example rendering configurations with changes:

$ ./elastic-agent debug provider docker -c testing-conf.yml
# {"id": "1",  "mapping": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}}, "processors": {"add_fields": {"container.name": "my-container"}}}
inputs:
  - type: logfile
    path: /var/log/containers/1.log
    processors:
      - add_fields:
          container.name: my-container
# {"id": "2", "mapping": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}}, "processors": {"add_fields": {"container.name": "other-container"}}}
inputs:
  - type: logfile
    path: /var/log/containers/1.log
    processors:
      - add_fields:
          container.name: my-container
  - type: logfile
    path: /var/log/containers/2.log
    processors:
      - add_fields:
          container.name: other-container
# {"id": "2", "mapping": nil}
inputs:
  - type: logfile
    path: /var/log/containers/1.log