Overview

To enable dynamic inputs #19225 Elastic Agent needs to support variable substitution. The key/values for the variable substitution come from sources known as providers. This issue is to add support for both the required variables and providers, the first part of handling dynamic inputs.

Providers

Providers provide the key/values that can be used in the variable substitution. Each providers keys are automatically prefixed with the name of the provider in the context of the Elastic Agent. This removes the requirement of worrying about conflicts/overwriting and any type of undetermined behavior.

Example if a provider named foo provides {"key1": "value1", "key2": "value2"} it would be placed in {"foo" : {"key1": "value1", "key2": "value2"}}. Allowing it to be referenced as {{foo.key1}} and {{foo.key2}}.

After discussions its clear that 2 different types of provides need to be supported.

Context Providers

Context providers provide the current context of the running Elastic Agent. Example is Agent information (id, version, etc.), Host information (hostname, IP addresses), Environment information (environment variables).

They can only provide a single key/value mapping. Think of them as singletons, an update of a key/value mapping will result in a re-evaluation of the entire configuration. These provides are normally very static, but it is not required that they behave in that manner. It is possible for a value to change resulting in re-evaluation.

ECS naming should be used for context providers when possible to ensure that documentation and understanding across projects for Elastic is clear and understandable.

Initial Set

• agent - provides agent information
  • id - current agent ID
  • version - current agent version information object
    • version - version string
    • commit - version commit
    • build_time - version build time
    • snapshot - version is snapshot build
• host - provides host information
  • name - host hostname
  • platform - host platform
  • architecture - host architecture
  • ip[] - host IP addresses
  • mac[] - host MAC addresses
• env - provides environment variables
  • key/values from current environment variables
• local - provides a configurable static mapping
  • vars - key/values come from the configuration user set in elastic-agent.yml

Dynamic Providers

Dynamic providers are different then context provides in that they actually provide an array of multiple key/value mappings. Each key/value mapping is combined with the previous context providers key/value mapping providing a new unique key/value mapping that is then used to generate a configuration.

Each unique mapping must provide a unique ID for that mapping. This allows the provider to modify the data of an already provided mapping or remove a mapping. This is represented below with the objects of .id and .mapping.

Each unique mapping can also provide processors on the input. This only applied in the case that a substitution was made on that input. Look at processors from the Docker provider example.

Below shows the flow of how this will work synchronously (it will be implemented asynchronously, so changes are handled as they occur):

config := getConfig()   // current unparsed Elastic Agent configuration
current := getContext()  // current contexts from all the Context Provides
for _, provider := range providers {
    for _, mapping := range provider.GetMappings() {
        providerContext := duplicateContext(current)  // duplicate current context
        providerContext.Merge(mapping)  // merge the current key/value
        newConfig := parseConfig(config, providerContext) // parse the config using the new key/value mapping
        //
        // use the new config to create inputs
        //
    }
}

To give a good example of this would be with a Docker dynamic provider. Imagine that the Docker provider provides the following:

[
    {
       "id": "1",
       "mapping:": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}},
       "processors": {"add_fields": {"container.name": "my-container"}},
    },
    {
        "id": "2",
        "mapping": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}},
        "processors": {"add_fields": {"container.name": "other-container"}},
    },
]

Elastic Agent automatically prefixes the result with docker.

[
    {"docker": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}}},
    {"docker": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}},
]

With the following defined configuration in Elastic Agent:

inputs:
  - type: logfile
    path: "${docker.paths.log}"

This would result in the following generated configuration:

inputs:
  - type: logfile
    path: "/var/log/containers/1.log"
    processors:
      - add_fields:
          container.name: my-container
  - type: logfile
    path: "/var/log/containers/2.log"
    processors:
      - add_fields:
          container.name: other-container

Initial Set

• docker - provides inventory of the docker
  • id - ID of the container
  • cmd - Arg path of container
  • name - Name of the container
  • image - Image of the container
  • labels - Labels of the container
  • ports - Ports of the container
  • paths - Object of paths for the container
    • log - Log path of the container
• local_dynamic - provides a static configuration of dynamic mappings
  • vars - List of key/value mappings

Variable Substitution

To align with similar syntax as Beats configuration variable substitution using ${ var } will be used.

When an input uses a variable substitution that is not present in the current key/value mapping being evaluated that input is removed in the result.

inputs:
  - type: logfile
    path: "/var/log/foo"
  - type: logfile
    path: "${ unknown.key }"

Result because no unknown.key exists:

inputs:
  - type: logfile
    path: "/var/log/foo"

Variable substitution can also define alternative variables or a constant. A constant must be defined with either ' or ". Once a constant is reached in the substitution evaluation of any remaining or variables will be ignored, so a constant should really be the last entry in the substitution. Defining alternatives is done with | followed by the next variable or constant. The power comes from allowing the input to defined the preference order of the substitution by chaining multiple |..var.. together.

inputs:
  - type: logfile
    path: "/var/log/foo"
  - type: logfile
    path: "${docker.paths.log|kubernetes.container.paths.log|'/var/log/other'}"

NOTE: Because ${ } will collide with go-ucfg library that Elastic Agent uses to parse the configuration file. Variable parsing by go-ucfg will be disabled for all of Elastic Agent.

Configuring

Configuring providers comes from the top-level key of providers in the elastic-agent.yml configuration. By default all registered providers are enabled, if they cannot connect (in docker case) they just produce no mappings.

providers:
  local:
    vars:
      foo: bar
  local_dynamic:
    vars:
      - item: key1
      - item: key2

A provider can be explicitly disabled with enabled: false when defined, and because all providers are prefixed and do not have a collision the name of the provider is the key in the configuration.

providers:
  docker:
    enabled: false

Debugging

Moved to elastic/elastic-agent#123