Skip to content

Duplicate data collection between generic inputs and specific integrations #274

@ChrsMark

Description

@ChrsMark

Today, we have generic inputs like Container Logs or Custom Logs. Based on dynamic variables and various providers we can also set this generic inputs to collect logs from all of the log files in a specific path.
For example in Kubernetes (or similarly in Docker) we can set the path like /var/log/containers/*${kubernetes.container.id}.log and we will have the input to be set for all the containers based on their container.id value.

In addition we can specify

- name: nginx
  type: nginx/logs
  use_output: default
  data_stream:
    namespace: default
  streams:
    - data_stream:
        dataset: nginx.access
        type: logs
      paths:
        - '/var/log/containers/*${kubernetes.container.id}.log'
      condition: ${kubernetes.labels.app} == 'nginx'

which is an input based on a condition for nginx.
In the end we will have this Pod's logs to be collected twice, from the generic Container Logs input and one for the specific one for Nginx.

cc: @mukeshelastic

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions