Skip to content

[Ingest] Support input-level config fields#60594

Merged
jen-huang merged 5 commits intoelastic:masterfrom
jen-huang:ingest/input-vars
Mar 20, 2020
Merged

[Ingest] Support input-level config fields#60594
jen-huang merged 5 commits intoelastic:masterfrom
jen-huang:ingest/input-vars

Conversation

@jen-huang
Copy link
Copy Markdown
Contributor

@jen-huang jen-huang commented Mar 19, 2020

Summary

Previously, all input-level vars defined on a package were copied into each stream. This PR changes that so that input level fields stay at the input level and do not get copied into each stream.

Example

With system, AWS, and nginx datasources (using package defaults for everything). AWS datasource has input-level fields:

Full agent config YAML example 
id: 16096a20-6974-11ea-accc-3fa5760bb7e7
outputs:
  default:
    type: elasticsearch
    hosts:
    - http://localhost:9200
    api_key: ''
datasources:
- id: system-1
  enabled: true
  use_output: default
  inputs:
  - type: system/metrics
    enabled: true
    streams:
    - id: system/metrics-system.core
      enabled: true
      dataset: system.core
      period: 10s
      metrics:
      - percentages
    - id: system/metrics-system.cpu
      enabled: true
      dataset: system.cpu
      period: 10s
      metrics:
      - percentages
      - normalized_percentages
    - id: system/metrics-system.diskio
      enabled: true
      dataset: system.diskio
      period: 10s
      include_devices: []
    - id: system/metrics-system.entropy
      enabled: true
      dataset: system.entropy
      period: 10s
      include_devices: []
    - id: system/metrics-system.filesystem
      enabled: true
      dataset: system.filesystem
      period: 1m
      ignore_types: []
    - id: system/metrics-system.fsstat
      enabled: true
      dataset: system.fsstat
      period: 1m
      ignore_types: []
    - id: system/metrics-system.load
      enabled: true
      dataset: system.load
      period: 10s
    - id: system/metrics-system.memory
      enabled: true
      dataset: system.memory
      period: 10s
    - id: system/metrics-system.network
      enabled: true
      dataset: system.network
      period: 10s
    - id: system/metrics-system.network_summary
      enabled: true
      dataset: system.network_summary
      period: 10s
    - id: system/metrics-system.process
      enabled: true
      dataset: system.process
      period: 10s
      processes:
      - ".*"
      include_top_n.enabled: true
      include_top_n.by_cpu: 5
      include_top_n.by_memory: 5
      cmdline.cache.enabled: true
      cgroups.enabled: true
      env.whitelist: []
      include_cpu_ticks: false
    - id: system/metrics-system.process_summary
      enabled: true
      dataset: system.process_summary
      period: 10s
    - id: system/metrics-system.raid
      enabled: true
      dataset: system.raid
      period: 10s
      mount_point: "/"
    - id: system/metrics-system.service
      enabled: true
      dataset: system.service
      period: 10s
      state_filter: []
    - id: system/metrics-system.socket_summary
      enabled: true
      dataset: system.socket_summary
      period: 10s
    - id: system/metrics-system.uptime
      enabled: true
      dataset: system.uptime
      period: 15m
    - id: system/metrics-system.users
      enabled: true
      dataset: system.users
      period: 10s
  - type: logs
    enabled: true
    streams:
    - id: logs-system.auth
      enabled: true
      dataset: system.auth
      paths:
      - "/var/log/auth.log*"
      - "/var/log/secure*"
    - id: logs-system.syslog
      enabled: true
      dataset: system.syslog
      paths:
      - "/var/log/messages*"
      - "/var/log/syslog*"
  package:
    name: system
    version: 0.9.0
- id: aws-1
  namespace: default
  enabled: true
  use_output: default
  inputs:
  - type: aws/metrics
    enabled: true
    streams:
    - id: aws/metrics-aws.billing
      enabled: true
      dataset: aws.billing
      period: 12h
      region:
      - us-east-1
    - id: aws/metrics-aws.cloudwatch
      enabled: true
      dataset: aws.cloudwatch
      period: 5m
      metrics:
      - namespace: AWS/EC2
        tags.resource_type_filter: ec2:instance
    - id: aws/metrics-aws.dynamodb
      enabled: true
      dataset: aws.dynamodb
      period: 5m
    - id: aws/metrics-aws.ebs
      enabled: true
      dataset: aws.ebs
      period: 5m
    - id: aws/metrics-aws.ec2
      enabled: true
      dataset: aws.ec2
      period: 5m
    - id: aws/metrics-aws.elb
      enabled: true
      dataset: aws.elb
      period: 1m
    - id: aws/metrics-aws.lambda
      enabled: true
      dataset: aws.lambda
      period: 5m
    - id: aws/metrics-aws.rds
      enabled: true
      dataset: aws.rds
      period: 5m
    - id: aws/metrics-aws.s3_daily_storage
      enabled: true
      dataset: aws.s3_daily_storage
      period: 24h
    - id: aws/metrics-aws.s3_request
      enabled: true
      dataset: aws.s3_request
      period: 24h
    - id: aws/metrics-aws.sns
      enabled: true
      dataset: aws.sns
      period: 5m
    - id: aws/metrics-aws.sqs
      enabled: true
      dataset: aws.sqs
      period: 5m
    - id: aws/metrics-aws.usage
      enabled: true
      dataset: aws.usage
      period: 1m
    - id: aws/metrics-aws.vpc
      enabled: true
      dataset: aws.vpc
      period: 1m
    access_key_id: ${AWS_ACCESS_KEY_ID:""}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY:""}
    AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN:""}
    credential_profile_name: test-mb
  package:
    name: aws
    version: 1.0.0
- id: nginx-1
  namespace: default
  enabled: true
  use_output: default
  inputs:
  - type: nginx/metrics
    enabled: true
    streams:
    - id: nginx/metrics-nginx.stubstatus
      enabled: true
      dataset: nginx.stubstatus
      period: 10s
    hosts:
    - http://127.0.0.1
  - type: logs
    enabled: true
    streams:
    - id: logs-nginx.access
      enabled: true
      dataset: nginx.access
      paths:
      - "/var/log/nginx/access.log*"
    - id: logs-nginx.error
      enabled: true
      dataset: nginx.error
      paths:
      - "/var/log/nginx/error.log*"
  package:
    name: nginx
    version: 1.2.0
revision: 4

@jen-huang jen-huang added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project Feature:Fleet Fleet team's agent central management project v7.7.0 labels Mar 19, 2020
@jen-huang jen-huang requested review from a team and ruflin March 19, 2020 03:35
@jen-huang jen-huang self-assigned this Mar 19, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 19, 2020

Pinging @elastic/ingest-management (Feature:Fleet)

ruflin
ruflin approved these changes Mar 20, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Output looks as expected.

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Mar 20, 2020

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jen-huang jen-huang merged commit b018322 into elastic:master Mar 20, 2020
@jen-huang jen-huang deleted the ingest/input-vars branch March 20, 2020 19:18
jen-huang added a commit to jen-huang/kibana that referenced this pull request Mar 20, 2020
@jen-huang
[Ingest] Support input-level config fields (elastic#60594)
a5e8836 
* Support input-level config fields

* Adjust tests

* Adjust server schema
@jen-huang jen-huang mentioned this pull request Mar 20, 2020
Merged
jen-huang added a commit that referenced this pull request Mar 21, 2020
@jen-huang
[Ingest] Support input-level config fields (#60594) (#60801)
5b217df 
* Support input-level config fields

* Adjust tests

* Adjust server schema
@jen-huang jen-huang added the Team:Fleet Team label for Observability Data Collection Fleet team label Mar 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ruflin ruflin ruflin approved these changes

Assignees

@jen-huang jen-huang

Labels

Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project Feature:Fleet Fleet team's agent central management project release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.7.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jen-huang @elasticmachine @kibanamachine @ruflin