Skip to content

Cherry-pick #20839 to 7.x: [Elastic Agent] Add support for variable replacement from providers#20964

Merged
blakerouse merged 1 commit intoelastic:7.xfrom
blakerouse:backport_20839_7.x
Sep 7, 2020
Merged

Cherry-pick #20839 to 7.x: [Elastic Agent] Add support for variable replacement from providers#20964
blakerouse merged 1 commit intoelastic:7.xfrom
blakerouse:backport_20839_7.x

Conversation

@blakerouse
Copy link
Copy Markdown
Contributor

@blakerouse blakerouse commented Sep 3, 2020

Cherry-pick of PR #20839 to 7.x branch. Original message:

What does this PR do?

Adds support for replacing {{local_dynamic.key|local_dynamic.other|'fallback'}} inside of the Elastic Agent inputs configuration.

Why is it important?

Finishes the core support for #20781

More code is needed to help debugging effort.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Running the Elastic Agent in local mode, use the following configuration:

outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    username: elastic
    password: changeme

providers:
  local_dynamic:
    items:
      - vars:
          key: value1
        processors:
          - add_fields:
              fields:
                custom: match1
              target: dynamic
      - vars:
          key: value2
        processors:
          - add_fields:
              fields:
                custom: match2
              target: dynamic
      - vars:
          key: value3
        processors:
          - add_fields:
              fields:
                custom: match3
              target: dynamic

inputs:
  - type: logfile
    enabled: true
    streams:
      - paths:
          - /var/log/{{local_dynamic.key}}

Then run the following to inspect the generated configuration:

$ ./elastic_agent inspect output -o default
[default] filebeat:
filebeat:
  inputs:
  - index: logs-generic-default
    paths:
    - /var/log/value1
    processors:
    - add_fields:
        fields:
          custom: match1
        target: dynamic
    - add_fields:
        fields:
          dataset: generic
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: generic
        target: event
    type: log
  - index: logs-generic-default
    paths:
    - /var/log/value2
    processors:
    - add_fields:
        fields:
          custom: match2
        target: dynamic
    - add_fields:
        fields:
          dataset: generic
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: generic
        target: event
    type: log
  - index: logs-generic-default
    paths:
    - /var/log/value3
    processors:
    - add_fields:
        fields:
          custom: match3
        target: dynamic
    - add_fields:
        fields:
          dataset: generic
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: generic
        target: event
    type: log
output:
  elasticsearch:
    hosts:
    - 127.0.0.1:9200
    password: changeme
    username: elastic

---
[default] FLEET_MONITORING:
output:
  elasticsearch:
    hosts:
    - 127.0.0.1:9200
    password: changeme
    type: elasticsearch
    username: elastic
programs:
- filebeat

---

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 3, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 3, 2020

Pinging @elastic/ingest-management (Team:Ingest Management)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 3, 2020
@blakerouse blakerouse self-assigned this Sep 3, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 3, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20964 updated]

  • Start Time: 2020-09-07T13:54:01.458+0000

  • Duration: 34 min 42 sec

Test stats 🧪

Test Results
Failed 0
Passed 1691
Skipped 3
Total 1694

michalpristas
michalpristas approved these changes Sep 7, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@michalpristas michalpristas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backport looks ok

@blakerouse
[Elastic Agent] Add support for variable replacement from providers (e…
2a39313 
…lastic#20839)

* Add ability to replace variables in the parsed AST tree.

* More vars replace improvements.

* Perform the variable replacement of the elastic-agent configuration.

* Clean-up testing and processors.

* Add changelog.

* Fix import sorting.

* Add more validation to variable substitution.

* Add log message about dynamic inputs being experimental.

* Update to new variable format. Handle replace of complete objects.

* Fix config importing to not replace vars in inputs.

* Fixes for vet.

* Fix fleet config change action to use new LoadConfig.

* Fixes from code review.

* Ensure processors are prepended to inputs.

(cherry picked from commit 121f23b)
@blakerouse blakerouse merged commit 3409acd into elastic:7.x Sep 7, 2020
@blakerouse blakerouse deleted the backport_20839_7.x branch September 7, 2020 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@michalpristas michalpristas michalpristas approved these changes

Assignees

@blakerouse blakerouse

Labels

backport [zube]: In Review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@blakerouse @elasticmachine @michalpristas