Skip to content

[Filebeat] Make set processor with copy_from compatible with ES < 7.13#26593

Closed
andrewkroh wants to merge 1 commit intoelastic:masterfrom
andrewkroh:bugfix/fb/copy-from-compat
Closed

[Filebeat] Make set processor with copy_from compatible with ES < 7.13#26593
andrewkroh wants to merge 1 commit intoelastic:masterfrom
andrewkroh:bugfix/fb/copy-from-compat

Conversation

@andrewkroh
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh commented Jun 29, 2021

What does this PR do?

Automatically rewrite Ingest Node set processors that use copy_from when connected to
Elasticsearch versions less than 7.13. The copy_from is replaced with {{{$copy_from value}}}.
The triple brace ensures the behavior is the same as copy_from in that no escaping is added.

Why is it important?

Ensures compatibility with earlier ES versions.

Checklist

  • My code follows the style guidelines of this project
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@andrewkroh andrewkroh added bug review Filebeat Filebeat Team:Security-External Integrations backport-v7.13.0 Automated backport with mergify backport-v7.14.0 Automated backport with mergify labels Jun 29, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 29, 2021

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 29, 2021
@andrewkroh andrewkroh force-pushed the bugfix/fb/copy-from-compat branch 2 times, most recently from d2da7be to 407f7c7 Compare June 29, 2021 23:02
@andrewkroh
Make set processor with copy_from compatible with ES < 7.13
2b94416 
Automatically rewrite Ingest Node `set` processors that use `copy_from` when connected to
Elasticsearch versions less than 7.13. The copy_from is replaced with `{{{$copy_from value}}}`.
The triple brace ensures the behavior is the same as copy_from in that no escaping is added.
@andrewkroh andrewkroh force-pushed the bugfix/fb/copy-from-compat branch from 407f7c7 to 2b94416 Compare June 29, 2021 23:03
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 30, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26593 updated

  • Start Time: 2021-06-29T23:04:16.872+0000

  • Duration: 100 min 7 sec

  • Commit: 2b94416

Test stats 🧪

Test Results
Failed 0
Passed 14150
Skipped 2311
Total 16461

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 14150
Skipped 2311
Total 16461

@adriansr
Copy link
Copy Markdown
Contributor

adriansr commented Jun 30, 2021 •
edited
Loading

I think this can break some of the usages we have:

panw.panos copying an object instead of a string:

beats/x-pack/filebeat/module/panw/panos/ingest/pipeline.yml

Lines 465 to 468 in 895f55f

- set:
field: user
copy_from: source.user
if: "ctx?.source?.user != null"

cyberarkpas.audit doing something weird I wrote:

beats/x-pack/filebeat/module/cyberarkpas/audit/ingest/pipeline.yml

Lines 942 to 950 in 895f55f

- foreach:
field: _tmp.values
ignore_missing: true
processor:
set:
field: '{{{_ingest._value.to}}}'
copy_from: '_ingest._value.value'
ignore_empty_value: true
override: true

Do we currently run the tests on ES 7.11 by any chance?

I'll rewrite those two without using copy_from.

@andrewkroh andrewkroh mentioned this pull request Jun 30, 2021
Closed
@andrewkroh
Copy link
Copy Markdown
Member Author

andrewkroh commented Jun 30, 2021

Thanks for raising the issue with object types. Since this solution won't work for all of the corner cases I have opened #26629 to discuss how to handle this.

I had considered replace set with a script to accomplish the copying of objects, but in one of the cases above this would not work because you cannot access _ingest from the script context.

@andrewkroh andrewkroh closed this Jun 30, 2021
@andrewkroh andrewkroh mentioned this pull request Jun 30, 2021
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport-v7.13.0 Automated backport with mergify backport-v7.14.0 Automated backport with mergify bug Filebeat Filebeat review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@andrewkroh @elasticmachine @adriansr