Skip to content

Cherry-pick #20908 to 7.x: [Filebeat] backwards compatibility for set processor#20969

Merged
leehinman merged 2 commits intoelastic:7.xfrom
leehinman:backport_20908_7.x
Sep 15, 2020
Merged

Cherry-pick #20908 to 7.x: [Filebeat] backwards compatibility for set processor#20969
leehinman merged 2 commits intoelastic:7.xfrom
leehinman:backport_20908_7.x

Conversation

@leehinman
Copy link
Copy Markdown
Contributor

@leehinman leehinman commented Sep 3, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #20908 to 7.x branch. Original message:

What does this PR do?

When loading a pipeline this change checks the elasticsearch version
and if the version is less than 7.9.0 it will replace the
"ignore_empty_value" option with an equivalent if statement on the set
processor.

Why is it important?

This allows filebeat > 7.9.0 to be used with older versions of
elasticsearch. Without it the pipelines fail to load because the
option isn't supported.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

run filebeat modules enable zeek && filebeat -e setup

Logs

Error

2020-09-01T12:42:48.781-0500	ERROR	fileset/setup.go:81	Error loading pipeline: 1 error: Error loading pipeline for fileset zeek/x509: couldn't load pipeline: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause [{"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]",
...

With fix:

2020-09-01T14:04:56.470-0500	DEBUG	[modules] fileset/pipelines.go:277     	in pipeline filebeat-8.0.0-zeek-x509-pipeline replacing unsupported 'ignore_empty_value' with if ctx?.zeek?.x509?.certificate?.subject?.state != null in set processor

@leehinman
[Filebeat] backwards compatibility for set processor (elastic#20908)
9ba271b 
- "ignore_empty_value" option for the set processor only works on
Elasticsearch >= 7.9.0.  This change removes that option and replaces
it with an if statement if pipeline is loaded on an earlier version of
elasticsearch.

(cherry picked from commit 0032c0c)
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 3, 2020

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 3, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 3, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20969 opened]

  • Start Time: 2020-09-03T18:09:02.968+0000

  • Duration: 59 min 40 sec

Test stats 🧪

Test Results
Failed 0
Passed 5579
Skipped 833
Total 6412

@leehinman leehinman merged commit ec794aa into elastic:7.x Sep 15, 2020
@leehinman leehinman deleted the backport_20908_7.x branch October 5, 2020 19:06
@zube zube bot removed the [zube]: Done label Dec 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees

No one assigned

Labels

backport

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@leehinman @elasticmachine @andrewkroh