Skip to content

Cherry-pick #20908 to 7.9: [Filebeat] backwards compatibility for set processor#21084

Merged
leehinman merged 2 commits intoelastic:7.9from
andrewkroh:backport_20908_7.9
Sep 15, 2020
Merged

Cherry-pick #20908 to 7.9: [Filebeat] backwards compatibility for set processor#21084
leehinman merged 2 commits intoelastic:7.9from
andrewkroh:backport_20908_7.9

Conversation

@andrewkroh
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh commented Sep 14, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #20908 to 7.9 branch. Original message:

What does this PR do?

When loading a pipeline this change checks the elasticsearch version
and if the version is less than 7.9.0 it will replace the
"ignore_empty_value" option with an equivalent if statement on the set
processor.

Why is it important?

This allows filebeat > 7.9.0 to be used with older versions of
elasticsearch. Without it the pipelines fail to load because the
option isn't supported.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

run filebeat modules enable zeek && filebeat -e setup

Logs

Error

2020-09-01T12:42:48.781-0500	ERROR	fileset/setup.go:81	Error loading pipeline: 1 error: Error loading pipeline for fileset zeek/x509: couldn't load pipeline: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause [{"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]",
...

With fix:

2020-09-01T14:04:56.470-0500	DEBUG	[modules] fileset/pipelines.go:277     	in pipeline filebeat-8.0.0-zeek-x509-pipeline replacing unsupported 'ignore_empty_value' with if ctx?.zeek?.x509?.certificate?.subject?.state != null in set processor

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 14, 2020

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 14, 2020
@leehinman @andrewkroh
[Filebeat] backwards compatibility for set processor (elastic#20908)
3cea2d0 
- "ignore_empty_value" option for the set processor only works on
Elasticsearch >= 7.9.0.  This change removes that option and replaces
it with an if statement if pipeline is loaded on an earlier version of
elasticsearch.

(cherry picked from commit 0032c0c)
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 14, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21084 updated]

  • Start Time: 2020-09-15T13:34:48.252+0000

  • Duration: 61 min 31 sec

Test stats 🧪

Test Results
Failed 0
Passed 5563
Skipped 824
Total 6387

leehinman
leehinman approved these changes Sep 15, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewkroh thanks for catching this backport

@leehinman leehinman merged commit 5a06f7e into elastic:7.9 Sep 15, 2020
@zube zube bot removed the [zube]: Done label Dec 15, 2020
@andrewkroh andrewkroh deleted the backport_20908_7.9 branch January 14, 2022 14:08
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
@andrewkroh @leehinman
[Filebeat] backwards compatibility for set processor (elastic#20908) (e…
d15f604 
…lastic#21084)

- "ignore_empty_value" option for the set processor only works on
Elasticsearch >= 7.9.0.  This change removes that option and replaces
it with an if statement if pipeline is loaded on an earlier version of
elasticsearch.

(cherry picked from commit b6162c4)

Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leehinman leehinman leehinman approved these changes

Assignees

No one assigned

Labels

backport

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@andrewkroh @elasticmachine @leehinman