Skip to content

Introducing preprocessors on the module input override#27154

Closed
mjmbischoff wants to merge 3 commits intoelastic:masterfrom
mjmbischoff:preprocessor
Closed

Introducing preprocessors on the module input override#27154
mjmbischoff wants to merge 3 commits intoelastic:masterfrom
mjmbischoff:preprocessor

Conversation

@mjmbischoff
Copy link
Copy Markdown
Contributor

@mjmbischoff mjmbischoff commented Jul 30, 2021

Introducing preprocessors on the module input override which are run after the input but before the processors in the module so some pre transformation can occur.

What does this PR do?

Allows preprocessors to be defined on the module input override which are run after the input but before the processors in the module as opposed to the processors defined at this level which are run after the module. The preprocessors are normal processors and after merging the config don't exist as a separate property but are merged with the processors.

Why is it important?

This PR introduces flexibility to filebeat and allows more module reuse vs local forking (breaking updates) and can simplify the ingest pipeline requiring fewer components

Checklist

  • My code follows the style guidelines of this project
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Use cases

When intermediate transport is used, for example syslog shipped over syslog, leading to double syslog headers which the module doesn't expect.

@mjmbischoff
Introducing preprocessors on the module input override which are run …
a3761d3 
…after the input but before the processors in the module so some pre transformation can occur.
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 30, 2021
@mjmbischoff mjmbischoff mentioned this pull request Jul 30, 2021
Closed
8 tasks
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jul 30, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-07-30T14:51:01.199+0000

  • Duration: 92 min 46 sec

  • Commit: dc5772b

Test stats 🧪

Test Results
Failed 0
Passed 14912
Skipped 2313
Total 17225

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 14912
Skipped 2313
Total 17225

kvch
kvch suggested changes Aug 5, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@kvch kvch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First I would refrain from merging this. I would like to see if parsers are good enough for inputs. In general, we do not want to have much processing in Beats. Everything should be done at the Ingest Node, if possible.

Also, I will bring this up in internal discussion with other devs on the team.

@mjmbischoff
Copy link
Copy Markdown
Contributor Author

mjmbischoff commented Aug 10, 2021

I'm afraid when the parsers can adres all the use-cases it will be a duplicated effort of the processors.
And with regard to processing, as long as as there is any processing in beats without this the data will be in the wrong structure and fail.

Let me know what the discussion yields.

@jsoriano jsoriano added the Team:Elastic-Agent Label for the Agent team label Aug 20, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 20, 2021

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 20, 2021
@mjmbischoff mjmbischoff mentioned this pull request Aug 22, 2021
Merged
6 tasks
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Sep 22, 2021

This pull request does not have a backport label. Could you fix it @mjmbischoff? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Sep 22, 2021
@botelastic
Copy link
Copy Markdown

botelastic bot commented Dec 15, 2021

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Dec 15, 2021
@botelastic
Copy link
Copy Markdown

botelastic bot commented Jan 14, 2022

Hi!
This PR has been stale for a while and we're going to close it as part of our cleanup procedure.
We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team.
Feel free to re-open this PR if you think it should stay open and is worth rebasing.
Thank you for your contribution!

@botelastic botelastic bot closed this Jan 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@kvch kvch kvch requested changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport-skip Skip notification from the automated backport with mergify enhancement Stalled Team:Elastic-Agent Label for the Agent team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mjmbischoff @elasticmachine @kvch @jsoriano