Inject Beat specific default processors by michalpristas · Pull Request #34149 · elastic/beats

michalpristas · 2022-12-30T10:26:48Z

What does this PR do?

Managed beats do not respect default processors configured in {beat}.yml file.
This results in events without host, k8s or docker metadata.

In this PR we take default processors from config file to callback responsible for handling configuration passed from agent and make sure they are included during generate step.

Why is it important?

Missing metadata may break dashboards

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Fixes: #34125

elasticmachine · 2022-12-30T10:26:50Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

elasticmachine · 2022-12-30T10:26:51Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

elasticmachine · 2022-12-30T11:42:16Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-12-30T16:05:51.205+0000
Duration: 85 min 38 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7408
Skipped	338
Total	7746

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

michalpristas · 2022-12-30T15:31:20Z

Reordered processors so default are first,

cmacknz · 2022-12-30T15:38:07Z

x-pack/libbeat/management/generate.go


 		// 3. stream processors
-		streamSource, err = injectStreamProcessors(raw, inputType, stream, streamSource)
+		streamSource, err = injectStreamProcessors(raw, inputType, stream, streamSource, defaultProcessors)


There is this above:

// 1. global processors streamSource = injectGlobalProcesssors(raw, streamSource)

The metadata processors behave more like global processors don't they? Shouldn't they be added there?

Both prepend and append are used here to control the order so just reading this code it is a bit tricky to follow what ends up where.

Could you post an example beat-rendered-config if you have one from diagnostics with this change?

yes but if you check the final order of the processors it's

// now the order of the processors on this input is as follows // 1. stream processors // 2. agentInfo processors // 3. global processors // 4. stream specific processors

so if we put it with global, they will be pushed to later stage by next 2 steps. this is why i went with stream

it makes sure it's first

it makes clear that it will be part of the stream

Got it, thanks.

cmacknz · 2022-12-30T15:59:09Z

x-pack/metricbeat/modules.d/enterprisesearch-xpack.yml.disabled

@@ -1,11 +0,0 @@
-# Module: enterprisesearch


This file was committed by mistake I think, I see this come in when changing branches occasionally.

yep probably changing branches

cmacknz

Thanks!

Inject Beat specific default processors (#34149) (cherry picked from commit 75ebd50)

Inject Beat specific default processors (#34149) (cherry picked from commit 75ebd50) Co-authored-by: Michal Pristas <michal.pristas@gmail.com>

fearful-symmetry · 2023-01-12T00:45:56Z

Just a heads up, I'm running into some weird behaviors with the processors under elastic-agent. They seem to be putting the beat in some kind of wedged state where elastic-agent won't actually stop when it gets a sigint. I can't manually remove the processors either, but commenting out the defaultProcessors line seems to fix it. Will investigate more tomorrow.

cmacknz · 2023-01-12T15:29:16Z

Thanks. FYI the default processors aren't new, they would have been configured in a different way prior to 8.6 (reading the default beat config file at startup) but they would have been running regardless.

Inject Beat specific default processors (#34149)

Inject Beat specific default processors

57d3d29

michalpristas added Team:Elastic-Agent Label for the Agent team Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team backport-v8.6.0 Automated backport with mergify labels Dec 30, 2022

michalpristas self-assigned this Dec 30, 2022

michalpristas requested review from a team as code owners December 30, 2022 10:26

michalpristas requested review from cmacknz and removed request for a team December 30, 2022 10:26

michalpristas requested a review from fearful-symmetry December 30, 2022 10:26

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 30, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 30, 2022

revert proc debug logs

6b5028d

add default first

1ea8c8d

cmacknz reviewed Dec 30, 2022

View reviewed changes

revert x-pack/metricbeat/modules.d/enterprisesearch-xpack.yml.disabled

2bd2018

cmacknz approved these changes Dec 30, 2022

View reviewed changes

michalpristas merged commit 75ebd50 into elastic:main Dec 30, 2022

mergify bot mentioned this pull request Dec 30, 2022

[8.6](backport #34149) Inject Beat specific default processors #34150

Merged

mergify bot pushed a commit that referenced this pull request Dec 30, 2022

Inject Beat specific default processors (#34149)

b481e5a

Inject Beat specific default processors (#34149) (cherry picked from commit 75ebd50)

michalpristas added a commit that referenced this pull request Dec 30, 2022

Inject Beat specific default processors (#34149) (#34150)

0d90255

Inject Beat specific default processors (#34149) (cherry picked from commit 75ebd50) Co-authored-by: Michal Pristas <michal.pristas@gmail.com>

cmacknz mentioned this pull request Apr 3, 2023

[Elastic Agent] Default processors created per input can result in high agent CPU usage #35000

Closed

cmacknz mentioned this pull request Apr 4, 2023

Migrate the default global Beat processors to the shipper elastic/elastic-agent-shipper#292

Open

fearful-symmetry mentioned this pull request Apr 5, 2023

Fix performance issues with processors scaling under agent #35031

Merged

4 tasks

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

Inject Beat specific default processors (#34149)

81a5a49

Inject Beat specific default processors (#34149)

Conversation

michalpristas commented Dec 30, 2022

What does this PR do?

Why is it important?

Checklist

Uh oh!

elasticmachine commented Dec 30, 2022

Uh oh!

elasticmachine commented Dec 30, 2022

Uh oh!

elasticmachine commented Dec 30, 2022 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

michalpristas commented Dec 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmacknz Dec 30, 2022

Choose a reason for hiding this comment

Uh oh!

michalpristas Dec 30, 2022

Choose a reason for hiding this comment

Uh oh!

cmacknz Dec 30, 2022

Choose a reason for hiding this comment

Uh oh!

cmacknz Dec 30, 2022

Choose a reason for hiding this comment

Uh oh!

michalpristas Dec 30, 2022

Choose a reason for hiding this comment

Uh oh!

cmacknz left a comment

Choose a reason for hiding this comment

Uh oh!

fearful-symmetry commented Jan 12, 2023

Uh oh!

cmacknz commented Jan 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

elasticmachine commented Dec 30, 2022 •

edited by jenkins-beats-ci bot

Loading

michalpristas commented Dec 30, 2022 •

edited

Loading