V2 packetbeat support by fearful-symmetry · Pull Request #33041 · elastic/beats

fearful-symmetry · 2022-09-09T17:43:00Z

What does this PR do?

This adds preliminary support for the V2 elastic-agent to packetbeat. I've partly tested this, however I need to verify that metadata fields are being added properly; other than that, it works.

Checklist

My code follows the style guidelines of this project
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Verify metadata fields are working

elasticmachine · 2022-09-09T17:43:03Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

elasticmachine · 2022-09-09T18:27:51Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-09-13T22:01:36.980+0000
Duration: 48 min 15 sec

Test stats 🧪

Test	Results
Failed	0
Passed	14
Skipped	1
Total	15

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

x-pack/packetbeat/cmd/agent.go

x-pack/packetbeat/cmd/root.go

fearful-symmetry · 2022-09-09T23:35:24Z

thanks for the feedback @efd6 !

x-pack/packetbeat/cmd/agent.go

efd6 · 2022-09-11T21:49:23Z

x-pack/packetbeat/cmd/agent.go

+// packetbeatCfg is a callback registered with central management to perform any needed config transformations
+// before agent configs are sent to a beat


Would it be true to say the following

// packetbeatCfg returns a packetbeat Elastic Agent client.Unit reload // configuration constructed from configuration details in rawIn.

in conjunction with a comment on the SetTransform call?

As a maintainer doing investigation into issues this is approximately the level of detail and direction in doc comments that helps focus or exclude parts of code to read.

Yah, that would work too, I can expand the doc to sort of merge the two. Will also expand the SetTransform Docs a bit, since that's a bit vague.

x-pack/packetbeat/cmd/root.go

x-pack/packetbeat/cmd/agent.go

efd6 · 2022-09-12T23:53:01Z

x-pack/packetbeat/cmd/root.go


+// packetbeatCfg is a callback registered via SetTransform that returns a packetbeat Elastic Agent client.Unit
+// configuration generated from a raw Elastic Agent config
+func packetbeatCfg(rawIn *proto.UnitExpectedConfig, _ *client.AgentInfo) ([]*reload.ConfigWithMeta, error) {


Caller before callee makes context easier to grasp, though in this situation there is ~0 difference.

* Update Metricbeat, Filebeat, libbeat with elastic-agent V2 support (#32673) * basic framework * continued tinkering * move away from ast code, use a struct * get metricbeat working, starting on filebeat * add notice update * add basic config register * move over processors to individual beats * remove comments * start to integrate V2 client changes * finishing touches * lint * cleanup merge * remove V1 controller * stil tinkering with linter * still fixing linter * plz linter * fmt x-pack files * notice update * fix output test * refactor stop functions, refactor tests, some misc cleanup * fix client version string * add devguide * linter * expand filebeat test * cleanup test * fix docs, add tests, debuggin * add signal handler * fix mutex issue in register * Fix osquerybeat configuration for V2 * clean up component registration * spelling * remove workaround for filebeat types * try to fix filebeat tests * add nil checks, fix test, fix unit stop * continue tinkering with nil type checks * add test for missing config datastreams, clean up nil handling * change nil protections, use getter methods * fix config access in output code Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> * V2 packetbeat support (#33041) * first attempt at auditbeat support * add license header * initial packetbeat support * fix bad branch * cleanup * typo in comment * clean up, move around files * add new processors to streams * First pass at auditbeat support (#33026) * first attempt at auditbeat support * add license header * cleanup * move files around * Add heartbeat support for V2 (#33157) * add v2 config * fix name * fix doc * fix go.mod * fix unchecked stream_id * fix unchecked stream_id (#33335) * Update elastic-agent-libs for output panic fix (#33336) * Fix errors for non-synth capable instances (#33310) Fixes #32694 by making sure we use the lightweight wrapper code always when monitors cannot be initialized. This also fixes an unrelated bug, where errors attached to non-summary events would not be indexed. * [Automation] Update elastic stack version to 8.6.0-5a8d757d for testing (#33323) Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> * add pid awareness to file locking (#33169) * add pid awareness to file locking * cleanup, logic for handling restarts with the same PID * add zombie-state awareness * fix file naming * add retry for unlock * was confused by unlock code, fix, cleanup * update notice * fix race with file creation, update deps * clean up tests, spelling * hack for cgo * add lic headers * notice * try to fix windows issues * fix typos * small fixes * use exclusive locks * remove feature to start with a specially named pidfile * clean up some error handling, fix test cleanup * forgot changelog * Fix sample config in log rotation docs (#33306) * Add banner to deprecate functionbeat (#33297) * fix unchecked stream_id * packetbeat/protos/dns: clean up package (#33286) * avoid magic numbers * fix hashableDNSTuple size and offsets * avoid use of String and Error methods in formatted print calls * remove redundant conversions * quieten linter * use plugin-owned logp.Logger * update elastic-agent-libs * Revert "fix unchecked stream_id" This reverts commit 26ef6da. * [Automation] Update elastic stack version to 8.6.0-40086bc7 for testing (#33339) Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> Co-authored-by: Andrew Cholakian <andrewvc@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> * update elastic-agent-client (#33552) Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> Co-authored-by: Andrew Cholakian <andrewvc@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

fearful-symmetry added 4 commits September 8, 2022 11:46

first attempt at auditbeat support

a785966

add license header

150a60c

initial packetbeat support

a8849ec

fix bad branch

4c32034

fearful-symmetry added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team v8.5.0 labels Sep 9, 2022

fearful-symmetry requested a review from a team as a code owner September 9, 2022 17:43

fearful-symmetry self-assigned this Sep 9, 2022

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 9, 2022

cmacknz added v8.6.0 and removed v8.5.0 labels Sep 9, 2022

efd6 reviewed Sep 9, 2022

View reviewed changes

x-pack/packetbeat/cmd/agent.go Outdated Show resolved Hide resolved

x-pack/packetbeat/cmd/agent.go Outdated Show resolved Hide resolved

x-pack/packetbeat/cmd/agent.go Outdated Show resolved Hide resolved

x-pack/packetbeat/cmd/root.go Outdated Show resolved Hide resolved

cleanup

071a8e5

typo in comment

642da5b

efd6 reviewed Sep 11, 2022

View reviewed changes

clean up, move around files

c946105

efd6 approved these changes Sep 12, 2022

View reviewed changes

add new processors to streams

111b7e8

fearful-symmetry merged commit 07d43e3 into elastic:feature-arch-v2 Sep 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

V2 packetbeat support#33041

V2 packetbeat support#33041
fearful-symmetry merged 8 commits intoelastic:feature-arch-v2from
fearful-symmetry:v2-packetbeat-support

fearful-symmetry commented Sep 9, 2022

Uh oh!

elasticmachine commented Sep 9, 2022

Uh oh!

elasticmachine commented Sep 9, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fearful-symmetry commented Sep 9, 2022

Uh oh!

Uh oh!

efd6 Sep 11, 2022

Uh oh!

fearful-symmetry Sep 12, 2022

Uh oh!

Uh oh!

Uh oh!

efd6 Sep 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		// packetbeatCfg is a callback registered with central management to perform any needed config transformations
		// before agent configs are sent to a beat

Conversation

fearful-symmetry commented Sep 9, 2022

What does this PR do?

Checklist

Author's Checklist

Uh oh!

elasticmachine commented Sep 9, 2022

Uh oh!

elasticmachine commented Sep 9, 2022 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fearful-symmetry commented Sep 9, 2022

Uh oh!

Uh oh!

efd6 Sep 11, 2022

Choose a reason for hiding this comment

Uh oh!

fearful-symmetry Sep 12, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

efd6 Sep 12, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

elasticmachine commented Sep 9, 2022 •

edited by jenkins-beats-ci bot

Loading