First pass at auditbeat support by fearful-symmetry · Pull Request #33026 · elastic/beats

fearful-symmetry · 2022-09-08T19:01:01Z

What does this PR do?

This PR adds auditbeat to the V2 setup/init system, as part of this: #32786

I've never touched auditbeat before, so I had bit of trouble setting this up and testing it, but I can say that in it's current state, it doesn't not work. I'm hoping that someone with more auditbeat knowledge can better prod at this and see where the missing pieces are, if any.

What this does

For anyone coming at this with auditbeat knowledge that's new to V2, the core component here is the auditbeatCfg function performs any necessary transformation of the config that comes from fleet.

Here's an example auditbeat config I managed to generate. This is a yaml representation of what's received by the rawIn argument to to auditbeatCfg :

  - id: audit/auditd-auditd-5aac139d-166a-4d8f-af34-a751364f7635
    name: auditd_manager-1
    revision: 1
    type: audit/auditd
    use_output: default
    meta:
      package:
        name: auditd_manager
        version: 1.3.0
    data_stream:
      namespace: default
    streams:
      - id: >-
          audit/auditd-auditd_manager.auditd-5aac139d-166a-4d8f-af34-a751364f7635
        type: audit/auditd
        data_stream:
          dataset: auditd_manager.auditd
          type: logs
        condition: '${host.platform} == ''linux'''
        include_raw_message: true
        socket_type: unicast
        immutable: false
        resolve_ids: true
        failure_mode: silent
        audit_rules: '-a always,exit -F arch=b32 -S all -F key=32bit-abi'
        backlog_limit: 8192
        rate_limit: 0
        include_warnings: false
        backpressure_strategy: auto
        tags:
          - auditd_manager-auditd

The auditbeatCfg should take this config and process it in a way to make it readable to auditbeat. Specifically, each item in the streams YAML array (which becomes that modules array in auditbeatCfg) will be sent to an auditbeat beater instance. Right now, the only change is to add the module: key based on the type value in the YAML.

To test this, just build this PR into auditbeat/agent, run against an 8.5 cluster, and configure auditbeat via fleet. Make sure whatever integrations and config you're adding are actually being applied.

How to test this

This code must be tested against the V2-enabled code in the feature-arch-v2 code. To test this PR:

Checkout the feature-arch-v2 branch of elastic-agent
Checkout this PR in the beats repo
in the main elastic-agent magefile.go, add an "auditbeat" entry to the packedBeats var, like this:

packedBeats := []string{"filebeat", "heartbeat", "metricbeat", "osquerybeat", "auditbeat"}

(note: there are other methods of building elastic-agent with a test/development beat, I just find this to be the most convenient. Feel free to use your own method if you prefer)

build elastic-agent with PACKAGES=TarGz mage dev:package (note:you may want to further constrain the build targets with PLATFORMS=linux/amd64 or something similar)
Configure elastic-agent. In the hard-coded elastic-agent.yml file, replace any existing data under the inputs: field with the example input config included above in the What this does section. Alternative, write your own input config, or modify the example as needed.
Either re-configure the included elasticsearch output in elastic-agent.yml, or use a file output:

outputs:
  default:
     type: file
     enabled: true
     path: "/tmp/testbeat_out"
     filename: "testbeat_log"

alternatively, instead of configuring elastic-agent.yml, connect elasticsearch to an existing 8.5.0-SNAPSHOT cluster, enroll the test elastic-agent binary and configure via fleet as usual.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

elasticmachine · 2022-09-08T19:01:04Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

elasticmachine · 2022-09-08T19:45:38Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-09-15T23:45:37.598+0000
Duration: 40 min 28 sec

Test stats 🧪

Test	Results
Failed	0
Passed	276
Skipped	49
Total	325

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

efd6

Nits only.

x-pack/auditbeat/cmd/agent.go

x-pack/auditbeat/cmd/root.go

fearful-symmetry · 2022-09-15T23:46:09Z

@efd6 did you get a chance to test this? I've never actually touched auditbeat before so it was a bit of a struggle, so I was hoping someone with more of a background might have better luck with it.

efd6 · 2022-10-04T05:18:52Z

I have now had a chance to test this and it looks like it is behaving correctly.

* Update Metricbeat, Filebeat, libbeat with elastic-agent V2 support (#32673) * basic framework * continued tinkering * move away from ast code, use a struct * get metricbeat working, starting on filebeat * add notice update * add basic config register * move over processors to individual beats * remove comments * start to integrate V2 client changes * finishing touches * lint * cleanup merge * remove V1 controller * stil tinkering with linter * still fixing linter * plz linter * fmt x-pack files * notice update * fix output test * refactor stop functions, refactor tests, some misc cleanup * fix client version string * add devguide * linter * expand filebeat test * cleanup test * fix docs, add tests, debuggin * add signal handler * fix mutex issue in register * Fix osquerybeat configuration for V2 * clean up component registration * spelling * remove workaround for filebeat types * try to fix filebeat tests * add nil checks, fix test, fix unit stop * continue tinkering with nil type checks * add test for missing config datastreams, clean up nil handling * change nil protections, use getter methods * fix config access in output code Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> * V2 packetbeat support (#33041) * first attempt at auditbeat support * add license header * initial packetbeat support * fix bad branch * cleanup * typo in comment * clean up, move around files * add new processors to streams * First pass at auditbeat support (#33026) * first attempt at auditbeat support * add license header * cleanup * move files around * Add heartbeat support for V2 (#33157) * add v2 config * fix name * fix doc * fix go.mod * fix unchecked stream_id * fix unchecked stream_id (#33335) * Update elastic-agent-libs for output panic fix (#33336) * Fix errors for non-synth capable instances (#33310) Fixes #32694 by making sure we use the lightweight wrapper code always when monitors cannot be initialized. This also fixes an unrelated bug, where errors attached to non-summary events would not be indexed. * [Automation] Update elastic stack version to 8.6.0-5a8d757d for testing (#33323) Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> * add pid awareness to file locking (#33169) * add pid awareness to file locking * cleanup, logic for handling restarts with the same PID * add zombie-state awareness * fix file naming * add retry for unlock * was confused by unlock code, fix, cleanup * update notice * fix race with file creation, update deps * clean up tests, spelling * hack for cgo * add lic headers * notice * try to fix windows issues * fix typos * small fixes * use exclusive locks * remove feature to start with a specially named pidfile * clean up some error handling, fix test cleanup * forgot changelog * Fix sample config in log rotation docs (#33306) * Add banner to deprecate functionbeat (#33297) * fix unchecked stream_id * packetbeat/protos/dns: clean up package (#33286) * avoid magic numbers * fix hashableDNSTuple size and offsets * avoid use of String and Error methods in formatted print calls * remove redundant conversions * quieten linter * use plugin-owned logp.Logger * update elastic-agent-libs * Revert "fix unchecked stream_id" This reverts commit 26ef6da. * [Automation] Update elastic stack version to 8.6.0-40086bc7 for testing (#33339) Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> Co-authored-by: Andrew Cholakian <andrewvc@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> * update elastic-agent-client (#33552) Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> Co-authored-by: Andrew Cholakian <andrewvc@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

first attempt at auditbeat support

a785966

fearful-symmetry added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team v8.5.0 labels Sep 8, 2022

fearful-symmetry requested review from andrewkroh and cmacknz September 8, 2022 19:01

fearful-symmetry requested a review from a team as a code owner September 8, 2022 19:01

fearful-symmetry self-assigned this Sep 8, 2022

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 8, 2022

add license header

150a60c

efd6 reviewed Sep 9, 2022

View reviewed changes

cmacknz added 8.6-candidate v8.6.0 and removed v8.5.0 8.6-candidate labels Sep 9, 2022

cleanup

af37024

fearful-symmetry requested a review from efd6 September 15, 2022 23:41

move files around

811e9a5

fearful-symmetry mentioned this pull request Sep 21, 2022

Add heartbeat support for V2 #33157

Merged

6 tasks

fearful-symmetry merged commit 6eecb84 into elastic:feature-arch-v2 Oct 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First pass at auditbeat support#33026

First pass at auditbeat support#33026
fearful-symmetry merged 4 commits intoelastic:feature-arch-v2from
fearful-symmetry:v2-auditbeat-support

fearful-symmetry commented Sep 8, 2022 •

edited

Loading

Uh oh!

elasticmachine commented Sep 8, 2022

Uh oh!

elasticmachine commented Sep 8, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Uh oh!

efd6 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fearful-symmetry commented Sep 15, 2022

Uh oh!

efd6 commented Oct 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

fearful-symmetry commented Sep 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

What this does

How to test this

Checklist

Uh oh!

elasticmachine commented Sep 8, 2022

Uh oh!

elasticmachine commented Sep 8, 2022 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fearful-symmetry commented Sep 15, 2022

Uh oh!

efd6 commented Oct 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fearful-symmetry commented Sep 8, 2022 •

edited

Loading

elasticmachine commented Sep 8, 2022 •

edited by jenkins-beats-ci bot

Loading