Add heartbeat support for V2 by fearful-symmetry · Pull Request #33157 · elastic/beats

fearful-symmetry · 2022-09-21T16:22:19Z

What does this PR do?

This PR adds V2 support for heartbeat running under elastic-agent.

I haven't thoroughly tested this yet, but I wanted to put in the PR while I juggle the 2-3 other things I'm working on. See the Auditbeat PR for more detailed instructions on how to test: #33026

Here's an example of an elastic-agent input config that should work with heartbeat:

- id: synthetics/http-synthetics-f42c9747-2455-491b-a66c-967adc98007a
    name: test
    revision: 1
    type: synthetics/http
    use_output: default
    meta:
      package:
        name: synthetics
        version: 0.10.2
    data_stream:
      namespace: default
    package_policy_id: f42c9747-2455-491b-a66c-967adc98007a
    streams:
      - id: synthetics/http-http-f42c9747-2455-491b-a66c-967adc98007a
        name: test
        type: http
        enabled: true
        data_stream:
          dataset: http
          type: synthetics
        urls: 'http://localhost:9200'
        schedule: '@every 3m'
        timeout: 16s
        max_redirects: 0
        response.include_headers: true
        response.include_body: on_error
        check.request.method: GET
        processors:
          - add_observer_metadata:
              geo:
                name: Fleet managed
          - add_fields:
              target: ''
              fields:
                monitor.fleet_managed: true

Checklist

Test under agent
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

…t-cfg

elasticmachine · 2022-09-23T00:32:45Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-10-10T22:13:08.191+0000
Duration: 39 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	313
Skipped	0
Total	313

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

fearful-symmetry · 2022-09-30T22:44:36Z

Alright, finally managed to test this.

paulb-elastic · 2022-10-03T10:15:31Z

Pinging @emilioalvap and @andrewvc as there was some discussion about this regarding #32163

andrewvc · 2022-10-03T21:18:13Z

@fearful-symmetry with the V2 protocol does the output.elasticsearch section still get written? #32163 , which was merged recently, depends on it, reading back from ES to get the last known state for heartbeat

andrewvc · 2022-10-03T21:18:50Z

You can see where we set this up, and how, here: https://github.com/elastic/beats/blob/main/heartbeat/beater/heartbeat.go#L72

fearful-symmetry · 2022-10-03T22:00:48Z

@andrewvc

with the V2 protocol does the output.elasticsearch section still get written?

Yes, it does. Until the shipper comes along, we're still using built-in elasticsearch output. Theoretically, nothing should change in terms of the actual configs that are sent to individual components of a beat; this V2 support mechanism is just a compatibility shim for V2.

fearful-symmetry · 2022-10-03T22:06:08Z

You can see where we set this up, and how, here: https://github.com/elastic/beats/blob/main/heartbeat/beater/heartbeat.go#L72

@andrewvc does this work under the current fleet setup? This happens outside of the reloaded registers here: https://github.com/elastic/beats/blob/main/heartbeat/beater/heartbeat.go#L213, which means that the code you linked to (I think) won't respond to changes in the fleet config, regardless of this PR.

andrewvc · 2022-10-04T02:00:15Z

@fearful-symmetry I'm realizing it's only been tested on the beat, thanks a bunch for noticing that, I've opened #33244 to track.

blakerouse

Looks good to me, minus the one inline comment.

x-pack/heartbeat/cmd/root.go

* Update Metricbeat, Filebeat, libbeat with elastic-agent V2 support (#32673) * basic framework * continued tinkering * move away from ast code, use a struct * get metricbeat working, starting on filebeat * add notice update * add basic config register * move over processors to individual beats * remove comments * start to integrate V2 client changes * finishing touches * lint * cleanup merge * remove V1 controller * stil tinkering with linter * still fixing linter * plz linter * fmt x-pack files * notice update * fix output test * refactor stop functions, refactor tests, some misc cleanup * fix client version string * add devguide * linter * expand filebeat test * cleanup test * fix docs, add tests, debuggin * add signal handler * fix mutex issue in register * Fix osquerybeat configuration for V2 * clean up component registration * spelling * remove workaround for filebeat types * try to fix filebeat tests * add nil checks, fix test, fix unit stop * continue tinkering with nil type checks * add test for missing config datastreams, clean up nil handling * change nil protections, use getter methods * fix config access in output code Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> * V2 packetbeat support (#33041) * first attempt at auditbeat support * add license header * initial packetbeat support * fix bad branch * cleanup * typo in comment * clean up, move around files * add new processors to streams * First pass at auditbeat support (#33026) * first attempt at auditbeat support * add license header * cleanup * move files around * Add heartbeat support for V2 (#33157) * add v2 config * fix name * fix doc * fix go.mod * fix unchecked stream_id * fix unchecked stream_id (#33335) * Update elastic-agent-libs for output panic fix (#33336) * Fix errors for non-synth capable instances (#33310) Fixes #32694 by making sure we use the lightweight wrapper code always when monitors cannot be initialized. This also fixes an unrelated bug, where errors attached to non-summary events would not be indexed. * [Automation] Update elastic stack version to 8.6.0-5a8d757d for testing (#33323) Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> * add pid awareness to file locking (#33169) * add pid awareness to file locking * cleanup, logic for handling restarts with the same PID * add zombie-state awareness * fix file naming * add retry for unlock * was confused by unlock code, fix, cleanup * update notice * fix race with file creation, update deps * clean up tests, spelling * hack for cgo * add lic headers * notice * try to fix windows issues * fix typos * small fixes * use exclusive locks * remove feature to start with a specially named pidfile * clean up some error handling, fix test cleanup * forgot changelog * Fix sample config in log rotation docs (#33306) * Add banner to deprecate functionbeat (#33297) * fix unchecked stream_id * packetbeat/protos/dns: clean up package (#33286) * avoid magic numbers * fix hashableDNSTuple size and offsets * avoid use of String and Error methods in formatted print calls * remove redundant conversions * quieten linter * use plugin-owned logp.Logger * update elastic-agent-libs * Revert "fix unchecked stream_id" This reverts commit 26ef6da. * [Automation] Update elastic stack version to 8.6.0-40086bc7 for testing (#33339) Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> Co-authored-by: Andrew Cholakian <andrewvc@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> * update elastic-agent-client (#33552) Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> Co-authored-by: Andrew Cholakian <andrewvc@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

fearful-symmetry added 2 commits September 15, 2022 16:42

add v2 config

a0f8fa6

fix name

19065c3

fearful-symmetry added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team v8.6.0 labels Sep 21, 2022

fearful-symmetry self-assigned this Sep 21, 2022

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 21, 2022

Merge remote-tracking branch 'upstream/feature-arch-v2' into heartbea…

48b5a5f

…t-cfg

fearful-symmetry requested a review from a team as a code owner September 22, 2022 23:49

cmacknz mentioned this pull request Sep 28, 2022

[Meta] V2 Feature Architecture (8.5/8.6) elastic/elastic-agent#836

Closed

52 tasks

andrewvc mentioned this pull request Oct 4, 2022

[Heartbeat] States ES connection should reset on fleet reload (mandatory for 8.6) #33244

Closed

blakerouse approved these changes Oct 10, 2022

View reviewed changes

x-pack/heartbeat/cmd/root.go Outdated Show resolved Hide resolved

fix doc

249f3bb

fearful-symmetry merged commit 8dc24d6 into elastic:feature-arch-v2 Oct 11, 2022

emilioalvap mentioned this pull request Oct 11, 2022

[Meta][Heartbeat] HB support for elastic-agent V2 #33314

Closed

6 tasks

emilioalvap mentioned this pull request Oct 31, 2022

[Heartbeat] States reloader and eslegclient #33405

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add heartbeat support for V2#33157

Add heartbeat support for V2#33157
fearful-symmetry merged 4 commits intoelastic:feature-arch-v2from
fearful-symmetry:heartbeat-cfg

fearful-symmetry commented Sep 21, 2022 •

edited

Loading

Uh oh!

elasticmachine commented Sep 23, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Uh oh!

fearful-symmetry commented Sep 30, 2022

Uh oh!

paulb-elastic commented Oct 3, 2022

Uh oh!

andrewvc commented Oct 3, 2022

Uh oh!

andrewvc commented Oct 3, 2022

Uh oh!

fearful-symmetry commented Oct 3, 2022

Uh oh!

fearful-symmetry commented Oct 3, 2022

Uh oh!

andrewvc commented Oct 4, 2022

Uh oh!

blakerouse left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

fearful-symmetry commented Sep 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Uh oh!

elasticmachine commented Sep 23, 2022 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

fearful-symmetry commented Sep 30, 2022

Uh oh!

paulb-elastic commented Oct 3, 2022

Uh oh!

andrewvc commented Oct 3, 2022

Uh oh!

andrewvc commented Oct 3, 2022

Uh oh!

fearful-symmetry commented Oct 3, 2022

Uh oh!

fearful-symmetry commented Oct 3, 2022

Uh oh!

andrewvc commented Oct 4, 2022

Uh oh!

blakerouse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fearful-symmetry commented Sep 21, 2022 •

edited

Loading

elasticmachine commented Sep 23, 2022 •

edited by jenkins-beats-ci bot

Loading