[Elastic Agent] Monitor Filebeat Inputs by kcreddy · Pull Request #5077 · elastic/integrations

kcreddy · 2023-01-23T02:43:10Z

What does this PR do?

Add new datastream filebeat_input to collect filebeat input metrics by monitoring filebeat's /inputs/ API endpoint. The endpoint currently supports 4 inputs
Most metrics/fields are dynamically mapped based on the naming convention.
Elastic Agent will then monitor the filebeat's /inputs/ when Update agent monitoring config to support /inputs of filebeat elastic-agent#2171 is merged

Local Tests

Add an input.
cat packages/elastic_agent/data_stream/filebeat_input_metrics/agent/stream/stream.yml.hbs

metricsets: ["json"]
namespace: "filebeat_input"
hosts: ["http+unix:///usr/share/elastic-agent/state/data/tmp/aws-s3-default.sock"]
period: 10s
json.is_array: true
path: "/inputs/"
tags:
{{#each tags as |tag|}}
  - {{tag}}
{{/each}}
processors:
- convert:
    mode: rename
    ignore_missing: true
    fields:
      - from: http.filebeat_input
        to: filebeat_input_metrics

Append below to packages/elastic_agent/data_stream/filebeat_input_metrics/manifest.yml

streams:
  - input: http/metrics
    vars:
      - name: tags
        type: text
        title: Tags
        multi: true
        required: true
        show_user: true
        default:
          - forwarded
          - elastic-agent-filebeat-input
    template_path: stream.yml.hbs
    title: Filebeat Input Metrics
    description: Collect Filebeat Input Metrics

Append below to packages/elastic_agent/manifest.yml:

policy_templates:
  - name: elastic_agent
    title: Elastic Agent Metrics
    description: Elastic Agent Metrics
    inputs:
      - type: http/metrics
        title: "Collect Metrics from Filebeat Inputs"
        description: "Collect Metrics from Filebeat Inputs"

Configure any datastream with aws-s3 input type (such as aws.s3access)
Add Elastic Agent integration. This should create filebeat input metric events under datastream metrics-elastic_agent.filebeat_input-default

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Relates https://github.com/elastic/security-team/issues/5525

elasticmachine · 2023-01-23T02:58:42Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-02-02T02:58:34.222+0000
Duration: 14 min 51 sec

Test stats 🧪

Test	Results
Failed	0
Passed	34
Skipped	0
Total	34

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-01-23T02:58:52Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`0/0`)	💚
Files	100.0% (`0/0`)	💚 12.5
Classes	100.0% (`0/0`)	💚 12.5
Methods	33.333% (`24/72`)	👎 -50.183
Lines	100.0% (`0/0`)	💚 7.286
Conditionals	100.0% (`0/0`)	💚

This reverts commit 379c8cb.

kcreddy · 2023-01-24T14:12:54Z

/test

packages/udp/data_stream/generic/sample_event.json

This reverts commit 8d74a7b.

ruflin · 2023-01-30T12:45:01Z

packages/elastic_agent/data_stream/filebeat_input_metrics/manifest.yml

@@ -0,0 +1,10 @@
+title: Elastic Agent
+dataset: elastic_agent.filebeat_input


@cmacknz Is this just a temporary solution? This seems to be monitoring data of v2 inputs? Will this all go under elastic_agent.inputs or similar in the future?

Hey @cmacknz @andrewkroh, I also need suggestion/direction on the name of the datastream as well, whether I need to create this new datastream filebeat_input or instead use existing elastic_agent datastream which is currently being used to send filebeat's /stats metrics.

Basically the field mappings in current PR were supposed to be used in conjunction with agent's inbuilt monitoring using http json metricset which monitors filebeat's /inputs endpoint. The PR is here: elastic/elastic-agent#2171 and would also need review
If I use this new datastream filebeat_input inside the agent's configuration instead of fixedAgentName and run the stack, I get following error during indexing:

{\"type\":\"security_exception\",\"reason\":\"action [indices:admin/auto_create] is unauthorized for API key id [j0XuAYYBIxEvrsej-8Ob] of user [elastic/fleet-server] on indices [metrics-elastic_agent.filebeat_input-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]\"}, dropping event!"

This seems to be related to elastic/elastic-agent#1814

If I use existing datastream elastic_agent and add all the fields mapping from this PR to elastic_agent datastream, it seem to work fine.

Also, whether we want to use TSDS or not since the CI is failing from usage of time_series mode probably due to current minimum version 7.16.0.

It definitely sounds like the path of least resistance would be to add these to the existing metrics-elastic_agent.filebeat-default data stream (without using TSDS). This avoids the issues related to permissions and allows the integration package to continue having a 7.x compatibility constraint.

This seems to be monitoring data of v2 inputs?

It's not strictly FB v2 inputs. Any Filebeat input regardless of the Filebeat interface used is eligible expose metrics. I think that all the ones with metrics happen to be implemented with Filebeat input v2 today. My expectation is that in the future we'll update Filebeat to publish metrics through the Agent V2 control interface at which time this metric collection can be removed.

This is hopefully temporary (on a to be determined time scale), we want to change the structure of the monitoring indexes because they are hard coded directly into Fleet right now. The path of least resistance here is definitely to add these to metrics-elastic_agent.filebeat-default.

Issue discussion how we should change the monitoring index structure V2: Elastic agent monitoring data stream names should not be per process elastic-agent#1814. Read from V2: Elastic agent monitoring data stream names should not be per process elastic-agent#1814 (comment)

We still need a path forward for collecting application level metrics in V2 without starting more and more Beat inputs to do it. We have an issue for discussing this as well but not concrete decisions yet. Determine path forward for collection of metrics for components/units elastic-agent#2181

Reading all the above, my current preference would be using filebeat_input as proposed in the PR and fix the permission issue in Fleet (hardcode). I expect as soon as v2 is fully done, data is likely shipped into input data stream but there might be breaking change, because of this the filebeat_input seems to be a good temporary solution. Also lets make sure we treat this as a beta data stream. For 7.16, I would expect this to only land in version > 8.7 or similar.

The part I worry about putting it into filebeat is that it could lead to potential conflicts and we ship 2 different datasets into the same data stream which is a bit against the principles.

Thanks for the clarification guys!

So if we intend to use new filebeat_input datastream itself, I have updated this PR and also elastic/elastic-agent#2171 to reflect filebeat_input datastream.

@ruflin @cmacknz to fix the permission issue in Fleet, may I know if this is the right place to add this new datastream?
https://github.com/elastic/kibana/blob/c5f20721e1879f1ebe161b0fa3b207f10ed2b6f7/x-pack/plugins/fleet/common/constants/agent_policy.ts#L15-L28
If so, I can work on a quick PR to add it.

We are targeting the filebeats inputs monitoring feature for 8.7 release, so any help would be appreciated to get it done. Thanks!

On my end, I'm only "suggestion", I'll leave it to @cmacknz to make decisions ;-) For the permissions, @nchaulet likely know the best place but what you linked above at least looks like part of the puzzle.

Mixing multiple datasets into the same datastream is something we should avoid, so I agree with Ruflin here in using a separate filebeat_input Beta datastream.

Here's an example of adding the new index to Fleet elastic/kibana@5d5603a

Note that if you make the change there I believe you will get both a logs and a metrics datastream. The logs datastream will effectively be unused so we may not actually want it created. There's nothing explicitly wrong with it being empty though. FYI @kpollich as well.

Thanks for the clarification and sharing the example @cmacknz
I have now added logs datastream to this PR as well.

Also I have created PR elastic/kibana#149974 to add new index. @kpollich could you please review it?

Thanks @nchaulet!

@AndersonQ / @michel-laterman Could you please review/approve this PR and also elastic/elastic-agent#2171

This reverts commit 3b40cb6.

This reverts commit fc1a8bd.

cmacknz

LGTM, except I think we could bump to the next minor version.

packages/elastic_agent/changelog.yml

packages/elastic_agent/manifest.yml

leehinman

LGTM

Non-blocking suggestion to use the external definition for the ECS fields.

leehinman · 2023-02-01T18:06:55Z

packages/elastic_agent/data_stream/filebeat_input_logs/fields/agent.yml

+  footnote: "Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on."
+  type: group
+  fields:
+    - name: account.id


Is there a reason not to have ECS fields like cloud.account.id use the external definition? This isn't blocking but it does make long term support easier and helps prevent type conflicts.

I need to explore the reason, but it looks like, for this package, all datastreams follow this convention, i.e., defining cloud related fields inside agent.yml. Maybe its worth creating an issue to replace all with ECS external definition instead.

elasticmachine · 2023-02-02T05:24:30Z

Package elastic_agent - 1.5.0 containing this change is available at https://epr.elastic.co/search?package=elastic_agent

kcreddy added 4 commits January 17, 2023 21:22

add fields hardcoded

f8967ae

update stream config

1960b20

update fields working fine

1c90790

add tsds options

bc0fd0a

kcreddy added enhancement New feature or request Integration:elastic_agent Elastic Agent labels Jan 23, 2023

kcreddy self-assigned this Jan 23, 2023

kcreddy changed the title ~~Elastic Agent | Monitor Filebeat Inputs~~ [Elastic Agent] Monitor Filebeat Inputs Jan 23, 2023

kcreddy added 4 commits January 24, 2023 11:16

update changelog

1458040

removing inputs from PR

379c8cb

Revert "removing inputs from PR"

40cda03

This reverts commit 379c8cb.

remove inputs

325e96f

andrewkroh reviewed Jan 24, 2023

View reviewed changes

packages/udp/data_stream/generic/sample_event.json Show resolved Hide resolved

kcreddy added 4 commits January 24, 2023 22:11

revert udp

9f6186b

remove unnecesary changes

2f449b7

removing tsds for ci to pass

8d74a7b

Revert "removing tsds for ci to pass"

d0be66f

This reverts commit 8d74a7b.

kcreddy marked this pull request as ready for review January 24, 2023 17:57

kcreddy requested a review from a team as a code owner January 24, 2023 17:57

kcreddy requested review from AndersonQ, michel-laterman and ruflin January 24, 2023 17:57

ruflin reviewed Jan 30, 2023

View reviewed changes

kcreddy added 4 commits January 31, 2023 16:10

remode filebeat_input ds

fc1a8bd

add field mapping to filebeat ds

3b40cb6

Revert "add field mapping to filebeat ds"

a1d0df7

This reverts commit 3b40cb6.

Revert "remode filebeat_input ds"

167b5d4

This reverts commit fc1a8bd.

kcreddy added 4 commits January 31, 2023 18:06

remove tsds

88775ec

update release to beta

1189a19

update root level name

12b3510

add filebeat_input logs

290f864

cmacknz approved these changes Feb 1, 2023

View reviewed changes

packages/elastic_agent/changelog.yml Outdated Show resolved Hide resolved

packages/elastic_agent/manifest.yml Outdated Show resolved Hide resolved

cmacknz requested a review from leehinman February 1, 2023 17:54

leehinman approved these changes Feb 1, 2023

View reviewed changes

updated version

212afb1

kcreddy merged commit 242c4cb into elastic:main Feb 2, 2023

andrewkroh mentioned this pull request Feb 6, 2023

[Metricbeat] beat module - add metricset for input metrics elastic/beats#33953

Closed

cmacknz mentioned this pull request Feb 28, 2023

Support monitoring the shipper from the Elastic Agent elastic/elastic-agent-shipper#267

Closed

constanca-m mentioned this pull request Jul 12, 2023

[Elastic Agent] Set dimension fields for metrics data streams #6939

Merged

16 tasks

kcreddy mentioned this pull request Sep 11, 2023

Remove mappings from filebeat_input_logs #7753

Merged

4 tasks

kcreddy deleted the filebeat-inputs-monitor branch February 7, 2025 08:37

		@@ -0,0 +1,10 @@
		title: Elastic Agent
		dataset: elastic_agent.filebeat_input

Conversation

kcreddy commented Jan 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Local Tests

Checklist

Related issues

Uh oh!

elasticmachine commented Jan 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Jan 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

kcreddy commented Jan 24, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmacknz Jan 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Jan 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Feb 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmacknz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

leehinman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Feb 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Feb 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kcreddy commented Jan 23, 2023 •

edited

Loading

elasticmachine commented Jan 23, 2023 •

edited

Loading

elasticmachine commented Jan 23, 2023 •

edited

Loading

cmacknz Jan 31, 2023 •

edited

Loading

kcreddy Jan 31, 2023 •

edited

Loading

kcreddy Feb 1, 2023 •

edited

Loading

cmacknz left a comment •

edited

Loading

kcreddy Feb 2, 2023 •

edited

Loading