Make kibana/stats metricset work for Stack Monitoring without xpack.enabled flag by sayden · Pull Request #21496 · elastic/beats

sayden · 2020-10-02T18:52:31Z

Ready for testing. Check data.json and settings_data.json for a look of how the events look like now.

elasticmachine · 2020-10-02T18:52:33Z

Pinging @elastic/stack-monitoring (Stack monitoring)

elasticmachine · 2020-10-02T18:52:33Z

Pinging @elastic/integrations-services (Team:Services)

botelastic · 2020-10-02T18:52:40Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2020-10-02T19:12:44Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #21496 updated
Start Time: 2020-12-10T18:51:53.856+0000
Duration: 56 min 15 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2265
Skipped	520
Total	2785

Steps errors

Expand to view the steps failures

`Terraform Apply on x-pack/metricbeat/module/aws`

Took 0 min 15 sec . View more details on here

`Terraform Apply on x-pack/metricbeat/module/aws`

Took 0 min 15 sec . View more details on here

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2265
Skipped	520
Total	2785

chrisronline · 2020-10-23T18:10:31Z

I'm seeing this error when running MB:

2020-10-23T14:10:10.317-0400 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: error loading template: could not load template. Elasticsearch returned: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."}],"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."},"status":400}. Response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."}],"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."},"status":400}.

elasticmachine · 2020-11-06T18:01:52Z

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	4
Passed	2194
Skipped	469
Total	2667

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

    * **Name**: `Build&Test / metricbeat-goIntegTest / TestXPackEnabled – kibana`
    * **Name**: `Build&Test / metricbeat-goIntegTest / TestFetch – status`
    * **Name**: `Build&Test / x-pack/metricbeat-build / test_migration – x-pack.metricbeat.tests.system.test_xpack_base.Test`
    * **Name**: `Build&Test / x-pack/metricbeat-build / test_template – x-pack.metricbeat.tests.system.test_xpack_base.Test`

chrisronline · 2020-11-09T18:03:28Z

I'm still getting the same error:

2020-11-09T12:59:34.256-0500 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: error loading template: could not load template. Elasticsearch returned: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."}],"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."},"status":400}. Response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."}],"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."},"status":400

…/stats_xpack_flag

…/stats_xpack_flag # Conflicts: # metricbeat/module/kibana/stats/stats.go

ycombinator · 2020-12-14T07:26:54Z

metricbeat/module/kibana/stats/_meta/data.json

            "snapshot": false,
-            "status": "green"
+            "status": "green",
+            "usage": {


With the recent changes in #22732, I don't think we even need to collect usage any more. WDYT @chrisronline - safe to omit usage collection in this PR here?

Yes! Please remove it

ycombinator · 2020-12-14T07:40:31Z

metricbeat/module/kibana/stats/stats.go

-	if m.XPackEnabled {
-		m.fetchSettings(r, now)
+	if err = m.fetchSettings(r); err != nil {
+		return errors.Wrap(err, "error trying to get settings data from Kibana")


So previously (before this PR), this metricset would behave as follows:

if xpack.enabled: true was set, it would produce two events in each Fetch iteration, indexed into .monitoring-kibana-*, one with type: kibana_stats and one with type: kibana_settings.

if xpack.enabled: false was set, it would produce one event in each Fetch iteration, indexed into metricbeat-*, corresponding to type: kibana_stats` (but not actually including that field).

With the changes in this PR, it seems like we would always be producing two events per Fetch iteration: one corresponding to type: kibana_stats and one corresponding to type: kibana_settings. I wonder if now it makes sense to keep the kibana/stats metricset for just the former and create a new metricset, kibana/settings, for just the latter? It would be more in line with how other Metricbeat modules work. WDYT?

Feel free to perform this split of metricsets in a follow up PR to the feature branch. For now, if you want to merge the PR with the current implementation, that's fine.

I'll do it in a different PR because I have seen that it requires that I change quite a lot of thing in the base module. I mean, I have to change them anyways but I discovered it now.

metricbeat/module/kibana/stats/_meta/settings_data.json

ycombinator

I've left a couple of comments/questions on this PR. Besides those, this PR LGTM. So I'm approving it now.

Feel free to address the comments before you merge this PR OR merge it as-is now and address the comments in the feature branch later, as you prefer.

…nabled flag (elastic#21496)

Atomic commit

9d0f7a8

sayden added Metricbeat Metricbeat Feature:Stack Monitoring Team:Services (Deprecated) Label for the former Integrations-Services team labels Oct 2, 2020

sayden self-assigned this Oct 2, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 2, 2020

Run make update

ebbbf85

sayden changed the base branch from master to feature-stack-monitoring-mb-ecs October 2, 2020 18:54

Fix typo

1c36cf5

chrisronline mentioned this pull request Oct 23, 2020

Make logstash/node metricset work for Stack Monitoring without xpack.enabled flag #21545

Closed

sayden added 4 commits November 3, 2020 13:47

Fix fields

5af71ce

Fix linting

2a46d8d

Rootfield not being defined before setting a value

a279b4d

Regenerate data.json

0bc41ac

sayden added 8 commits December 3, 2020 11:48

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/kibana…

b8cb768

…/stats_xpack_flag

Add TestData and re-generate data.json

e91c748

Update fields.go

ca4a775

Fix linting

37492eb

Fix fields.yml

82efdce

Remove duplicated field

08247b6

Remove test that isn't used anymore

c99c5bd

Fix broken test

7451eb7

sayden added 4 commits December 3, 2020 21:10

Remove new testdata function

fa5dd30

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/kibana…

6ec26a8

…/stats_xpack_flag # Conflicts: # metricbeat/module/kibana/stats/stats.go

Fix broken test

d28540f

Remove TODO from field description

5ebf645

sayden requested a review from ycombinator December 10, 2020 18:50

ycombinator reviewed Dec 14, 2020

View reviewed changes

metricbeat/module/kibana/stats/_meta/settings_data.json Show resolved Hide resolved

ycombinator approved these changes Dec 15, 2020

View reviewed changes

sayden merged commit 283ef2a into elastic:feature-stack-monitoring-mb-ecs Dec 16, 2020

sayden mentioned this pull request Apr 27, 2021

[Metricbeat] Remove xpack enabled flag on ES, Logstash, Beats and Kibana #24427

Merged

6 tasks

sayden deleted the feature/mb/kibana/stats_xpack_flag branch August 25, 2022 12:54

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Make kibana/stats metricset work for Stack Monitoring without xpack.e…

edf8b85

…nabled flag (elastic#21496)

Conversation

sayden commented Oct 2, 2020

Uh oh!

elasticmachine commented Oct 2, 2020

Uh oh!

elasticmachine commented Oct 2, 2020

Uh oh!

botelastic bot commented Oct 2, 2020

Uh oh!

elasticmachine commented Oct 2, 2020 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

Terraform Apply on x-pack/metricbeat/module/aws

Terraform Apply on x-pack/metricbeat/module/aws

💚 Flaky test report

Test stats 🧪

Uh oh!

chrisronline commented Oct 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Nov 6, 2020

🐛 Flaky test report

Test stats 🧪

Genuine test errors

Uh oh!

chrisronline commented Nov 9, 2020

Uh oh!

ycombinator Dec 14, 2020

Choose a reason for hiding this comment

Uh oh!

chrisronline Dec 15, 2020

Choose a reason for hiding this comment

Uh oh!

ycombinator Dec 14, 2020

Choose a reason for hiding this comment

Uh oh!

ycombinator Dec 15, 2020

Choose a reason for hiding this comment

Uh oh!

sayden Dec 16, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ycombinator left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

elasticmachine commented Oct 2, 2020 •

edited by jenkins-beats-ci bot

Loading

`Terraform Apply on x-pack/metricbeat/module/aws`

`Terraform Apply on x-pack/metricbeat/module/aws`

chrisronline commented Oct 23, 2020 •

edited

Loading