[Metricbeat] gcp: add GKE metricset by endorama · Pull Request #26824 · elastic/beats

endorama · 2021-07-09T14:22:45Z

What does this PR do?

Add a lightweight module to pull GCP GKE metrics.

Metrics gathered are all GA metrics as of today available for Kubernetes on GCP.

GCP Docs: https://cloud.google.com/monitoring/api/metrics_kubernetes

Why is it important?

Implement GKE integration as requested in elastic/integrations#816

Dates

Development Start date: Tue, Aug 17 2021
Feature Freeze date: Wed, Aug 18 2021
Release date: Tue, Sep 21 2021

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

You can build and run the metricbeat binary from this PR to collect GKE metrics. You may use this configuration for gcp.yml module:

- module: gcp
  metricsets:
    - gke
  zone: "us-"
  project_id: "<fill>"
  credentials_file_path: "<fill>"
  exclude_labels: false
  period: 1m

Related issues

Closes [Google Cloud] Add GKE integration integrations#816
Requires [Metricbeat] gcp: allow service metric prefix override #26960
Requires [Metricbeat] gcp: refactor service based metric filter creation #26973
Requires [Metricbeat] gcp: use ServiceMetrixPrefix in field extractor too #27286
Relates [Metricbeat] gcp: fields do not respect naming best practices #27231
NOTE: due to this issue the fields do not align with Beats naming convention or ECS, to keep the behaviour across Metricsets consistent. Once this is addressed I'll tackle migrating all Metricsets to the correct format.

Use cases

Feature: GKE metric collection

  GKE is the dedicated Kubernetes distribution from GCP. It exposes a set of dedicated 
  metrics useful for Kubernetes operators and developers.

  Scenario: Collect GKE specific metrics
	Given that gke metricset collection has been enabled and configured
    When running metricbeat
	Then I receive GKE specific metrics in Elasticsearch

Screenshots

Logs

Metricbeat Module / Dataset release checklist

This checklist is intended for Devs which create or update a module to make sure modules are consistent.

Modules

~~For a metricset to go GA, the following criterias should be met:~~

gcp module is currently in beta.

Metricbeat module

Example data.json exists and an automated way to generate it exists (go test -data)
Test environment in Docker exist for integration tests

elasticmachine · 2021-07-09T14:50:54Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-17T21:06:04.733+0000
Duration: 101 min 29 sec
Commit: 9dbbbaa

Test stats 🧪

Test	Results
Failed	0
Passed	9495
Skipped	2519
Total	12014

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	9495
Skipped	2519
Total	12014

mtojek

This is a good start, keep pushing the missing items. You can copy the TODO list from the template: https://github.com/elastic/beats/issues/new?assignees=&labels=&template=module-checklist.md

I think that the next step is to make it working and passing lint.

mtojek · 2021-07-12T07:49:00Z

x-pack/metricbeat/module/gcp/gke/manifest.yml

To review available metrics, you can sync with @sorantis .

mtojek · 2021-07-12T07:51:41Z

x-pack/metricbeat/module/gcp/gke/_meta/fields.yml

I think we used pct for percentages. See:

- name: memory.broker.pct description: The percentage of the memory limit used. type: scaled_float format: percent

Sometimes you need an additional Javascript processor to convert to the right format:

processors: - script: lang: javascript source: > function process(event) { var broker_memory_broker_pct = event.Get("activemq.broker.memory.broker.pct") if (broker_memory_broker_pct != null) { event.Put("activemq.broker.memory.broker.pct", broker_memory_broker_pct / 100.0) } var broker_memory_temp_pct = event.Get("activemq.broker.memory.temp.pct") if (broker_memory_temp_pct != null) { event.Put("activemq.broker.memory.temp.pct", broker_memory_temp_pct / 100.0) } var broker_memory_store_pct = event.Get("activemq.broker.memory.store.pct") if (broker_memory_store_pct != null) { event.Put("activemq.broker.memory.store.pct", broker_memory_store_pct / 100.0) } }

(based on the ActiveMQ module.

mtojek · 2021-07-12T07:51:54Z

x-pack/metricbeat/module/gcp/gke/_meta/fields.yml

Hm.. empty name?

mtojek · 2021-07-12T07:53:08Z

x-pack/metricbeat/module/gcp/gke/_meta/fields.yml

Try to follow the Beats naming convention: https://www.elastic.co/guide/en/beats/devguide/current/event-conventions.html

Should be max_bytes or allocatable.bytes?

In this case - memory.allocable.bytes. But this comment applies to all fields you submitted.

:) Will update all occurrences

kaiyan-sheng

I see this PR is still in draft, just want to bring elastic/ecs#1441 to your attention. As a part of the inventory schema work, we are planning to add some additional container metrics. It would be good if you can adopt these fields in this new metricset 🙂

endorama · 2021-07-21T15:58:02Z

To keep consistency with the GCP documentation and implementation, I'll rename gke to kubernetes. It's the name used in the GCP docs so I think the consistency is a must here.

endorama · 2021-08-10T09:26:50Z

Found a bug in #26960, fix split in #27286

endorama · 2021-08-11T10:17:01Z

Added GKE overview dashboard. Preview:

endorama · 2021-08-17T16:30:33Z

#27416 has been merged

Rebasing onto master and triggering a new CI check.

kaiyan-sheng

LGTM! Feel free to ship the dashboard separately.

kaiyan-sheng · 2021-08-17T17:02:39Z

When shipping a dashboard, please also add a screenshot into this folder under Metricbeat. Also reference the dashboard screenshot in the documentation.

endorama · 2021-08-17T17:30:39Z

I removed the dashboard... Now waiting on the CI

Add a lightweight module to pull GCP GKE metrics. Metrics are all GA metrics as of today available for Kubernetes on GCP. Docs: https://cloud.google.com/monitoring/api/metrics_kubernetes

In elastic#27231 I discovered that all `gcp` metricsets do not respect the Beats naming conventions. In order to preserve consistency across metricset this commit reverts the changes that aligned the newly added metrics to naming conventions.

Add a lightweight module to pull GCP GKE metrics. Metrics are all GA metrics as of today available for Kubernetes on GCP. Docs: cloud.google.com/monitoring/api/metrics_kubernetes (cherry picked from commit 1140740)

Add a lightweight module to pull GCP GKE metrics. Metrics are all GA metrics as of today available for Kubernetes on GCP. Docs: cloud.google.com/monitoring/api/metrics_kubernetes (cherry picked from commit 1140740) Co-authored-by: endorama <526307+endorama@users.noreply.github.com>

Co-authored-by: Edoardo Tenani <edoardo.tenani@elastic.co>

Add a lightweight module to pull GCP GKE metrics. Metrics are all GA metrics as of today available for Kubernetes on GCP. Docs: cloud.google.com/monitoring/api/metrics_kubernetes

endorama added the enhancement label Jul 9, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 9, 2021

mtojek self-requested a review July 12, 2021 07:04

mtojek assigned endorama Jul 12, 2021

mtojek reviewed Jul 12, 2021

View reviewed changes

kaiyan-sheng reviewed Jul 16, 2021

View reviewed changes

endorama force-pushed the xpack-gcp-gke branch 3 times, most recently from ea66269 to 8d22db0 Compare July 19, 2021 15:22

This was referenced Jul 19, 2021

[Metricbeat] gcp: allow service metric prefix override #26960

Merged

[Metricbeat] gcp: refactor service based metric filter creation #26973

Merged

endorama force-pushed the xpack-gcp-gke branch 2 times, most recently from 3af2e22 to 28f4a83 Compare July 20, 2021 12:55

endorama changed the title ~~add GCP GKE metricbeat metricset~~ [Metricbeat] gcp: add GKE metricset Jul 20, 2021

endorama force-pushed the xpack-gcp-gke branch from 28f4a83 to 4497059 Compare July 20, 2021 15:39

endorama force-pushed the xpack-gcp-gke branch from 4497059 to f3d1ab1 Compare July 27, 2021 14:31

kaiyan-sheng added the Team:Integrations Label for the Integrations team label Jul 27, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 27, 2021

endorama added the backport-v7.15.0 Automated backport with mergify label Aug 4, 2021

endorama mentioned this pull request Aug 4, 2021

[Metricbeat] gcp: fields do not respect naming best practices #27231

Closed

endorama force-pushed the xpack-gcp-gke branch 2 times, most recently from 2a1afe1 to 7b36f30 Compare August 10, 2021 09:26

endorama force-pushed the xpack-gcp-gke branch 3 times, most recently from 4bc794d to 2b93129 Compare August 13, 2021 08:04

endorama marked this pull request as ready for review August 13, 2021 08:14

endorama force-pushed the xpack-gcp-gke branch 2 times, most recently from e73f364 to 7e70b34 Compare August 17, 2021 16:29

kaiyan-sheng approved these changes Aug 17, 2021

View reviewed changes

endorama force-pushed the xpack-gcp-gke branch from 7e70b34 to 35abd50 Compare August 17, 2021 17:30

endorama added 10 commits August 17, 2021 23:05

add gke module

a917f44

Add a lightweight module to pull GCP GKE metrics. Metrics are all GA metrics as of today available for Kubernetes on GCP. Docs: https://cloud.google.com/monitoring/api/metrics_kubernetes

update fields

a9b681c

gcp.gke: improvements to fields

4f22a4e

gcp: update fields

ae18171

gcp: add gke specific region filter creation

455f3fb

gcp: revert fields naming

fb42de3

In elastic#27231 I discovered that all `gcp` metricsets do not respect the Beats naming conventions. In order to preserve consistency across metricset this commit reverts the changes that aligned the newly added metrics to naming conventions.

gcp: update assets

649ac7b

gcp: mage check

6e16258

update metricbeat docs

9170796

add CHANGELOG.next entry

9dbbbaa

endorama force-pushed the xpack-gcp-gke branch from 35abd50 to 9dbbbaa Compare August 17, 2021 21:05

jsoriano added the backport-v7.16.0 Automated backport with mergify label Aug 20, 2021

endorama merged commit 1140740 into elastic:master Aug 20, 2021

endorama deleted the xpack-gcp-gke branch August 20, 2021 08:39

mergify bot mentioned this pull request Aug 20, 2021

[7.x](backport #26824) [Metricbeat] gcp: add GKE metricset #27521

Merged

mergify bot mentioned this pull request Aug 20, 2021

[7.15](backport #26824) [Metricbeat] gcp: add GKE metricset #27522

Merged

endorama added a commit that referenced this pull request Aug 20, 2021

[7.15](backport #26824) [Metricbeat] gcp: add GKE metricset (#27522)

5f586f0

Co-authored-by: Edoardo Tenani <edoardo.tenani@elastic.co>

This was referenced Aug 23, 2021

[Metricbeat] gcp: add GKE overview dashboard #27532

Merged

[Metricbeat] gcp: add gke metricset documentation #27613

Merged

Conversation

endorama commented Jul 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Dates

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Metricbeat Module / Dataset release checklist

Modules

Metricbeat module

Uh oh!

elasticmachine commented Jul 9, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

mtojek left a comment

Choose a reason for hiding this comment

Uh oh!

mtojek Jul 12, 2021

Choose a reason for hiding this comment

Uh oh!

mtojek Jul 12, 2021

Choose a reason for hiding this comment

Uh oh!

mtojek Jul 12, 2021

Choose a reason for hiding this comment

Uh oh!

mtojek Jul 12, 2021

Choose a reason for hiding this comment

Uh oh!

endorama Jul 14, 2021

Choose a reason for hiding this comment

Uh oh!

mtojek Jul 14, 2021

Choose a reason for hiding this comment

Uh oh!

endorama Jul 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

endorama commented Jul 21, 2021

Uh oh!

endorama commented Aug 10, 2021

Uh oh!

endorama commented Aug 11, 2021

Uh oh!

endorama commented Aug 17, 2021

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

kaiyan-sheng commented Aug 17, 2021

Uh oh!

endorama commented Aug 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

endorama commented Jul 9, 2021 •

edited

Loading

elasticmachine commented Jul 9, 2021 •

edited by jenkins-beats-ci bot

Loading

endorama Jul 14, 2021 •

edited

Loading