Fix annotation enrichment by tetianakravchenko · Pull Request #29605 · elastic/beats

tetianakravchenko · 2021-12-24T14:20:24Z

Signed-off-by: Tetiana Kravchenko tetiana.kravchenko@elastic.co

What does this PR do?

node annotations enrichment doesn't work, with the config:

add_resource_metadata:
  node:
    include_annotations: ["test.annotation"]

events were not enriched with nodes annotation metadata

Why is it important?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Create a kind kubernetes cluster.
run metricbeat (how-to) with k8s configuration:

kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      add_resource_metadata:
        node:
          include_annotations: ["another.annotation", "example.annotation"]
      period: 10s
      host: ${NODE_NAME}
      hosts: ["https://${NODE_NAME}:10250"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.verification_mode: "none"

edit kind node so that the list of annotations include another.annotation and example.annotation:

$ kubectl describe node kind-control-plane
...
Annotations:        another.annotation: another
                    example.annotation: example
                    kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
...

Related issues

Closes Kubernetes: missing annotations enrichment #29597

Use cases

Screenshots

All node/namespace metadata:

note that annotations metadata were only added to pod and container metricsets:

Logs

mergify · 2021-12-24T14:20:29Z

This pull request does not have a backport label. Could you fix it @tetianakravchenko? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2021-12-24T14:46:35Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-01-12T11:46:46.538+0000
Duration: 105 min 48 sec
Commit: 955ab99

Test stats 🧪

Test	Results
Failed	0
Passed	47850
Skipped	4284
Total	52134

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2021-12-24T15:49:42Z

Pinging @elastic/integrations (Team:Integrations)

tetianakravchenko · 2021-12-25T15:41:42Z

/package

mergify · 2022-01-04T09:49:15Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add_resource_meta_enrichment upstream/add_resource_meta_enrichment
git merge upstream/master
git push upstream add_resource_meta_enrichment

MichaelKatsoulis · 2022-01-04T15:10:23Z

@tetianakravchenko great! Could you add some Kibana screenshots with the node annotations being added ?

tetianakravchenko · 2022-01-05T14:17:49Z

@MichaelKatsoulis I've added screenshots and the steps to reproduce test

MichaelKatsoulis · 2022-01-05T14:24:07Z

CHANGELOG.next.asciidoc

This is not very descriptive of your change. Something like Enrich kubernetes metadata with node annotations

tetianakravchenko · 2022-01-05T16:33:26Z

/package

tetianakravchenko · 2022-01-10T09:54:26Z

/package

mergify · 2022-01-11T20:49:35Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add_resource_meta_enrichment upstream/add_resource_meta_enrichment
git merge upstream/master
git push upstream add_resource_meta_enrichment

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

tetianakravchenko · 2022-01-12T15:50:01Z

/package

* add annotation enrichment Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * add pr number Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * fix tests Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * update changelog message Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * clean up changelog after rebase Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> (cherry picked from commit 2917392) # Conflicts: # libbeat/common/kubernetes/metadata/pod_test.go

* Fix annotation enrichment (#29605) * add annotation enrichment Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * add pr number Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * fix tests Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * update changelog message Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * clean up changelog after rebase Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> (cherry picked from commit 2917392) # Conflicts: # libbeat/common/kubernetes/metadata/pod_test.go * Update CHANGELOG.next.asciidoc * Fix conflict in tests Co-authored-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* upstream/7.17: (30 commits) [7.17](backport #29966) Add the Elastic product origin header when talking to Elasticsearch or Kibana. (#30000) [Heartbeat] Change size of data on ICMP packet (#29948) (#29978) Add clarification about enableing dashboard loading (#29985) (#29989) Improve aws-s3 gzip file detection to avoid false negatives (#29969) (#29974) ci: docker login step for pulling then pushing (#29960) (#29963) x-pack/auditbeat/module/system/socket: get full length path and arg from /proc when not available from kprobe (#29410) (#29958) [Automation] Update elastic stack version to 7.17.0-ab4975a2 for testing (#29956) [Automation] Update elastic stack version to 7.17.0-1bd58b32 for testing (#29938) [7.17](backport #29913) [Metricbeat] gcp.gke: fix overview dashboard (#29914) [7.17](backport #29605) Fix annotation enrichment (#29834) [Automation] Update elastic stack version to 7.17.0-e1efbe3a for testing (#29922) [Automation] Update elastic stack version to 7.17.0-68da5d12 for testing (#29904) [7.17][Heartbeat] Defer monitor / ICMP errors to monitor runtime / ES (backport #29413) (#29896) Merge pull request from GHSA-rj4h-hqvq-cc6q [7.17](backport #29681) Change docker image from CentOS 7 to Ubuntu 20.04 (#29817) Fix YAML indentation in `parsers` examples (#29663) (#29894) [Automation] Update elastic stack version to 7.17.0-079761a0 for testing (#29864) Fix Filebeat dissect processor field tokenization in documentation (#29680) (#29883) Enable require_alias for Bulk requests for all actions when target is a write alias (#29879) Update Index template loading guide to use the correct endpoint (#29869) (#29877) ...

* upstream/7.17: (30 commits) [7.17](backport elastic#29966) Add the Elastic product origin header when talking to Elasticsearch or Kibana. (elastic#30000) [Heartbeat] Change size of data on ICMP packet (elastic#29948) (elastic#29978) Add clarification about enableing dashboard loading (elastic#29985) (elastic#29989) Improve aws-s3 gzip file detection to avoid false negatives (elastic#29969) (elastic#29974) ci: docker login step for pulling then pushing (elastic#29960) (elastic#29963) x-pack/auditbeat/module/system/socket: get full length path and arg from /proc when not available from kprobe (elastic#29410) (elastic#29958) [Automation] Update elastic stack version to 7.17.0-ab4975a2 for testing (elastic#29956) [Automation] Update elastic stack version to 7.17.0-1bd58b32 for testing (elastic#29938) [7.17](backport elastic#29913) [Metricbeat] gcp.gke: fix overview dashboard (elastic#29914) [7.17](backport elastic#29605) Fix annotation enrichment (elastic#29834) [Automation] Update elastic stack version to 7.17.0-e1efbe3a for testing (elastic#29922) [Automation] Update elastic stack version to 7.17.0-68da5d12 for testing (elastic#29904) [7.17][Heartbeat] Defer monitor / ICMP errors to monitor runtime / ES (backport elastic#29413) (elastic#29896) Merge pull request from GHSA-rj4h-hqvq-cc6q [7.17](backport elastic#29681) Change docker image from CentOS 7 to Ubuntu 20.04 (elastic#29817) Fix YAML indentation in `parsers` examples (elastic#29663) (elastic#29894) [Automation] Update elastic stack version to 7.17.0-079761a0 for testing (elastic#29864) Fix Filebeat dissect processor field tokenization in documentation (elastic#29680) (elastic#29883) Enable require_alias for Bulk requests for all actions when target is a write alias (elastic#29879) Update Index template loading guide to use the correct endpoint (elastic#29869) (elastic#29877) ...

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 24, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Dec 24, 2021

tetianakravchenko requested a review from ChrsMark December 24, 2021 15:22

tetianakravchenko added bug Team:Integrations Label for the Integrations team labels Dec 24, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 24, 2021

tetianakravchenko requested a review from MichaelKatsoulis December 27, 2021 13:12

tetianakravchenko added the backport-7.17 Automated backport to the 7.17 branch with mergify label Jan 5, 2022

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Jan 5, 2022

MichaelKatsoulis reviewed Jan 5, 2022

View reviewed changes

MichaelKatsoulis approved these changes Jan 5, 2022

View reviewed changes

tetianakravchenko force-pushed the add_resource_meta_enrichment branch from ae75c6f to 49d1911 Compare January 10, 2022 17:25

tetianakravchenko added 4 commits January 12, 2022 12:45

add annotation enrichment

5a5e70f

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

add pr number

9a83a1c

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

fix tests

8674e1e

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

update changelog message

8d15052

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

tetianakravchenko force-pushed the add_resource_meta_enrichment branch from 6edeecb to 8d15052 Compare January 12, 2022 11:45

clean up changelog after rebase

955ab99

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

tetianakravchenko merged commit 2917392 into elastic:master Jan 13, 2022

tetianakravchenko deleted the add_resource_meta_enrichment branch January 13, 2022 09:39

mergify bot mentioned this pull request Jan 13, 2022

[7.17](backport #29605) Fix annotation enrichment #29834

Merged

Conversation

tetianakravchenko commented Dec 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

mergify bot commented Dec 24, 2021

Uh oh!

elasticmachine commented Dec 24, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

elasticmachine commented Dec 24, 2021

Uh oh!

tetianakravchenko commented Dec 25, 2021

Uh oh!

mergify bot commented Jan 4, 2022

Uh oh!

MichaelKatsoulis commented Jan 4, 2022

Uh oh!

tetianakravchenko commented Jan 5, 2022

Uh oh!

MichaelKatsoulis Jan 5, 2022

Choose a reason for hiding this comment

Uh oh!

tetianakravchenko commented Jan 5, 2022

Uh oh!

tetianakravchenko commented Jan 10, 2022

Uh oh!

mergify bot commented Jan 11, 2022

Uh oh!

tetianakravchenko commented Jan 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tetianakravchenko commented Dec 24, 2021 •

edited

Loading

elasticmachine commented Dec 24, 2021 •

edited by jenkins-beats-ci bot

Loading