Skip to content

Add container ECS fields in kubernetes metadata#20984

Merged
ChrsMark merged 13 commits intoelastic:masterfrom
ChrsMark:fix_container_id_issue
Sep 9, 2020
Merged

Add container ECS fields in kubernetes metadata#20984
ChrsMark merged 13 commits intoelastic:masterfrom
ChrsMark:fix_container_id_issue

Conversation

@ChrsMark
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark commented Sep 4, 2020 •
edited
Loading

What does this PR do?

This PR adds container.id, container.runtime and container.image.name in kubernetes metadata to be compliant with https://www.elastic.co/guide/en/ecs/current/ecs-container.html

Why is it important?

In final events the metadata do not include container.id, container.runtime and container.image.name.

Related issues

How to test this PR

  1. Configure and run Filebeat in k8s:
filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

Make sure that container.id, container.runtime and container.image.name exist in the final documents in ES.

  1. Configure and run Filebeat in k8s:
filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            indexers:
            - container:
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

Screenshots

Screenshot 2020-09-08 at 13 56 03

@ChrsMark
Add container id in kubemeta initialisation
fd0583c 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark added bug needs_backport PR is waiting to be backported to other branches. v7.9.0 v7.10.0 labels Sep 4, 2020
@ChrsMark ChrsMark requested a review from jsoriano September 4, 2020 12:22
@ChrsMark ChrsMark self-assigned this Sep 4, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 4, 2020
@ChrsMark ChrsMark added the Team:Platforms Label for the Integrations - Platforms team label Sep 4, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 4, 2020

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 4, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 4, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20984 updated]

  • Start Time: 2020-09-09T07:11:12.179+0000

  • Duration: 76 min 8 sec

Test stats 🧪

Test Results
Failed 0
Passed 19781
Skipped 1837
Total 21618

@ChrsMark
Add runtime in initialisation
81399ae 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@jsoriano jsoriano added v7.9.2 and removed v7.9.0 labels Sep 4, 2020
jsoriano
jsoriano approved these changes Sep 4, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@jsoriano jsoriano left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! LGTM

It'd be nice to have some test to detect the loss of these fields. We would also need to check if add_kubernetes_metadata has similar issues.

@ChrsMark
Add tests and fields
56864c8 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
Add container runtime
6938cbd 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark requested a review from jsoriano September 7, 2020 07:39
@ChrsMark ChrsMark changed the title Add container id in kubemeta initialisation Add container id and runtime in kubernetes metadata Sep 7, 2020
@ChrsMark
copy container.* to ECS fields
8bebb22 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark requested a review from a team as a code owner September 7, 2020 13:11
@ChrsMark
wip
fa0dd4b 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
wip
a70bac0 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark changed the title Add container id and runtime in kubernetes metadata Add container ECS fields in kubernetes metadata Sep 8, 2020
@ChrsMark
Add ECS fields properly
ac88a68 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
fix tests
d634e0f 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
Add changelog entry
18c5979 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
fmt
cb01e35 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
Add ECS fields in IPPortIndexer
042cbe2 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
jsoriano
jsoriano approved these changes Sep 8, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@ChrsMark
fmt
5701e23 
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark merged commit bcb4e0c into elastic:master Sep 9, 2020
ChrsMark added a commit to ChrsMark/beats that referenced this pull request Sep 9, 2020
@ChrsMark
Add container ECS fields in kubernetes metadata (elastic#20984)
7bdb522 
(cherry picked from commit bcb4e0c)
@ChrsMark ChrsMark mentioned this pull request Sep 9, 2020
Merged
@ChrsMark ChrsMark removed the needs_backport PR is waiting to be backported to other branches. label Sep 9, 2020
@ChrsMark ChrsMark mentioned this pull request Sep 9, 2020
Merged
ChrsMark added a commit to ChrsMark/beats that referenced this pull request Sep 9, 2020
@ChrsMark
Add container ECS fields in kubernetes metadata (elastic#20984)
6ed92b0 
(cherry picked from commit bcb4e0c)
ChrsMark added a commit that referenced this pull request Sep 9, 2020
@ChrsMark
Cherry-pick #20984 to 7.9: Add container ECS fields in kubernetes met…
768dfc9 
…adata (#21034)
ChrsMark added a commit that referenced this pull request Sep 9, 2020
@ChrsMark
Add container ECS fields in kubernetes metadata (#20984) (#21033)
54e379c 
(cherry picked from commit bcb4e0c)
v1v added a commit to v1v/beats that referenced this pull request Sep 14, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-10
cc07c42 
* upstream/master: (362 commits)
  Add vendoring to Google Cloud Functions again (elastic#21070)
  [Elastic Agent] Add fleet.host.id for sending to endpoint. (elastic#21042)
  Do not need Google credentials before using it (elastic#21072)
  [Filebeat][New Module] Zoom webhook module (elastic#20414)
  Add support for GMT timezone offset in decode_cef (elastic#20993)
  Filebeat: Fix random error on harvester close (elastic#21048)
  Add ingress controller dashboards (elastic#21052)
  Fix loggers in composable module. (elastic#21047)
  [Ingest Manager] Increase kibana client timeout to 5 minutes (elastic#21037)
  Add changelog. (elastic#21041)
  [Elastic Agent] Add support for EQL based conditions (elastic#20994)
  Disable Kafka metricsets based on Jolokia (elastic#20989)
  Update apm agent (elastic#21031)
  Add container ECS fields in kubernetes metadata (elastic#20984)
  Sanitize event.host in Metricbeat (elastic#21022)
  Update api-keys.asciidoc - API key prerequisites (elastic#21026)
  [Filebeat][suricata] Map x509 for suricata/eve fileset (elastic#20973)
  [Filebeat][santa] Map x509 fields in santa module (elastic#20976)
  [Filebeat][fortinet] Map x509 ecs fields for fortinet fw fileset (elastic#20983)
  Bump zeek kerberos/ssl/x509 ecs version (elastic#21003)
  ...
v1v added a commit to v1v/beats that referenced this pull request Sep 14, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-7-64
fcc41b4 
* upstream/master: (364 commits)
  Add vendoring to Google Cloud Functions again (elastic#21070)
  [Elastic Agent] Add fleet.host.id for sending to endpoint. (elastic#21042)
  Do not need Google credentials before using it (elastic#21072)
  [Filebeat][New Module] Zoom webhook module (elastic#20414)
  Add support for GMT timezone offset in decode_cef (elastic#20993)
  Filebeat: Fix random error on harvester close (elastic#21048)
  Add ingress controller dashboards (elastic#21052)
  Fix loggers in composable module. (elastic#21047)
  [Ingest Manager] Increase kibana client timeout to 5 minutes (elastic#21037)
  Add changelog. (elastic#21041)
  [Elastic Agent] Add support for EQL based conditions (elastic#20994)
  Disable Kafka metricsets based on Jolokia (elastic#20989)
  Update apm agent (elastic#21031)
  Add container ECS fields in kubernetes metadata (elastic#20984)
  Sanitize event.host in Metricbeat (elastic#21022)
  Update api-keys.asciidoc - API key prerequisites (elastic#21026)
  [Filebeat][suricata] Map x509 for suricata/eve fileset (elastic#20973)
  [Filebeat][santa] Map x509 fields in santa module (elastic#20976)
  [Filebeat][fortinet] Map x509 ecs fields for fortinet fw fileset (elastic#20983)
  Bump zeek kerberos/ssl/x509 ecs version (elastic#21003)
  ...
@ChrsMark ChrsMark mentioned this pull request Feb 3, 2021
Closed
This was referenced Mar 5, 2021
Merged
Merged
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
@ChrsMark
Cherry-pick elastic#20984 to 7.9: Add container ECS fields in kuberne…
7575b0d 
…tes metadata (elastic#21034)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jsoriano jsoriano jsoriano approved these changes

Assignees

@ChrsMark ChrsMark

Labels

bug Team:Platforms Label for the Integrations - Platforms team v7.9.2 v7.10.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Container id not included in Kubernetes events

5 participants

@ChrsMark @elasticmachine @jsoriano @masci @andresrc