Skip to content

Cherry-pick #20171 to 7.x: Add host inventory metrics to ec2 metricset#20673

Merged
kaiyan-sheng merged 1 commit intoelastic:7.xfrom
kaiyan-sheng:backport_20171_7.x
Aug 20, 2020
Merged

Cherry-pick #20171 to 7.x: Add host inventory metrics to ec2 metricset#20673
kaiyan-sheng merged 1 commit intoelastic:7.xfrom
kaiyan-sheng:backport_20171_7.x

Conversation

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng commented Aug 18, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #20171 to 7.x branch. Original message:

What does this PR do?

This PR is to add proposed host common fields into ec2 metricset:

  • host.id
  • host.name
  • host.cpu.pct
  • host.network.in.bytes
  • host.network.in.packets
  • host.network.out.bytes
  • host.network.out.packets
  • host.disk.read.bytes
  • host.disk.write.bytes

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. Start Metricbeat AWS module: ./metricbeat modules enable aws
  2. Edit modules.d/aws.yml to only include ec2 metricset:
- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  metricsets:
    - ec2
  1. Change add_host_metadata processor config in metricbeat.yml file:
processors:
  - add_host_metadata:
      replace_fields: false
  1. Start metricbeat
  2. You should see metrics from ec2 metricset and includes fields listed above.
    (host.name only equals to EC2 instance name when EC2 instance name is available.)

Event Example

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "aws": {
        "ec2": {
            "cpu": {
                "credit_balance": 144,
                "credit_usage": 0.05477,
                "surplus_credit_balance": 0,
                "surplus_credits_charged": 0,
                "total": {
                    "pct": 1.166484417714406
                }
            },
            "diskio": {
                "read": {
                    "bytes": 0,
                    "bytes_per_sec": 0,
                    "count": 0,
                    "count_per_sec": 0
                },
                "write": {
                    "bytes": 0,
                    "bytes_per_sec": 0,
                    "count": 0,
                    "count_per_sec": 0
                }
            },
            "instance": {
                "core": {
                    "count": 1
                },
                "image": {
                    "id": "ami-04bc3da8f14823e88"
                },
                "monitoring": {
                    "state": "disabled"
                },
                "private": {
                    "dns_name": "ip-172-31-9-119.us-west-1.compute.internal",
                    "ip": "172.31.9.119"
                },
                "public": {
                    "dns_name": "ec2-13-52-163-56.us-west-1.compute.amazonaws.com",
                    "ip": "13.52.163.56"
                },
                "state": {
                    "code": 16,
                    "name": "running"
                },
                "threads_per_core": 1
            },
            "network": {
                "in": {
                    "bytes": 6644.4,
                    "bytes_per_sec": 22.148,
                    "packets": 44.6,
                    "packets_per_sec": 0.14866666666666667
                },
                "out": {
                    "bytes": 6016.6,
                    "bytes_per_sec": 20.055333333333333,
                    "packets": 35.4,
                    "packets_per_sec": 0.118
                }
            },
            "status": {
                "check_failed": 0,
                "check_failed_instance": 0,
                "check_failed_system": 0
            }
        },
        "tags": {
            "Name": "mysql-test",
            "created-by": "ks"
        }
    },
    "cloud": {
        "account": {
            "id": "428152502467",
            "name": "elastic-beats"
        },
        "availability_zone": "us-west-1b",
        "instance": {
            "id": "i-0516ddaca5c1d231f",
            "name": "mysql-test"
        },
        "machine": {
            "type": "t2.micro"
        },
        "provider": "aws",
        "region": "us-west-1"
    },
    "event": {
        "dataset": "aws.ec2",
        "duration": 115000,
        "module": "aws"
    },
    "host": {
        "cpu": {
            "pct": 1.166484417714406
        },
        "disk": {
            "read": {
                "bytes": 0
            }
        },
        "diskio": {
            "write": {
                "bytes": 0
            }
        },
        "id": "i-0516ddaca5c1d231f",
        "name": "mysql-test",
        "network": {
            "in": {
                "bytes": 6644.4,
                "packets": 44.6
            },
            "out": {
                "bytes": 6016.6,
                "packets": 35.4
            }
        }
    },
    "metricset": {
        "name": "ec2",
        "period": 10000
    },
    "service": {
        "type": "aws"
    }
}

Add host inventory metrics to ec2 metricset (#20171)
2e1da64 
* Add host inventory metrics to ec2 metricset
* move host fields into metricbeat/_meta/fields.common.yml

(cherry picked from commit 54d2268)
@kaiyan-sheng kaiyan-sheng added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Aug 18, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 18, 2020

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 18, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Aug 18, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 18, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20673 opened]

  • Start Time: 2020-08-18T18:23:57.272+0000

  • Duration: 66 min 31 sec

Test stats 🧪

Test Results
Failed 0
Passed 3538
Skipped 797
Total 4335

@kaiyan-sheng kaiyan-sheng merged commit 0996442 into elastic:7.x Aug 20, 2020
@kaiyan-sheng kaiyan-sheng deleted the backport_20171_7.x branch August 20, 2020 14:11
@Bernhard-Fluehmann Bernhard-Fluehmann mentioned this pull request Sep 29, 2020
Closed
@zube zube bot removed the [zube]: Done label Nov 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jsoriano jsoriano jsoriano approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

backport Team:Platforms Label for the Integrations - Platforms team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kaiyan-sheng @elasticmachine @jsoriano