Skip to content

Cherry-pick #14823 to 7.x: Add autodiscover for aws_ec2#16184

Merged
kaiyan-sheng merged 1 commit intoelastic:7.xfrom
kaiyan-sheng:backport_14823_7.x
Feb 7, 2020
Merged

Cherry-pick #14823 to 7.x: Add autodiscover for aws_ec2#16184
kaiyan-sheng merged 1 commit intoelastic:7.xfrom
kaiyan-sheng:backport_14823_7.x

Conversation

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng commented Feb 6, 2020

Cherry-pick of PR #14823 to 7.x branch. Original message:

This PR is to add aws_ec2 as a new provider for autodiscover. It allows autodiscover to react to EC2 instances status, information currently included in autodiscover events, to be used in autodiscover conditions are:

cloud.availability_zone
cloud.instance.id
cloud.machine.type
cloud.provider
cloud.region

and

aws.ec2.architecture
aws.ec2.image.id
aws.ec2.kernel.id
aws.ec2.monitoring.state
aws.ec2.private.dns_name
aws.ec2.private.ip
aws.ec2.public.dns_name
aws.ec2.public.ip
aws.ec2.root_device_name
aws.ec2.state.code
aws.ec2.state.name
aws.ec2.subnet.id
aws.ec2.tags
aws.ec2.vpc.id

Autodiscover config looks like:

logging.level: debug
metricbeat.autodiscover:
  # List of enabled autodiscover providers
  providers:
    - type: aws_ec2
      period: 1m
      credential_profile_name: elastic-beats
      templates:
        - condition:
            equals:
              aws.ec2.tags.created-by: "ks"
          config:
            - module: mysql
              metricsets: ["status", "galera_status"]
              period: 10s
              hosts: ["tcp(${data.aws.ec2.public.ip}:3306)/"]
              username: kaiyan
              password: kaiyan

TODO List

  • Documentation
  • Unit tests
  • Test with mysql module in Metricbeat

Event generated by autodiscovered mysql:

{
  "_index": "metricbeat-8.0.0-2020.02.05-000001",
  "_type": "_doc",
  "_id": "y5crGHABZy5rQOjN9Rqq",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-02-06T01:44:48.040Z",
    "metricset": {
      "period": 10000,
      "name": "status"
    },
    "error": {
      "message": "dial tcp 3.86.223.141:3306: i/o timeout"
    },
    "aws": {
      "ec2": {
        "subnet": {
          "id": "subnet-76404958"
        },
        "public": {
          "dns_name": "ec2-3-86-223-141.compute-1.amazonaws.com",
          "ip": "3.86.223.141"
        },
        "state": {
          "name": "running",
          "code": 16
        },
        "vpc": {
          "id": "vpc-e8722d92"
        },
        "private": {
          "ip": "172.31.81.156",
          "dns_name": "ip-172-31-81-156.ec2.internal"
        },
        "tags": {
          "service": "mysql",
          "Name": "mysql-test",
          "created-by": "ks"
        },
        "root_device_name": "/dev/xvda",
        "monitoring": {
          "state": "disabled"
        },
        "kernel": {
          "id": ""
        },
        "architecture": "x86_64",
        "image": {
          "id": "ami-00068cd7555f543d5"
        }
      }
    },
    "cloud": {
      "region": "us-east-1",
      "instance": {
        "id": "i-05381e495aa0f95ef"
      },
      "machine": {
        "type": "t2.micro"
      },
      "availability_zone": "us-east-1b",
      "provider": "aws"
    },
    "host": {
      "architecture": "x86_64",
      "name": "KaiyanMacBookPro",
      "os": {
        "kernel": "17.7.0",
        "build": "17G10021",
        "platform": "darwin",
        "version": "10.13.6",
        "family": "darwin",
        "name": "Mac OS X"
      },
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D",
      "hostname": "KaiyanMacBookPro"
    },
    "agent": {
      "hostname": "KaiyanMacBookPro",
      "id": "23fd6ee9-2255-43f3-9565-78a051337c04",
      "version": "8.0.0",
      "type": "metricbeat",
      "ephemeral_id": "5d697e16-910b-4363-a375-f36e8e362ffc"
    },
    "ecs": {
      "version": "1.4.0"
    },
    "event": {
      "dataset": "mysql.status",
      "module": "mysql",
      "duration": 10004769210
    },
    "service": {
      "address": "3.86.223.141:3306",
      "type": "mysql"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-02-06T01:44:48.040Z"
    ]
  },
  "sort": [
    1580953488040
  ]
}

How to test it

  • Start an EC2 instance in AWS
  • Add created-by: ks as tag for this EC2 instance
  • Install and start mysql service in the EC2 instance
  • Change default password of mysql
  • Now you can try autodiscover with provider aws_ec2 by attach config below to metricbeat.yml under x-pack/metricbeat:
logging.level: debug
metricbeat.autodiscover:
  # List of enabled autodiscover providers
  providers:
    - type: aws_ec2
      period: 1m
      credential_profile_name: elastic-beats
      templates:
        - condition:
            equals:
              aws.ec2.tags.created-by: "ks"
          config:
            - module: mysql
              metricsets: ["status", "galera_status"]
              period: 10s
              hosts: ["tcp(${data.aws.ec2.public.ip}:3306)/"]
              username: kaiyan
              password: kaiyan

closes #12518

Add autodiscover for aws_ec2 (#14823)
a0e6f81 
* Add autodiscover for aws_ec2
* Add aws.ec2.* to autodiscover template

(cherry picked from commit adcd962)
@kaiyan-sheng kaiyan-sheng self-assigned this Feb 6, 2020
ChrsMark
ChrsMark approved these changes Feb 7, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@kaiyan-sheng kaiyan-sheng merged commit 3e3b260 into elastic:7.x Feb 7, 2020
@kaiyan-sheng kaiyan-sheng deleted the backport_14823_7.x branch February 7, 2020 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

backport review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kaiyan-sheng @ChrsMark