Add Agent example input autodiscover config for standalone k8s by ChrsMark · Pull Request #24157 · elastic/beats

ChrsMark · 2021-02-22T14:31:25Z

What does this PR do?

This PR is a follow up of #23938 to add a commented out section within the k8s manifest as an example for enabling autodiscover for specific target based on dynamic inputs and k8s provider.

It comes along with elastic/observability-docs#401.

Part of #23613

Why is it important?

So as to provide an easy way for the users to define their own autodiscover condition based inputs.

How to test this PR locally

Uncoment the added section
Deploy Agent on k8s: kubectl apply -f elastic-agent-standalone-kubernetes.yml
Deploy a target Redis Pod:

apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  clusterIP: None
  ports:
  - name: web
    port: 6379
    protocol: TCP
  selector:
    app: redis
  type: ClusterIP
---
apiVersion: v1
kind: Pod
metadata:
  name: redis
  labels:
    role: main
    app: redis
spec:
  containers:
    - name: redis
      image: redis

Verify that redis integration start collecting redis metrics from the Pod. In Kibana discover filter on redis.info for event.dataset and expect to events arriving looking like the one below inScreenshots part.

Screenshots

Sample event:

{
  "_index": ".ds-metrics-redis.info-default-2021.02.22-000001",
  "_type": "_doc",
  "_id": "m_cZyncB1P74KB3B7J-v",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2021-02-22T14:16:55.832Z",
    "service": {
      "version": "6.0.10",
      "address": "redis://10.244.1.8:6379",
      "type": "redis"
    },
    "os": {
      "full": "Linux 4.9.184-linuxkit x86_64"
    },
    "redis": {
      "info": {
        "stats": {
          "latest_fork_usec": 0,
          "keyspace": {
            "hits": 0,
            "misses": 0
          },
          "active_defrag": {
            "key_hits": 0,
            "key_misses": 0,
            "hits": 0,
            "misses": 0
          },
          "pubsub": {
            "patterns": 0,
            "channels": 0
          },
          "connections": {
            "received": 1,
            "rejected": 0
          },
          "keys": {
            "expired": 0,
            "evicted": 0
          },
          "commands_processed": 18,
          "sync": {
            "full": 0,
            "partial": {
              "err": 0,
              "ok": 0
            }
          },
          "instantaneous": {
            "input_kbps": 0,
            "output_kbps": 0,
            "ops_per_sec": 0
          },
          "migrate_cached_sockets": 0,
          "slave_expires_tracked_keys": 0,
          "net": {
            "input": {
              "bytes": 504
            },
            "output": {
              "bytes": 32651
            }
          }
        },
        "cluster": {
          "enabled": false
        },
        "clients": {
          "blocked": 0,
          "connected": 1,
          "longest_output_list": 0,
          "biggest_input_buf": 8,
          "max_output_buffer": 0,
          "max_input_buffer": 8
        },
        "persistence": {
          "loading": false,
          "rdb": {
            "bgsave": {
              "in_progress": false,
              "last_time": {
                "sec": -1
              },
              "current_time": {
                "sec": -1
              },
              "last_status": "ok"
            },
            "copy_on_write": {
              "last_size": 0
            },
            "last_save": {
              "changes_since": 0,
              "time": 1614003324
            }
          },
          "aof": {
            "fsync": {},
            "enabled": false,
            "rewrite": {
              "last_time": {
                "sec": -1
              },
              "current_time": {
                "sec": -1
              },
              "buffer": {},
              "in_progress": false,
              "scheduled": false
            },
            "bgrewrite": {
              "last_status": "ok"
            },
            "write": {
              "last_status": "ok"
            },
            "copy_on_write": {
              "last_size": 0
            },
            "buffer": {},
            "size": {}
          }
        },
        "replication": {
          "slave": {},
          "role": "master",
          "connected_slaves": 0,
          "master_offset": 0,
          "backlog": {
            "size": 1048576,
            "first_byte_offset": 0,
            "histlen": 0,
            "active": 0
          },
          "master": {
            "sync": {},
            "offset": 0,
            "second_offset": -1
          }
        },
        "memory": {
          "max": {
            "value": 0,
            "policy": "noeviction"
          },
          "fragmentation": {
            "ratio": 9.53,
            "bytes": 7031992
          },
          "active_defrag": {
            "is_running": false
          },
          "allocator": "jemalloc-5.1.0",
          "allocator_stats": {
            "allocated": 1135544,
            "active": 1355776,
            "resident": 4169728,
            "fragmentation": {
              "ratio": 1.19,
              "bytes": 220232
            },
            "rss": {
              "bytes": 2813952,
              "ratio": 3.08
            }
          },
          "used": {
            "rss": 7856128,
            "peak": 866680,
            "lua": 37888,
            "dataset": 43008,
            "value": 866680
          }
        },
        "cpu": {
          "used": {
            "sys_children": 0,
            "user_children": 0,
            "sys": 0.07,
            "user": 0.07
          }
        },
        "server": {
          "mode": "standalone",
          "run_id": "fa112bef9093d2ac5ccff855be0ae270375ad0f6",
          "lru_clock": 3390679,
          "hz": 10,
          "gcc_version": "8.3.0",
          "build_id": "1c3aed2f9fd88f03",
          "config_file": "",
          "git_dirty": "0",
          "uptime": 91,
          "tcp_port": 6379,
          "arch_bits": "64",
          "multiplexing_api": "epoll",
          "git_sha1": "00000000"
        },
        "slowlog": {
          "count": 0
        }
      }
    },
    "data_stream": {
      "type": "metrics",
      "dataset": "redis.info",
      "namespace": "default"
    },
    "elastic_agent": {
      "id": "9392caf0-cdfc-4abf-981b-f98cb6b9edad",
      "snapshot": true,
      "version": "8.0.0"
    },
    "host": {
      "hostname": "kind-worker",
      "architecture": "x86_64",
      "os": {
        "family": "redhat",
        "name": "CentOS Linux",
        "kernel": "4.9.184-linuxkit",
        "codename": "Core",
        "platform": "centos",
        "version": "7 (Core)"
      },
      "id": "17af4efd6a5716fb64a196f3cd53c4f8",
      "containerized": true,
      "name": "kind-worker",
      "ip": [
        "10.244.1.1",
        "10.244.1.1",
        "10.244.1.1",
        "172.18.0.2",
        "fc00:f853:ccd:e793::2",
        "fe80::42:acff:fe12:2"
      ],
      "mac": [
        "ba:4a:a1:a2:9a:81",
        "9a:80:f1:73:34:7a",
        "2e:0a:1c:40:46:f8",
        "02:42:ac:12:00:02"
      ]
    },
    "process": {
      "pid": 1
    },
    "event": {
      "module": "redis",
      "duration": 666897,
      "dataset": "redis.info"
    },
    "metricset": {
      "name": "info",
      "period": 10000
    },
    "kubernetes": {
      "namespace": "default",
      "pod": {
        "labels": {
          "role": "main",
          "app": "redis"
        },
        "name": "redis",
        "uid": "150148b5-b0a0-4237-b1ea-b2c2634fba59",
        "ip": "10.244.1.8"
      }
    },
    "agent": {
      "ephemeral_id": "7b49f72c-73c2-49be-99c9-b09d55fcd7b9",
      "id": "22e06d3f-a7cb-464c-b813-dee4302ca78a",
      "name": "kind-worker",
      "type": "metricbeat",
      "version": "8.0.0"
    },
    "ecs": {
      "version": "1.7.0"
    }
  },
  "fields": {
    "@timestamp": [
      "2021-02-22T14:16:55.832Z"
    ]
  },
  "highlight": {
    "event.dataset": [
      "@kibana-highlighted-field@redis.info@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1614003415832
  ]
}

Signed-off-by: chrismark <chrismarkou92@gmail.com>

elasticmachine · 2021-02-22T14:31:28Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-02-22T14:31:28Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-02-22T15:25:21Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #24157 opened
Start Time: 2021-02-22T14:32:11.177+0000
Duration: 53 min 54 sec
Commit: b828ce8

Trends 🧪

❕ Flaky test report

No test was executed to be analysed.

…-arm * upstream/master: (24 commits) Add example input autodsicover config (elastic#24157) Empty configuration options generate `<no value>` string for azure-eventhub input (elastic#24156) Remove OSS reference for kibana and elasticsearch (elastic#24164) Skip flaky TestActions on MacOSx (elastic#23966) [Filebeat][AWS] Fix vpcflow pipeline exception: Cannot invoke "Object.getClass()" because "receiver" is null (elastic#24167) [Elastic Agent] Fix docker entrypoint for elastic-agent. (elastic#24155) [PACKAGING] Push docker images with the architecture in the version (elastic#24121) [Agent] Add agent standalone manifests for system module & Pod's log collection (elastic#23938) indicator type url is in upper case (elastic#24152) [Filebeat] Document netflow internal_networks and set default (elastic#24110) [Filebeat] Adding fixes to the TI module (elastic#24133) [Enhancement] Add RotateOnStartup feature flag for file output (elastic#19347) [Ingest Manager] Fix: Successfully installed and enrolled agent running standalone (elastic#24128) Set Elastic licence type for APM server Beats update job (elastic#24122) Add logrotation section on Running Filebeat on k8s (elastic#24120) [CI] Run if manual UI (elastic#24116) [CI] enable x-pack/heartbeat in the CI (elastic#23873) chore: comment out the E2E (elastic#24109) chore: add-backport-next (elastic#24098) Adjust the position of the architecture name in Dockerlogbeat tarball (elastic#24095) ...

…dows-7 * upstream/master: [CI] Add ARM packaging (elastic#24041) Add example input autodsicover config (elastic#24157) Empty configuration options generate `<no value>` string for azure-eventhub input (elastic#24156)

Add example input autodsicover config

b828ce8

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark added Team:Integrations Label for the Integrations team kubernetes Enable builds in the CI for kubernetes Team:Elastic-Agent Label for the Agent team labels Feb 22, 2021

ChrsMark requested review from blakerouse and jsoriano February 22, 2021 14:31

ChrsMark self-assigned this Feb 22, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 22, 2021

jsoriano approved these changes Feb 23, 2021

View reviewed changes

ChrsMark merged commit 05b523a into elastic:master Feb 23, 2021

ChrsMark mentioned this pull request Feb 25, 2021

[Backport] Add elastic agent standalone manifests #24231

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Agent example input autodiscover config for standalone k8s#24157

Add Agent example input autodiscover config for standalone k8s#24157
ChrsMark merged 1 commit intoelastic:masterfrom
ChrsMark:add_k8s_agent_autodisocver_part

ChrsMark commented Feb 22, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Feb 22, 2021

Uh oh!

elasticmachine commented Feb 22, 2021

Uh oh!

elasticmachine commented Feb 22, 2021

Build stats

Trends 🧪

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ChrsMark commented Feb 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

How to test this PR locally

Screenshots

Uh oh!

elasticmachine commented Feb 22, 2021

Uh oh!

elasticmachine commented Feb 22, 2021

Uh oh!

elasticmachine commented Feb 22, 2021

💚 Build Succeeded

Build stats

Trends 🧪

❕ Flaky test report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ChrsMark commented Feb 22, 2021 •

edited

Loading