Pinging @elastic/integrations (Team:Integrations)

Pinging @elastic/agent (Team:Agent)

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

❕ Flaky test report

No test was executed to be analysed.

This pull request doesn't have a Team:<team> label.

@ruflin @michalpristas @blakerouse Do we need to add ids for inputs or stream?

Adding the section to collect logs from Pod's using k8s provider works too. We can add it here or we can add it in separate follow-up PR.

Sample event:

{
  "_index": ".ds-logs-generic-default-2021.02.17-000001",
  "_type": "_doc",
  "_id": "BayDsHcB281QBbW7tkDy",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2021-02-17T15:02:37.006Z",
    "elastic_agent": {
      "snapshot": true,
      "version": "7.12.0",
      "id": "0edb7f45-68dc-48c0-af31-72b512c1a075"
    },
    "ecs": {
      "version": "1.6.0"
    },
    "log": {
      "offset": 111721,
      "file": {
        "path": "/var/log/containers/elastic-agent-wnp9g_kube-system_elastic-agent-62f55fefad8c999c37f82a03894cd604cd75aa8df1f7ec0e0926386b467f8b55.log"
      }
    },
    "message": "{\"log\":\"2021-02-17T15:02:36.050Z\\u0009INFO\\u0009operation/operator.go:245\\u0009operation 'operation-start' skipped for filebeat.7.12.0-SNAPSHOT\\n\",\"stream\":\"stderr\",\"time\":\"2021-02-17T15:02:36.050439402Z\"}",
    "input": {
      "type": "log"
    },
    "kubernetes": {
      "namespace": "kube-system",
      "pod": {
        "name": "elastic-agent-wnp9g",
        "uid": "82ba7656-6bfd-47ab-aff4-b607f1919d68",
        "ip": "10.128.0.17",
        "labels": {
          "controller-revision-hash": "665cbc7d9c",
          "pod-template-generation": "1",
          "app": "elastic-agent"
        }
      },
      "container": {
        "image": "docker.elastic.co/beats/elastic-agent:7.12.0-SNAPSHOT",
        "name": "elastic-agent",
        "runtime": "docker",
        "id": "62f55fefad8c999c37f82a03894cd604cd75aa8df1f7ec0e0926386b467f8b55"
      }
    },
    "data_stream": {
      "type": "logs",
      "dataset": "generic",
      "namespace": "default"
    },
    "event": {
      "dataset": "generic"
    },
    "host": {
      "mac": [
        "42:01:0a:80:00:11",
        "02:42:37:90:a6:0b",
        "b6:29:89:3b:b4:a4",
        "36:df:52:c9:18:89",
        "5e:16:1b:92:74:6a"
      ],
      "hostname": "gke-chrismark-test-agent-default-pool-c44aff9a-cxrg",
      "architecture": "x86_64",
      "name": "gke-chrismark-test-agent-default-pool-c44aff9a-cxrg",
      "os": {
        "codename": "Core",
        "platform": "centos",
        "version": "7 (Core)",
        "family": "redhat",
        "name": "CentOS Linux",
        "kernel": "4.19.150+"
      },
      "id": "5fdb5cc51a59a640a350f8f0b0c762f5",
      "containerized": true,
      "ip": [
        "10.128.0.17",
        "fe80::4001:aff:fe80:11",
        "169.254.123.1",
        "10.40.0.1",
        "fe80::b429:89ff:fe3b:b4a4",
        "fe80::34df:52ff:fec9:1889",
        "fe80::5c16:1bff:fe92:746a"
      ]
    },
    "cloud": {
      "account": {
        "id": "elastic-observability"
      },
      "provider": "gcp",
      "instance": {
        "id": "879949892028547785",
        "name": "gke-chrismark-test-agent-default-pool-c44aff9a-cxrg"
      },
      "machine": {
        "type": "e2-medium"
      },
      "availability_zone": "us-central1-c",
      "project": {
        "id": "elastic-observability"
      }
    },
    "agent": {
      "name": "gke-chrismark-test-agent-default-pool-c44aff9a-cxrg",
      "type": "filebeat",
      "version": "7.12.0",
      "hostname": "gke-chrismark-test-agent-default-pool-c44aff9a-cxrg",
      "ephemeral_id": "33e6ffd1-7152-42e5-9dc2-55d52b50e7b3",
      "id": "61b439d7-e5f2-497f-a194-87969434799b"
    }
  },
  "fields": {
    "@timestamp": [
      "2021-02-17T15:02:37.006Z"
    ]
  },
  "sort": [
    1613574157006
  ]
}

So at this state the manifest file supports the following:

1. Runs kubernetes integration (from previous PR)
2. Runs system integration (the same metricsets that Metricbeat was running in the past with

   beats/deploy/kubernetes/metricbeat-kubernetes.yaml

   Line 78 in bb78931

   metricsets:

   )
3. Collects Pod's logs using k8s provider and dynamic inputs

@blakerouse @jsoriano @david-kow feel free to review when you have the time

Looks good, but I wonder if we want any debug logging enabled by default.

Ok, makes sense, I will remove it for now.

Phenomenal progress in getting standalone agent mode working to get logs and metrics for system and K8s control plane. Thanks @ChrsMark for getting us here so quickly in last few weeks, Kudos!

Like you said in the description, we will need a single standalone manifest that collects metrics and logs for both system and k8s but looking at this standalone manifest and that doesn't have the system integration yet. Is that something we are planning to get next week?

Once we have that single standalone manifest, I am assuming we can just fit this standalone manifest right in the "add agent in standalone mode" in Fleet UI with K8s integration config in the standalone manifest filled with kubernetes section from agent policy.

So users can just copy this standalone manifest from Fleet UI, add ES creds, run it on K8s and bingo, they have the system and K8s observability. Is that right?

This will also work with ECK agent CRD I assume where users can just insert the standalone manifest in appropriate section in agent config @shubhaat fyi

Like you said in the description, we will need a single standalone manifest that collects metrics and logs for both system and k8s but looking at this standalone manifest and that doesn't have the system integration yet. Is that something we are planning to get next week?

Hey! This PR adds system integration and logs' collection from Pods. You can see the full manifest at https://github.com/elastic/beats/pull/23938/files.

Once we have that single standalone manifest, I am assuming we can just fit this standalone manifest right in the "add agent in standalone mode" in Fleet UI with K8s integration config in the standalone manifest filled with kubernetes section from agent policy.

So users can just copy this standalone manifest from Fleet UI, add ES creds, run it on K8s and bingo, they have the system and K8s observability. Is that right?

In the past we had been sharing this kind of manifests(https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml) through GH and guiding our users through https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html. I expect we will have sth similar here, not sure if if this should be added in Fleet too :).

@ChrsMark I may be missing something here but when I look at standalone config in master I see only one logfile input which collects the logs from var/log/containers but what about auth and syslog datasets? Here is what I see on the /var/log directory on single node K8s cluster. And I think we should be providing the auth.log and syslog similar to how our system integration logfile input collects.

/var/log$ ls alternatives.log apt auth.log btmp containers daemon.log debug dpkg.log faillog kern.log lastlog messages ntpstats pods syslog syslog.1 syslog.2.gz

@mukeshelastic system logs will be collected too after #24185.