Requirements

We expect Elastic Agent inside of Kubernetes in standalone mode (no connection to fleet-server, not using fleet-server to update agent config) to collect

• metrics and logs for kubernetes components - master node and the processes it runs, worker node and the processes it runs
• system metrics and logs of master and worker nodes
• pod metrics and logs to ES via auto-discovery through dynamic inputs and hints based discovery i.e. nginx running on K8s pods can send their metrics and logs to ES auto-magically as long as the agent policy contains appropriate conditional logic

In addition we will provide

• K8s integration that operator needs to install to observe K8s master and worker nodes and the infrastructure running K8s cluster, when installed it will display the K8s dashboards we provide today with metricbeat
• Documentation guide that walks through steps required to enable K8s and pod level observability with elastic agent in standalone mode

Note:
This issue explicitly ignores Fleet Mode, once standalone mode is working and documented then Fleet Mode will be the next target.

Core Issues

• Update K8s resource definition to use a host mount point for storing the running filebeat registry.
  • At the moment this is nested under the Elastic Agent data directory that changes on every build of Elastic Agent /usr/share/data/elastic-agent-{{ commit_short }}/run. Because of the {{commit_short}} being changed on every build this causes an issue with using a host mount point. [Elastic Agent] Allow separation of runtime state location and installation and executable location.  #24160
• Dynamic inputs fail to get validated [Agent] Dynamic inputs fail to get validated #23685
• Documentation guide that walks through steps required to enable K8s and pod level observability with elastic agent in standalone mode [Elastic Agent] Add documentation for the k8s provider #21848, PR: Add agent_on_k8s docs and k8s provider observability-docs#401
• Add Metricbeat --system.hostfs equivalent to Elastic Agent Add Metricbeat --system.hostfs equivalent to Elastic Agent #22915 PR: Add system.hostfs configuration option for system module #23831

Logs & Metrics

• System-level log collection - Logs from nodes of the Kubernetes cluster like /var/log/syslog', /var/log/secure`, etc. should be collected by Elastic Agent. PR: Add system level logs for Agent on k8s #24185
  • Update the K8s resource definition to include the correct host-level mount points to collect these logs.
  • Update the K8s resource definition to enable the system level log collection.
• Pod-level log collection - Logs from the running Pods in the cluster should be collected by Elastic Agent [Agent] Add agent standalone manifests for system module & Pod's log collection #23938
  • Update the K8s resource definition to include the correct host-level mount points to collect these logs.
  • Update the K8s resource definition to enable the log collection using inputs with variables.
• System-level metrics collection - Metrics (cpu/memory/etc.) from the nodes of the Kubernetes cluster [Agent] Add agent standalone manifests for system module & Pod's log collection #23938
  • Update the K8s resource definition to include the correct host-level mount points to collect these metrics from /proc and /sys.
    • Known Issue: Add Metricbeat --system.hostfs equivalent to Elastic Agent #22915
  • Update the K8s resource definition to enable the system metrics collection.
• Pod-level metrics collections - Metrics (cpu/memory/etc.) from the pods running in the Kubernetes cluster Add kubernetes manifests for standalone Agent with kubernetes integration  #23618
  • Update the K8s resource definition to collect metrics from kubelet.
• Kubernetes state metrics collection - Metrics (create/update/delete/etcs.) from the Kubernetes state Add kubernetes manifests for standalone Agent with kubernetes integration  #23618
  • Update the K8s resource definition to collect metrics from kube-state-metrics.
• K8s integrations package (template, dashboard) Migrate Kubernetes integration integrations#511

Need to be investigated

• hints support in K8s provider for agent Support hints based auto-discovery in kubernetes provider for elastic agent #23876