What does this PR do?

This PR adds k8s manifest for running Elastic Agent in standalone mode with the k8s integration enabled by default. This one deploys Agent as Daemonset Pods on all k8s nodes and as Deployment Pod on the cluster. Deamonset Pods are responsible for collecting metrics from node's kubelet API, kubeproxy metrics and try to autodiscover k8s Scheduler Pod and k8s Controller Manager Pod (which are deployed on master node(s)) and start collecting from them dynamically using the respective metricsets. Deployment pod is responsible for collecting cluster wide metrics from kube_state_metrics service running on the cluster.

@blakerouse @masci @ph @ruflin I would love your feedback here.

Disclaimer: The manifest works if we disable the dynamic inputs part. Find full information about the issues in the bottom of this description: #23685

How to test this PR locally

1. Run a kind cluster locally using the following:

# three node (two workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

kind create cluster --config kind-mutly.yaml
2. Uncomment the scheduler and controllermanager config section and deploy Agent: kubectl apply -f elastic-agent-standalone-kubernetes.yml
3. Verify that all data streams ship data:

• kubernetes.volume
• kubernetes.container
• kubernetes.pod
• kubernetes.proxy
• kubernetes.node
• kubernetes.system
• kubernetes.controllermanager
• kubernetes.scheduler
• kubernetes.state_container
• kubernetes.state_pod 
• kubernetes.state_replicaset
• kubernetes.state_node
• kubernetes.state_deployment 
• kubernetes.state_service  
• kubernetes.state_storageclass
• kubernetes.apiserver
• kubernetes.event
• kubernetes.state_persistentvolumeclaim
• kubernetes.state_statefulset 
• kubernetes.state_cronjob 
• kubernetes.state_persistentvolume 
• kubernetes.state_resourcequota

4. Install Kubernetes integration from Fleet UI and verify that Dashboards work properly as well as Metrics UI.

Related issues

• Closes Add kubernetes manifests for standalone Agent with kubernetes integration  #23618
• Related to Migrate Kubernetes integration integrations#511

Open Issues

1. Dynamic inputs / Former Autodiscover
   Dynamic inputs setup to automatically discover scheduler and controllermanager Pods does not completely work right now and we get the following error:

2021-01-25T15:36:29.224Z	DEBUG	application/periodic.go:40	Failed to read configuration, error: could not emit configuration: could not create the AST from the configuration: missing field accessing 'inputs' (source:'/etc/agent.yml')

Converting ${NODE_NAME} placeholders to ${env.NODE_NAME} does not fix the problem and even if we remove all other datastream configs and leave only the dynamic one it still gives the error:

- id: >-
    kubernetes/metrics-kubernetes.controllermanager-3d50c483-2327-40e7-b3e5-d877d4763fe1
  data_stream:
    dataset: kubernetes.controllermanager
    type: metrics
  metricsets:
    - controllermanager
  hosts:
    - '${kubernetes.pod.ip}:10252'
  period: 10s
  condition: ${kubernetes.pod.labels.component} == 'kube-controller-manager'

In addition, if we remove the dynamic inputs part and have ${env.NODE_NAME} we still get the same error.

In this, there might be a bug in Agent which does not allow us to combine these 2 configuration approaches.

2. Package setup requires manual interaction with the UI
   After deploying the manifests the package is not automaticaly installed and requires the user to manually install it from Fleet UI. This is already known but I'm putting it here for reference.