We want to improve our coverage with integration tests for Kubernetes. We should explore our options and come up with a solution that allow us to run Filebeat & Metricbeat against a running cluster.

In order to simplify this we can start by targeting testing a single version of Kubernetes (latest) in Jenkins jobs. We can reuse this in the future for compatibility matrix based testing.

There is a bunch of pieces we can test for, I think these would be the most important:

• Metricbeat Kubernetes module

  • retrieving kubelet performance related metrics
  • apiserver, proxy, controllermanager and others
  • kube-state-metrics? (I would make this optional for the first iteration)

• Metricbeat autodiscover

  • Check that a module is launched when a pod is created
  • Check the module stops when the pod is deleted

• Filebeat autodiscover

  • Check that a container input launched when a pod is created
  • Check the input stops when the pod is deleted (after 60s of cleanup_time)
  • Test hints based autodiscover (ie a JSON output)

There are several paths we can follow to do this. For instance:

• leveraging GKE and run tests there, but this would mean we need to build & upload a docker image for each beat.
• Use kind to deploy a local cluster, run MB/FB from "outside", just the local binary. There are some previous experiences using it here: Support for verification of the Helm charts using BDD approach e2e-testing#100