Cherry-pick of PR #24742 to 7.x branch. Original message:

What does this PR do?

Refactor logic in kubernetes autodiscover that decides when to generate events to try to address #22718.

Kubernetes autodiscover can generate events without network information now (without host or port/ports). This allows to generate events for pods that haven't started yet, or have succeeded/failed before generating a running event. These events still include the container id, so they can be used to collect logs. Still, no start event is generated if no pod ip and no container ids are available.

Some helpers have been added to obtain relevant information from pods and their containers.

Some additional small refactors are done to improve readability.

Why is it important?

Current logic is checking similar things at the pod and container levels, try to simplify this logic focusing in containers only.

No matter what is the state of the pod, if there is a container running or trying to run, even if it is unhealthy, some configuration should be generated, so logs can be collected.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. But more tests would be needed.
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Fix update so it definitely stops configurations, respecting the cleanup timeout.
• Double-check that nothing breaks with events without data.host.
  This is logged at the debug level:

  2021-04-08T12:53:34.453+0200	DEBUG	[autodiscover]	template/config.go:156	Configuration template cannot be resolved: field 'data.host' not available in event or environment accessing 'hosts' (source:'/home/jaime/tmp/metricbeat-autodiscover-k8s.yml')
  

• Actually check that the refactor solves the issues and doesn't break anything else.
• Try to add more testing. A couple of cases added to current tests.
• Check that nothing breaks with hints including data.host or data.ports.
  • data.ports.* doesn't seem to be working, check if this is a regression. Named ports were not added to container events. They are now.
  • Multiple configurations generated for pods with multiple containers, expected? compare with master. False alarm.
• Check autodiscover with heartbeat.

Author's notes for the future

• When using autodiscover templates, if the condition matches pod conditions, container events will match too because they also include pod metadata. In pods with multiple containers this may lead to events with metadata of the incorrect container. This could be possibly avoided by matching per pod metadata only in the pod events. This is also happening in master, not solving in this PR.
• data.ports is not included in container-level events, so it doesn't work when conditions for specific containers are used.

How to test this PR locally

• Check that everything keeps running with some autodiscover happy cases.
• Check that logs are collected from a pod in CrashLoopBackOff (kubectl run --image=redis redis -- echo foo).

Related issues

• Closes Kubernetes autodiscover doesn't discover short living jobs (and pods?) #22718.

Use cases

Collect logs from containers in short-living or failing pods.