Problem

When a pod is assigned an IP address that was previously used by a pod that is in a failed or completed state (either because a pod failed or a cronjob ran to completion), viz will attempt to scrape both pods.

Since the IP address is the same on both pods, this leads to duplicate metrics with different kubernetes metadata.

Solution

The proposed solution is to add a relabel to the scraping config that only keeps metrics where the pod phase is either Running or Pending states.

Validation

• Run a large number of jobs to completion in a cluster, aiming to use up all address space.
• Run a pod exposing a port and ensure it gets assigned an IP address that is also used by one of the completed jobs
• Create a Server resource selecting the pod/port and ensure the default policy is set to deny
• Generate traffic from a non-meshed pod to the meshed pod hosting the service and observe the inbound_http_authz_deny_total metric in prometheus

Previous to the fix, you will see multiple metrics entries for the same IP with different pod metadata. After the fix is applied, only the running pod will be referenced.

Fixes #10562