Background

This issue was first detected when trying to stop node and namespace watchers from starting (issue is this one). At first, it seemed that we only had these watchers based on features add_resource_metadata and hints, but having a better look, we were actually starting watchers at many places.

Issue

All state_* metricsets are using this function

This function adds a metadata enricher to each metricset by calling NewResourceMetadataEnricher function:

Also pod and node metricsets call NewResourceMetadataEnricher to enrich the events with metadata.

This enricher will create 3 watchers:

One for the resource of that metricset, one for node and one for namespace.

Since we can have multiple metricsets enabled, then we will also have multiple node and namespace watchers running. One for each metricset.

To solve this, we need to find a way to share these watchers between the metricsets, similar to what we did to fetch metrics from KSM (PR) and Kubelet.

Additionally, from the way we create the resource specific watcher

More on this, we seem to always start the three watchers for all the resources that the NewResourceMetadataEnricher is called by. These watchers are created for metadata enrichment and for some resources the existence of one or some of these watchers is not relevant. For example node watcher is not needed in case we want to enrich deployments or statefulsets.
We need to start only the relevant watchers for each resource.

Possibly this can be solved if we share the watchers between metricsets.

The better handling of watchers initialization will lead to less Kubernetes API calls and possible issues related to that in large scale clusters.

Issues

Relates to elastic/elastic-agent#3801.

Current watchers

These are the current watchers when starting metricbeat with default configurations.

- type: kubernetes
  scope: cluster
  node: ${NODE_NAME}
  unique: true
  templates:
    - config:
        - module: kubernetes
          hosts: ["kube-state-metrics:8080"]
          period: 10s
          metricsets:
            - state_node
            - state_deployment
            - state_daemonset
            - state_replicaset
            - state_pod
            - state_container
            - state_cronjob
            - state_job
            - state_resourcequota
            - state_statefulset
            - state_service
            - state_persistentvolume
            - state_persistentvolumeclaim
            - state_storageclass
            - state_namespace

For the state_* metricsets grouped as described above:

Note: It is 14 and not 13, because the resource watcher for namespace adds 1 to the namespace watcher, and same happens for node watcher.

Note 2: state pod can also create 2 more watchers if we add resource_metadata for job and deployments (disabled by default).

For all the other metricsets:

Note: Container and state container can also create 2 more watchers if we add resource_metadata for job and deployments (disabled by default).

Expected watchers

Watchers needed for each metricset by default (without counting add_resource_metadata.deployment/cronjob):

Checks

• We don't start watchers based on the resource at the same time we start watchers for node/namespace, without adding a condition to avoid duplicates.
• We don't have multiple watchers for the same resource. Instead we need to find a way to have them shared.
• We only start watchers that are needed based on either the resource or the configuration.

PRs

• [Metricbeat][Kubernetes] Share watchers between metricsets #37332
• Add enricher to state resource quota metricset: [Metricbeat][Kubernetes] Enrich metadata from state resource quota #38004
• High level documentation of enrichment process Add a markdown with high level kubernetes metadata enrichment explanation #38757

### Tasks