Cherry-pick of PR #26056 to 7.x branch. Original message:

What does this PR do?

This PR add cluster identifier fields (defined in ECS) as part of k8s metadata in:

• event's enrichment with autodiscovery
• event's enrichment in kubernetes module (where already happens)
• event's enrichment in add_kubernetes_metadata processor

Note: [MetaGenerators' refactoring ] The identifiers are stored under orchestrator.cluster.url/name and because of this the metadata generators are refactored a little bit so as to cover the addition of such fields that are out of kubernetes.* namespace. The change is transparent and kubernetes.* metadata are still reported in the same way. The refactoring is about making it easier to handle in the future ECS fields populated by k8s metadata generators. The logic is covered in interfaces' docs.

The transparency of the refactoring is ensured by Event's testing in tests below:

1. https://github.com/elastic/beats/pull/26056/files#diff-40f67a7afe61b20e5336ebd9ad20c40c717794f0821ba9234301a7e4544315efR114
2. https://github.com/elastic/beats/pull/26056/files#diff-9f24ba274864b933a726842c2b9fb8ee487a13545579938f9e373d0463611142R335
3. https://github.com/elastic/beats/pull/26056/files#diff-4abca4d17de2c5a0ed69171309dc21636be3ca778d0d34943e7747f2a7c6dbc1R235

The fields are populated following the flow bellow:

1. Try to get this info from kube_config if provided
2. Else (when inCluster mode) try to get the info from kubeadm-config configMap (if available). Only for clusters setup with kubeadm.
3. Else try to get the info from cloud’s meta api (only on GKE)
4. Else these fields will not be populated (see [Metricbeat] Add cluster identifier to Kubernetes metadata #17467 (comment))

Why is it important?

To add cluster identifier ECS fields as part of k8s metadata.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• update docs in metadata generators
• update cluster roles
• manual testing

How to test this PR locally

A. Verify that events from state_* metricsets are enriched properly

Enable kubernetes module with the following datasets:

- module: kubernetes
  metricsets:
    - state_node
    - state_deployment
    - state_pod
    - state_container
    - state_service
  period: 10s
  hosts: ["0.0.0.0:8081"]
  add_metadata: true
  kube_config: /Users/chrismark/.kube/config

Note: In the example above I run kube-state-metrics on local cluster using kind and I expose it to my host machine using kubectl -n kube-system port-forward svc/kube-state-metrics 8081:8080. In this case I need to define add_metadata as true and also provide the proper kube_config so as to reach the k8s API. You can try kubectl config view -o jsonpath='{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}' to verify the values.
2. Ensure that orchestrator.cluster.name, orchestrator.cluster.name, kubernetes.namespace and kubernetes.node.name are being populated properly.
3. Perform same test while running with inCluster mode, running metricbeat as Pod in the cluster (Note that the k8s cluster should be create with kubeadm since values for cluster info are retrieved from kubeadm-config configmap, you can try kubectl -n kube-system get configmap kubeadm-config -o yaml to verify it)

B. Verify that events from add_kuberentes_metadata are enriched properly

0. Use updated manifests from https://github.com/elastic/beats/tree/master/deploy/kubernetes
1. Deploy Filebeat on kubernetes (cluster should be created with kubadm [ie a kind cluster] ) and configure log collection like this:

    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

2. Ensure that orchestrator.cluster.name, orchestrator.cluster.name, kubernetes.namespace and kubernetes.node.name are being populated properly.

C. Verify that events from autodiscover provider are enriched properly

0. Use updated manifests from https://github.com/elastic/beats/tree/master/deploy/kubernetes
1. Deploy Filebeat on kubernetes (cluster should be created with kubadm [ie a kind cluster] ) and configure log collection like this:

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

2. Ensure that orchestrator.cluster.name, orchestrator.cluster.name, kubernetes.namespace and kubernetes.node.name are being populated properly.

D. Perform one of the above scenarios with Metricbeat running as Pod on GKE.

Related issues

• Closes [Metricbeat] Add cluster identifier to Kubernetes metadata #17467