FYI : @gizas

Use Case : As a Kubernetes Platform Owner / Operator I want to be able to define labels at the namespace level (such as org, environment, product line etc) so that I can route / filter / apply workflow to the kube-state-metrics and container metrics based on these defined namespace labels.

The Following kube-state-metrics should support namespace labels.

The request is for 7.17.x backport and the 8.x release for metricbeat

Currently Supported

• state_pod
• state_container
• state_service

Not Currently Supported - But Required

• state_deployment
• state_replicaset

EDIT removed - state_node per @MichaelKatsoulis explanation

Container metrics are / are not supported depending on what version is used see the matrix below.

Matrix of what the test indicate that are currently supported.

Steps to reproduce

I used GKE and the Google Online Boutique Sample

1. Deploy a microservices app such as the online boutique

2. With custom namespace labels example

$ cat product-catalog.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: product-catalog
  labels:
    ecosystem: prod
    org: product 

3. Update kube-state-metrics manifiest to include custom labels and deploy
   Note : Tested with kube-state-metrics 2.4.2 and 2.6.0

https://github.com/kubernetes/kube-state-metrics/blob/v2.4.2/examples/standard/deployment.yaml#L24

kube-state-metrics/examples/standard/deployment.yml 
    spec:
      automountServiceAccountToken: true
      containers:
      - image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.4.2
        args: ['--metric-labels-allowlist=namespaces=[org,ecosystem]']

4. Deploy metricbeat with standard kube-state-metrics configuration in 7.X 8.X

5. Go To Discover and Observer the kube-state-metrics that the custom namespace labels and those that do not.

Not Currently Supported - But Required

• state_deployment
• state_replicaset

7. Note it we exec into a metricbeat container and directly curl the kube-state-metrics endpoint we see the labels are there. So it is unclear why they are not showing up on the state_deployment and state_replicaset metricsets

root@gke-stephenb-gke-istio-t-default-pool-31633e50-i1y9:/usr/share/metricbeat# curl http://kube-state-metrics:8080/metrics | grep label_ecosystem
kube_namespace_labels{namespace="payment",label_ecosystem="prod",label_org="finance"} 1
kube_namespace_labels{namespace="product-catalog",label_ecosystem="prod",label_org="product"} 1
root@gke-stephenb-gke-istio-t-default-pool-31633e50-i1y9:/usr/share/metricbeat# 

It does Appear that in 8.X the custom matcher code for container metrics is not needed, it is unclear when look at the matrix above whether it is needed or not in 7.17

For reference here is the custom matcher code we had to use in 7.15.x to get the container metrics to have namespace labels

    processors:
      - add_cloud_metadata:
      # Custom Matcher Code
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          default_indexers.enabled: false
          default_matchers.enabled: false
          annotations.dedot: false
          indexers:
            - pod_uid:
          matchers:
            - fields:
                lookup_fields: ['kubernetes.pod.uid']
          #Why not just this enough???
          add_resource_metadata:
            namespace:
              enabled: true
              include_labels: ["org","ecosystem","env","region","costCenter"]