Skip to content

How to use ServiceMonitor spec.attachMetadata.node #7166

New issue
New issue
Open
Open
How to use ServiceMonitor spec.attachMetadata.node#7166
Labels
good first issuehelp wantedkind/documentationkind/support
@daviderli614

Description

@daviderli614
daviderli614
opened on Dec 4, 2024

What happened?

Description

I use kube-prometheus-stack helm-chart to deploy prometheus components, and enable ServiceMonitor spec.attachMetadata.node does not work.

I want to add aws eks node labels to metrics, for example: beta.kubernetes.io/instance-type: t3.large, beta.kubernetes.io/arch: amd64, but not found in node-exporter metrics:

image

Resource Version:

  • kube-prometheus-stack helm chart version: 54.0.1
  • prometheus image: quay.io/prometheus/prometheus:v2.47.2
  • operator image: quay.io/prometheus-operator/prometheus-operator:v0.69.1
  • node-exporter image: quay.io/prometheus/node-exporter:v1.6.1

node-exporter ServiceMonitor yaml:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  creationTimestamp: "2023-02-03T06:32:34Z"
  generation: 7
  labels:
    app.kubernetes.io/component: metrics
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: prometheus-node-exporter
    app.kubernetes.io/part-of: prometheus-node-exporter
    app.kubernetes.io/version: 1.6.1
    argocd.argoproj.io/instance: kube-prometheus-stack-eks-ap-southeast-1-qa1
    helm.sh/chart: prometheus-node-exporter-4.23.2
    jobLabel: node-exporter
    release: prometheus
  name: prometheus-prometheus-node-exporter
  namespace: monitoring
  resourceVersion: "435375472"
  uid: 5db2ab95-eae0-4905-9892-666ff6e27bd2
spec:
  attachMetadata:
    node: true
  endpoints:
  - port: http-metrics
    scheme: http
  jobLabel: jobLabel
  selector:
    matchLabels:
      app.kubernetes.io/instance: prometheus
      app.kubernetes.io/name: prometheus-node-exporter

Prometheus configuration node-exporter job:

- job_name: serviceMonitor/monitoring/prometheus-prometheus-node-exporter/0
  honor_timestamps: true
  scrape_interval: 10s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
    separator: ;
    regex: (prometheus-node-exporter);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - monitoring
    attach_metadata:
      node: true

Steps to Reproduce

Expected Result

Actual Result

Prometheus Operator Version

kubectl get deploy prometheus-kube-prometheus-operator -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "7"
  creationTimestamp: "2023-02-03T06:32:32Z"
  generation: 7
  labels:
    app: kube-prometheus-stack-operator
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 54.0.1
    argocd.argoproj.io/instance: kube-prometheus-stack-eks-ap-southeast-1-qa1
    chart: kube-prometheus-stack-54.0.1
    heritage: Helm
    release: prometheus
  name: prometheus-kube-prometheus-operator
  namespace: monitoring
  resourceVersion: "419030415"
  uid: 899d9088-1e5b-4709-bf9b-861105391d11
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: kube-prometheus-stack-operator
      release: prometheus
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kube-prometheus-stack-operator
        app.kubernetes.io/instance: prometheus
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/part-of: kube-prometheus-stack
        app.kubernetes.io/version: 54.0.1
        chart: kube-prometheus-stack-54.0.1
        heritage: Helm
        release: prometheus
    spec:
      containers:
      - args:
        - --kubelet-service=kube-system/prometheus-kube-prometheus-kubelet
        - --localhost=127.0.0.1
        - --prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.69.1
        - --config-reloader-cpu-request=50m
        - --config-reloader-cpu-limit=200m
        - --config-reloader-memory-request=64Mi
        - --config-reloader-memory-limit=328Mi
        - --thanos-default-base-image=quay.io/thanos/thanos:v0.32.5
        - --secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1
        - --web.enable-tls=true
        - --web.cert-file=/cert/cert
        - --web.key-file=/cert/key
        - --web.listen-address=:10250
        - --web.tls-min-version=VersionTLS13
        image: quay.io/prometheus-operator/prometheus-operator:v0.69.1
        imagePullPolicy: IfNotPresent
        name: kube-prometheus-stack
        ports:
        - containerPort: 10250
          name: https
          protocol: TCP
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 50m
            memory: 64Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /cert
          name: tls-secret
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
        seccompProfile:
          type: RuntimeDefault
      serviceAccount: prometheus-kube-prometheus-operator
      serviceAccountName: prometheus-kube-prometheus-operator
      terminationGracePeriodSeconds: 30
      volumes:
      - name: tls-secret
        secret:
          defaultMode: 420
          secretName: prometheus-kube-prometheus-admission


### Kubernetes Version

```yaml
kubectl version
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.6-eks-7f9249a
WARNING: version difference between client (1.28) and server (1.30) exceeds the supported minor version skew of +/-1



### Kubernetes Cluster Type

EKS

### How did you deploy Prometheus-Operator?

helm chart:prometheus-community/kube-prometheus-stack

### Manifests

_No response_

### prometheus-operator log output

```shell
ts=2024-12-04T03:42:13.530967167Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.12.42:55268: EOF"
ts=2024-12-04T03:42:14.279705Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.12.42:55324: EOF"
ts=2024-12-04T03:42:14.334552417Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.12.42:55354: EOF"
ts=2024-12-04T03:46:21.24822524Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:32832: EOF"
ts=2024-12-04T03:46:21.248641111Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:32866: EOF"
ts=2024-12-04T03:46:21.249609752Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:32838: EOF"
ts=2024-12-04T03:46:21.253523639Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:32858: EOF"
ts=2024-12-04T03:58:34.734155635Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:45552: EOF"
level=info ts=2024-12-04T03:58:35.272572437Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T03:58:36.266585808Z caller=operator.go:655 component=alertmanageroperator key=monitoring/prometheus-kube-prometheus-alertmanager msg="sync alertmanager"
level=info ts=2024-12-04T03:58:36.446135031Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
ts=2024-12-04T04:18:38.527285833Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:50602: EOF"
ts=2024-12-04T04:18:38.546209914Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:50620: EOF"
ts=2024-12-04T04:18:38.636531833Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:50626: EOF"
ts=2024-12-04T04:18:38.64139357Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:50644: EOF"
ts=2024-12-04T04:18:39.436136995Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:54018: EOF"
ts=2024-12-04T04:18:39.542568854Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:54002: read tcp 10.0.56.189:10250->10.0.10.139:54002: read: connection reset by peer"
level=info ts=2024-12-04T04:18:39.787407304Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T04:18:40.831730707Z caller=operator.go:655 component=alertmanageroperator key=monitoring/prometheus-kube-prometheus-alertmanager msg="sync alertmanager"
level=info ts=2024-12-04T04:18:40.938452446Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
ts=2024-12-04T04:29:49.24130868Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.12.42:48532: EOF"
level=info ts=2024-12-04T04:29:50.095652621Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T04:29:51.200969252Z caller=operator.go:655 component=alertmanageroperator key=monitoring/prometheus-kube-prometheus-alertmanager msg="sync alertmanager"
level=info ts=2024-12-04T04:29:51.447499559Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T04:29:52.558482907Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
ts=2024-12-04T04:37:17.9324788Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.12.42:37076: read tcp 10.0.56.189:10250->10.0.12.42:37076: read: connection reset by peer"
level=info ts=2024-12-04T04:37:18.681591502Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T04:37:19.75134116Z caller=operator.go:655 component=alertmanageroperator key=monitoring/prometheus-kube-prometheus-alertmanager msg="sync alertmanager"
level=info ts=2024-12-04T04:37:19.888949615Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T04:37:21.194550454Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T07:48:21.087802968Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T07:48:22.110485719Z caller=operator.go:655 component=alertmanageroperator key=monitoring/prometheus-kube-prometheus-alertmanager msg="sync alertmanager"
level=info ts=2024-12-04T07:48:22.231379414Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T08:16:11.411392261Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T08:16:12.223870235Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T08:23:42.487682853Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T08:23:43.487647905Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
ts=2024-12-04T08:23:44.625595817Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:59522: read tcp 10.0.56.189:10250->10.0.10.139:59522: read: connection reset by peer"
ts=2024-12-04T08:23:44.628627458Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:59526: read tcp 10.0.56.189:10250->10.0.10.139:59526: read: connection reset by peer"
level=info ts=2024-12-04T08:26:16.083415841Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T08:26:17.031710795Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
ts=2024-12-04T08:26:17.144785877Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.10.139:50684: EOF"
level=info ts=2024-12-04T09:12:27.724285128Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
ts=2024-12-04T09:12:28.871126517Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.12.42:33488: EOF"
level=info ts=2024-12-04T09:12:28.957983974Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
ts=2024-12-04T09:12:29.94793252Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.12.42:33548: EOF"
level=info ts=2024-12-04T09:27:55.055051901Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"
level=info ts=2024-12-04T09:27:55.993316368Z caller=operator.go:1181 component=prometheusoperator key=monitoring/prometheus-kube-prometheus-prometheus msg="sync prometheus"



### Anything else?

_No response_

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issuehelp wantedkind/documentationkind/support

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions