What did you do?

Trying to use Prometheus to monitor a Kubernetes cluster externally, e.g. without running the Prometheus inside the Kubernetes cluster itself. For this I use the configuration below. The main magic of the configuration happens with the ca_file, cert_file and key_file, as in our environment we provide access to Kubernetes with short lived Hashicorp Vault issued certificates.

This means that the Prometheus instance has a Vault identity and automatically updates certificates in the locations of tls_config every 30 minutes.
While the certificates are properly updated, Prometheus refuses to update it's running configuration with these certificates without a full Prometheus restart. Restarting Prometheus every 30 minutes would mean several minutes of downtime with every restart.

What did you expect to see?

Either Prometheus automatically tracks the files for changes or reloads the certificates on a HUP signal.

What did you see instead? Under which circumstances?

Certificates are not updated. When the certificates TTL arrives, Prometheus loses access to Kubernetes, even though valid certificates exist in the defined places, which can be used to access the Kubernetes endpoint. HUP'ing the Prometheus process does not help, only a full restart of the Prometheus service.

Environment

• System information:

Linux 3.10.0-1127.8.2.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 2.30.3 (branch: HEAD, revision: f29caccc42557f6a8ec30ea9b3c8c089391bd5df)
build user:       root@5cff4265f0e3
build date:       20211005-16:10:52
go version:       go1.17.1
platform:         linux/amd64

• Prometheus configuration file:

  - job_name: 'kubernetes-cadvisor'
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: https
    kubernetes_sd_configs:
      - api_server: https://endpoint:6443
        role: node
        tls_config:
          ca_file: /etc/prometheus/secrets/kubernetes/ca.pem
          cert_file: /etc/prometheus/secrets/kubernetes/cert.pem
          key_file: /etc/prometheus/secrets/kubernetes/key.pem
          insecure_skip_verify: false

    tls_config:
      ca_file: /etc/prometheus/secrets/kubernetes/ca.pem
      cert_file: /etc/prometheus/secrets/kubernetes/cert.pem
      key_file: /etc/prometheus/secrets/kubernetes/key.pem
      insecure_skip_verify: false

    relabel_configs:
      - separator: ;
        regex: __meta_kubernetes_node_label_(.+)
        replacement: $1
        action: labelmap

      - separator: ;
        regex: (.*)
        target_label: __address__
        replacement: endpoint:6443
        action: replace

      - source_labels: [__meta_kubernetes_node_name]
        separator: ;
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
        action: replace