Skip to content

Operator is not applying (CA) certificates for HTTP configuration in Alertmanager's global configuration #8039

New issue
New issue
Closed
#8089
Closed
Operator is not applying (CA) certificates for HTTP configuration in Alertmanager's global configuration#8039
#8089
Assignees
dongjiang1989
Labels
kind/bug
@ctxch

Description

@ctxch
ctxch
opened on Oct 16, 2025

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

Description

Similar / same problem as #6760
The operator is not applying certificates referenced in alertmanager.spec.alertmanagerConfiguration.global.httpConfig.tlsConfig.ca.configMap

Steps to Reproduce

Create a ConfigMap / Secret containing CA certificate

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: monitoring
  name: proxy-ca-certificate
data:
  certificate: |
    -----BEGIN CERTIFICATE-----
...

Reference it in the Alertmanager resource

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  labels:
    app.kubernetes.io/component: monitoring
  name: main
spec:
  alertmanagerConfigSelector: {}
  alertmanagerConfiguration:
    global:
      httpConfig:
        followRedirects: true
        tlsConfig:
          ca:
            configMap:
              key: certificate
              name: proxy-ca-certificate
...

Expected Result

When configured alertmanager.spec.alertmanagerConfiguration.global.httpConfig.tlsConfig.ca.configMap Prometheus Operator mounts/applies certificates from the referenced ConfigMap (potentially from referenced Secret)

Actual Result

time=2025-10-16T07:24:52.453Z level=INFO source=main.go:191 msg="Starting Alertmanager" version="(version=0.28.1, branch=HEAD, revision=b2099eaa2c9ebc25edb26517cb9c732738e93910)"
time=2025-10-16T07:24:52.453Z level=INFO source=main.go:192 msg="Build context" build_context="(go=go1.23.7, platform=linux/amd64, user=root@fa3ca569dfe4, date=20250307-15:05:18, tags=netgo)"
time=2025-10-16T07:24:52.753Z level=INFO source=coordinator.go:112 msg="Loading configuration file" component=configuration file=/etc/alertmanager/config_out/alertmanager.env.yaml
time=2025-10-16T07:24:52.753Z level=INFO source=coordinator.go:125 msg="Completed loading of configuration file" component=configuration file=/etc/alertmanager/config_out/alertmanager.env.yaml
time=2025-10-16T07:24:52.754Z level=ERROR source=coordinator.go:131 msg="one or more config change subscribers failed to apply new config" component=configuration file=/etc/alertmanager/config_out/alertmanager.env.yaml err="unable to read CA cert: unable to read file /etc/alertmanager/certs/1_monitoring_proxy-ca-certificate_certificate: open /etc/alertmanager/certs/1_monitoring_proxy-ca-certificate_certificate: no such file or directory"

Content of the secret alertmanager-main-tls-assets-0 is empty

apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: "2025-10-10T09:20:13Z"
  labels:
    app.kubernetes.io/component: monitoring
    app.kubernetes.io/managed-by: prometheus-operator
    managed-by: prometheus-operator
  name: alertmanager-main-tls-assets-0
  ownerReferences:
  - apiVersion: monitoring.coreos.com/v1
    blockOwnerDeletion: true
    controller: true
    kind: Alertmanager
    name: main
    uid: c1da5bf6-a351-49b0-90e6-f57e61e36f12
  resourceVersion: "4450802"
  uid: e000ec93-eb08-459d-91c1-80af98295f2b
type: Opaque

Prometheus Operator Version

0.86.0

Kubernetes Version

clientVersion:
  buildDate: "2022-12-08T19:58:30Z"
  compiler: gc
  gitCommit: b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d
  gitTreeState: clean
  gitVersion: v1.26.0
  goVersion: go1.19.4
  major: "1"
  minor: "26"
  platform: linux/amd64
kustomizeVersion: v4.5.7
serverVersion:
  buildDate: "2024-09-25T08:59:15Z"
  compiler: gc
  gitCommit: cd4ec38d283c18e1b43e5df235febcafa39f181f
  gitTreeState: clean
  gitVersion: v1.31.1
  goVersion: go1.22.7 4493 X:boringcrypto
  major: "1"
  minor: "31"
  platform: linux/amd64

Kubernetes Cluster Type

Other (please comment)

How did you deploy Prometheus-Operator?

Other (please comment)

Manifests



prometheus-operator log output


ts=2025-10-16T07:23:28.860018338Z level=info caller=/workspace/cmd/operator/main.go:214 msg="Starting Prometheus Operator" version="(version=0.83.0, branch=, revision=5cf2f5d)" build_context="(go=go1.24.3, platform=linux/amd64, user=, date=20250530-07:45:21, tags=unknown)" feature_gates="PrometheusAgentDaemonSet=false,PrometheusShardRetentionPolicy=false,PrometheusTopologySharding=false,StatusForConfigurationResources=false"
ts=2025-10-16T07:23:28.86080452Z level=info caller=/workspace/internal/goruntime/cpu.go:27 msg="Updating GOMAXPROCS=1: using minimum allowed GOMAXPROCS"
ts=2025-10-16T07:23:28.86088358Z level=info caller=/workspace/cmd/operator/main.go:227 msg="namespaces filtering configuration " config="{allow_list=\"monitoring\",deny_list=\"\",prometheus_allow_list=\"monitoring\",alertmanager_allow_list=\"monitoring\",alertmanagerconfig_allow_list=\"monitoring\",thanosruler_allow_list=\"monitoring\"}"
ts=2025-10-16T07:23:30.906609373Z level=info caller=/workspace/cmd/operator/main.go:268 msg="connection established" kubernetes_version=1.31.1
ts=2025-10-16T07:23:30.910450137Z level=warn caller=/workspace/cmd/operator/main.go:91 msg="missing permission on resource \"storageclasses\" (group: \"storage.k8s.io/v1\")" reason="missing \"get\" permission on resource \"storageclasses\" (group: \"storage.k8s.io\") for all namespaces"
ts=2025-10-16T07:23:30.913740074Z level=warn caller=/workspace/cmd/operator/main.go:322 msg="missing permission to emit events" reason="missing \"create\" permission on resource \"events\" (group: \"\") for all namespaces"
ts=2025-10-16T07:23:30.9137632Z level=warn caller=/workspace/cmd/operator/main.go:322 msg="missing permission to emit events" reason="missing \"patch\" permission on resource \"events\" (group: \"\") for all namespaces"
ts=2025-10-16T07:23:30.920707538Z level=info caller=/workspace/cmd/operator/main.go:353 msg="Kubernetes API capabilities" endpointslices=true
ts=2025-10-16T07:23:30.969381642Z level=info caller=/workspace/pkg/server/server.go:293 msg="starting insecure server" address=[::]:8080
ts=2025-10-16T07:23:30.969508385Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheus
ts=2025-10-16T07:23:30.969559893Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheusagent
ts=2025-10-16T07:23:30.969591652Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=alertmanager
ts=2025-10-16T07:23:30.969635007Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=thanos
ts=2025-10-16T07:23:31.069539191Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheus
ts=2025-10-16T07:23:31.069572632Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheus
ts=2025-10-16T07:23:31.069578591Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheus
ts=2025-10-16T07:23:31.06958457Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheus
ts=2025-10-16T07:23:31.069588436Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheus
ts=2025-10-16T07:23:31.069594265Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheus
ts=2025-10-16T07:23:31.06959796Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheus
ts=2025-10-16T07:23:31.069603148Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheus
ts=2025-10-16T07:23:31.069607075Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheus
ts=2025-10-16T07:23:31.069612082Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheus
ts=2025-10-16T07:23:31.069615728Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheus
ts=2025-10-16T07:23:31.069620926Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheus
ts=2025-10-16T07:23:31.069624581Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheus
ts=2025-10-16T07:23:31.069629899Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheus
ts=2025-10-16T07:23:31.069638141Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=alertmanager
ts=2025-10-16T07:23:31.069646043Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=alertmanager
ts=2025-10-16T07:23:31.069658742Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheusagent
ts=2025-10-16T07:23:31.069667936Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheusagent
ts=2025-10-16T07:23:31.069682379Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheusagent
ts=2025-10-16T07:23:31.069688268Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheusagent
ts=2025-10-16T07:23:31.069692233Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheusagent
ts=2025-10-16T07:23:31.069703811Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheusagent
ts=2025-10-16T07:23:31.069707356Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheusagent
ts=2025-10-16T07:23:31.069713035Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheusagent
ts=2025-10-16T07:23:31.0697167Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheusagent
ts=2025-10-16T07:23:31.069722319Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheusagent
ts=2025-10-16T07:23:31.069725874Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheusagent
ts=2025-10-16T07:23:31.069730711Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheusagent
ts=2025-10-16T07:23:31.069734358Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheusagent
ts=2025-10-16T07:23:31.069739525Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheusagent
ts=2025-10-16T07:23:31.069747658Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheusagent
ts=2025-10-16T07:23:31.069753116Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheusagent
ts=2025-10-16T07:23:31.069756641Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheusagent
ts=2025-10-16T07:23:31.06976244Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheusagent
ts=2025-10-16T07:23:31.069766175Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheusagent
ts=2025-10-16T07:23:31.069772034Z level=info caller=/workspace/pkg/prometheus/agent/operator.go:470 msg="successfully synced all caches" component=prometheusagent-controller
ts=2025-10-16T07:23:31.069794378Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=thanos
ts=2025-10-16T07:23:31.069811745Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=thanos
ts=2025-10-16T07:23:31.069818144Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=thanos
ts=2025-10-16T07:23:31.069826257Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=thanos
ts=2025-10-16T07:23:31.069830243Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=thanos
ts=2025-10-16T07:23:31.069836632Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=thanos
ts=2025-10-16T07:23:31.069842811Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=thanos
ts=2025-10-16T07:23:31.069852396Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=thanos
ts=2025-10-16T07:23:31.069858345Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=thanos
ts=2025-10-16T07:23:31.06986776Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=thanos
ts=2025-10-16T07:23:31.069882352Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=thanos
ts=2025-10-16T07:23:31.069890554Z level=info caller=/workspace/pkg/thanos/operator.go:304 msg="successfully synced all caches" component=thanos-controller
ts=2025-10-16T07:23:31.170724939Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=alertmanager
ts=2025-10-16T07:23:31.17077134Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=alertmanager
ts=2025-10-16T07:23:31.170777058Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=alertmanager
ts=2025-10-16T07:23:31.170785861Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=alertmanager
ts=2025-10-16T07:23:31.170790869Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=alertmanager
ts=2025-10-16T07:23:31.170796918Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=alertmanager
ts=2025-10-16T07:23:31.170800704Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=alertmanager
ts=2025-10-16T07:23:31.170806522Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=alertmanager
ts=2025-10-16T07:23:31.170810158Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=alertmanager
ts=2025-10-16T07:23:31.170815826Z level=info caller=/workspace/pkg/alertmanager/operator.go:327 msg="successfully synced all caches" component=alertmanager-controller
ts=2025-10-16T07:23:31.171190314Z level=info caller=/workspace/pkg/alertmanager/operator.go:546 msg="sync alertmanager" component=alertmanager-controller key=monitoring/main
ts=2025-10-16T07:23:31.170685359Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheus
ts=2025-10-16T07:23:31.171421405Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheus
ts=2025-10-16T07:23:31.171428095Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheus
ts=2025-10-16T07:23:31.171434364Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheus
ts=2025-10-16T07:23:31.1714383Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheus
ts=2025-10-16T07:23:31.171443518Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:350 msg="Waiting for caches to sync" controller=prometheus
ts=2025-10-16T07:23:31.171447104Z level=info caller=/go/pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/shared_informer.go:357 msg="Caches are synced" controller=prometheus
ts=2025-10-16T07:23:31.171452482Z level=info caller=/workspace/pkg/prometheus/server/operator.go:421 msg="successfully synced all caches" component=prometheus-controller
ts=2025-10-16T07:23:31.17176832Z level=info caller=/workspace/pkg/prometheus/server/operator.go:770 msg="sync prometheus" component=prometheus-controller key=monitoring/k8s
ts=2025-10-16T07:23:31.211919003Z level=info caller=/workspace/pkg/prometheus/server/operator.go:770 msg="sync prometheus" component=prometheus-controller key=monitoring/k8s
ts=2025-10-16T07:23:31.266413863Z level=info caller=/workspace/pkg/alertmanager/operator.go:546 msg="sync alertmanager" component=alertmanager-controller key=monitoring/main


Anything else?


No response
Metadata
Metadata
Assignees
Labels
kind/bug
Type
No type
Projects
No projects
Milestone
No milestone
Relationships
None yet
Development
No branches or pull requests
Issue actions