-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
The upstream Prometheus Alertmanager supports configuring custom HTTP headers (for example, X-API-Key) via the http_headers field in http_config for webhooks and other HTTP-based receivers (see Alertmanager docs: http_header).
However, the AlertmanagerConfig CRD in prometheus-operator does not currently support this field:
- The Go types in
pkg/alertmanager/types.go(specificallyhttpClientConfig) do not expose any option to set arbitrary HTTP headers (such ashttp_headersorheaders). - As a result, it is not possible to set custom headers (e.g., for API authentication via
X-API-Key) for webhooks or other HTTP receivers using the operator.
This is inconsistent with the upstream Alertmanager and prevents use-cases which require HTTP header-based authentication for notifications.
Steps to reproduce:
- Try to use the
http_headersfield underhttp_configfor awebhook_configsreceiver in anAlertmanagerConfig. - The operator does not recognize this key; the custom header is not set and not present in outgoing requests.
Expected behavior:
It should be possible to configure arbitrary HTTP headers for HTTP-based receivers via http_config.http_headers as supported by the upstream Alertmanager.
Actual behavior:
There is currently no way (via CRD) to declare custom HTTP headers for outgoing webhooks.
Affected versions:
- prometheus-operator:
- This problem is visible in
pkg/alertmanager/types.go
Proposed solution:
Add support for the http_headers field (as specified in the Alertmanager documentation) to the httpClientConfig struct and expose it through the AlertmanagerConfig CRD.
Thank you!
Prometheus Operator Version
quay.io/prometheus-operator/prometheus-operator:v0.87.1
labels:
app: kube-prometheus-stack-operator
app.kubernetes.io/component: prometheus-operator
app.kubernetes.io/instance: gst-monitoring
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
app.kubernetes.io/part-of: kube-prometheus-stack
app.kubernetes.io/version: 80.4.1
chart: kube-prometheus-stack-80.4.1Kubernetes Version
1.32.9Kubernetes Cluster Type
AKS
How did you deploy Prometheus-Operator?
helm chart:prometheus-community/kube-prometheus-stack
Manifests
prometheus-operator log output
ts=2026-01-29T08:32:02.861226964Z level=error caller=/workspace/pkg/operator/resource_reconciler.go:678 msg="Unhandled Error" logger=UnhandledError err="sync \"monitoring/gst-monitoring-kube-promet-alertmanager\" failed: provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors:\n line 44: field http_headers not found in type alertmanager.httpClientConfig"Anything else?
No response