Skip to content

AlertmanagerConfig does not support http_headers in webhook_configs (missing support for custom HTTP headers) #8341

@bwilhelm-gst

Description

@bwilhelm-gst

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

The upstream Prometheus Alertmanager supports configuring custom HTTP headers (for example, X-API-Key) via the http_headers field in http_config for webhooks and other HTTP-based receivers (see Alertmanager docs: http_header).

However, the AlertmanagerConfig CRD in prometheus-operator does not currently support this field:

  • The Go types in pkg/alertmanager/types.go (specifically httpClientConfig) do not expose any option to set arbitrary HTTP headers (such as http_headers or headers).
  • As a result, it is not possible to set custom headers (e.g., for API authentication via X-API-Key) for webhooks or other HTTP receivers using the operator.

This is inconsistent with the upstream Alertmanager and prevents use-cases which require HTTP header-based authentication for notifications.


Steps to reproduce:

  1. Try to use the http_headers field under http_config for a webhook_configs receiver in an AlertmanagerConfig.
  2. The operator does not recognize this key; the custom header is not set and not present in outgoing requests.

Expected behavior:

It should be possible to configure arbitrary HTTP headers for HTTP-based receivers via http_config.http_headers as supported by the upstream Alertmanager.

Actual behavior:

There is currently no way (via CRD) to declare custom HTTP headers for outgoing webhooks.


Affected versions:


Proposed solution:

Add support for the http_headers field (as specified in the Alertmanager documentation) to the httpClientConfig struct and expose it through the AlertmanagerConfig CRD.


Thank you!

Prometheus Operator Version

quay.io/prometheus-operator/prometheus-operator:v0.87.1

labels:
    app: kube-prometheus-stack-operator
    app.kubernetes.io/component: prometheus-operator
    app.kubernetes.io/instance: gst-monitoring
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 80.4.1
    chart: kube-prometheus-stack-80.4.1

Kubernetes Version

1.32.9

Kubernetes Cluster Type

AKS

How did you deploy Prometheus-Operator?

helm chart:prometheus-community/kube-prometheus-stack

Manifests

prometheus-operator log output

ts=2026-01-29T08:32:02.861226964Z level=error caller=/workspace/pkg/operator/resource_reconciler.go:678 msg="Unhandled Error" logger=UnhandledError err="sync \"monitoring/gst-monitoring-kube-promet-alertmanager\" failed: provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors:\n  line 44: field http_headers not found in type alertmanager.httpClientConfig"

Anything else?

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions