Skip to content

prometheus v3.7.0 fails to start with relabeling config #17370

@KyriosGN0

Description

@KyriosGN0

What did you do?

i upgraded prometheus on GKE via prometheus-operator from version 3.6.0 to 3.7.0

What did you expect to see?

prometheus starts up successfully

What did you see instead? Under which circumstances?

prometheus fails to start with the error parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: \"$1\" is invalid 'replacement' for labelmap action

System information

Kuberentes 1.32

Prometheus version

Prometheus 3.7.0

Prometheus configuration file

- job_name: podMonitor/com-dev-kafka-cluster/entity-operator-resources-metrics/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names:
      - com-dev-kafka-cluster
  metrics_path: /metrics
  relabel_configs:
  - source_labels:
    - job
    target_label: __tmp_prometheus_job_name
  - action: drop
    source_labels:
    - __meta_kubernetes_pod_phase
    regex: (Failed|Succeeded)
  - action: keep
    source_labels:
    - __meta_kubernetes_pod_label_strimzi_io_component_type
    - __meta_kubernetes_pod_labelpresent_strimzi_io_component_type
    regex: (entity-operator);true
  - action: keep
    source_labels:
    - __meta_kubernetes_pod_container_port_name
    regex: healthcheck
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_pod_container_name
    target_label: container
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - target_label: job
    replacement: com-dev-kafka-cluster/entity-operator-resources-metrics
  - target_label: endpoint
    replacement: healthcheck
  - regex: __meta_kubernetes_pod_label_(strimzi_io_.+)
    action: labelmap
  - source_labels:
    - __meta_kubernetes_namespace
    separator: ;
    target_label: namespace
    regex: (.*)
    replacement: $1
    action: replace

Alertmanager version


Alertmanager configuration file

Logs

{
  "time": "2025-10-20T07:08:18.575139903Z",
  "level": "ERROR",
  "source": "main.go:642",
  "msg": "Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)",
  "file": "/etc/prometheus/config_out/prometheus.env.yaml",
  "err": "parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: \"$1\" is invalid 'replacement' for labelmap action"
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions