labelkeep dropping internal labels #12355
Description
What did you do?
Use labelkeep to set a list of allowed labels
What did you expect to see?
only allowed labels in regex would appear
What did you see instead? Under which circumstances?
nothing, as the target got into problems because labelkeep was dropping required labels
For example:
Creating target failed {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "xyz-controlplane-cadvisor", "error": "instance 0 in group node/480e21d6-np20: error parsing scrape interval: empty duration string", "errorVerbose": "error parsing scrape interval: empty duration string\ngithub.com/prometheus/prometheus/scrape.PopulateLabels\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/target.go:441\ngithub.com/prometheus/prometheus/scrape.TargetsFromGroup\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/target.go:506\ngithub.com/prometheus/prometheus/scrape.(*scrapePool).Sync\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/scrape.go:495\ngithub.com/prometheus/prometheus/scrape.(*Manager).reload.func1\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/manager.go:228\nruntime.goexit\n\truntime/asm_amd64.s:1594\ninstance 0 in group node/nodepool1-480e21d6-np20\ngithub.com/prometheus/prometheus/scrape.TargetsFromGroup\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/target.go:508\ngithub.com/prometheus/prometheus/scrape.(*scrapePool).Sync\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/scrape.go:495\ngithub.com/prometheus/prometheus/scrape.(*Manager).reload.func1\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/manager.go:228\nruntime.goexit\n\truntime/asm_amd64.s:1594"}
or
otel 2023-05-11T02:25:55.204Z error scrape/scrape.go:497 Creating target failed {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "controlplane-cadvisor", "error": "instance 0 in group node/nodepool1-480e21d6-np20: no address", "errorVerbose": "no address\ngithub.com/prometheus/prometheus/scrape.PopulateLabels\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/target.go:383\ngithub.com/prometheus/prometheus/scrape.TargetsFromGroup\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/target.go:506\ngithub.com/prometheus/prometheus/scrape.(*scrapePool).Sync\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/scrape.go:495\ngithub.com/prometheus/prometheus/scrape.(*Manager).reload.func1\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/manager.go:228\nruntime.goexit\n\truntime/asm_amd64.s:1594\ninstance 0 in group node/nodepool1-480e21d6-np20\ngithub.com/prometheus/prometheus/scrape.TargetsFromGroup\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/target.go:508\ngithub.com/prometheus/prometheus/scrape.(*scrapePool).Sync\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/scrape.go:495\ngithub.com/prometheus/prometheus/scrape.(*Manager).reload.func1\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/manager.go:228\nruntime.goexit\n\truntime/asm_amd64.s:1594"}
I think labelkeep should not drop any of __address__|__metrics_path__|__scrape_interval__|__scrape_timeout__|__scheme__
to avoid these issues
System information
No response
Prometheus version
No response
Prometheus configuration file
receivers:
prometheus:
config:
scrape_configs:
- job_name: pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $$1:$$2
source_labels:
- __address__
- __meta_kubernetes_pod_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: kubernetes_pod_name
- regex: 'vcluster_loft_sh_.*'
action: labeldrop
metric_relabel_configs:
- source_labels: [__name__]
regex: 'istio_.*'
action: drop
- regex: 'kustomize_.+'
action: labeldrop
- regex: 'x_.+'
action: labeldrop
- job_name: services
kubernetes_sd_configs:
- role: service
metrics_path: /probe
params:
module:
- http_2xx
relabel_configs:
- source_labels:
- __param_target
target_label: instance
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_service_name
target_label: kubernetes_name
metric_relabel_configs:
- source_labels: [__name__]
regex: 'istio_.*'
action: drop
- job_name: controlplane-cadvisor
kubernetes_sd_configs:
- role: node
scrape_timeout: 10s
scheme: https
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
metrics_path: /metrics/cadvisor
relabel_configs:
- action: replace
source_labels: [container_label_io_kubernetes_container_name]
target_label: container_name
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc.cluster.local:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/$${1}/proxy/metrics/cadvisor
- regex: 'kubernetes_io_.*|topology_kubernetes_io_.*|node_kubernetes_io_.*|job|instance'
action: labelkeep
metric_relabel_configs:
- source_labels:
- __name__
regex: '(container_memory_working_set_bytes|container_cpu_usage_seconds_total).*'
action: keepAlertmanager version
No response
Alertmanager configuration file
No response
Logs
2023-05-11T02:25:55.204Z error scrape/scrape.go:497 Creating target failed {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "controlplane-cadvisor", "error": "instance 0 in group node/nodepool1-480e21d6-np20: no address", "errorVerbose": "no address\ngithub.com/prometheus/prometheus/scrape.PopulateLabels\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/target.go:383\ngithub.com/prometheus/prometheus/scrape.TargetsFromGroup\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/target.go:506\ngithub.com/prometheus/prometheus/scrape.(*scrapePool).Sync\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/scrape.go:495\ngithub.com/prometheus/prometheus/scrape.(*Manager).reload.func1\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/manager.go:228\nruntime.goexit\n\truntime/asm_amd64.s:1594\ninstance 0 in group node/nodepool1-480e21d6-np20\ngithub.com/prometheus/prometheus/scrape.TargetsFromGroup\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/target.go:508\ngithub.com/prometheus/prometheus/scrape.(*scrapePool).Sync\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/scrape.go:495\ngithub.com/prometheus/prometheus/scrape.(*Manager).reload.func1\n\tgithub.com/prometheus/prometheus@v0.42.1-0.20230210113933-af1d9e01c7e4/scrape/manager.go:228\nruntime.goexit\n\truntime/asm_amd64.s:1594"}