Relevant telegraf.conf:
[[inputs.prometheus]]
metric_version = 2
monitor_kubernetes_pods = trueSystem info:
Telegraf 1.18.3
Image: telegraf:1.18.3-alpine on k8s 1.19.8-eks-96780e
Expected behavior:
On pod launch with debug log-level enabled, the logs should show a list of pods to scrape metrics from:
2021-06-10T10:23:55Z I! Starting Telegraf 1.18.2
2021-06-10T10:23:55Z I! Using config file: /etc/telegraf/telegraf.conf
2021-06-10T10:23:55Z I! Loaded inputs: internal prometheus
2021-06-10T10:23:55Z I! Loaded aggregators:
2021-06-10T10:23:55Z I! Loaded processors:
2021-06-10T10:23:55Z I! Loaded outputs: health prometheus_client
2021-06-10T10:23:55Z I! Tags enabled: host=telegraf-polling-service
2021-06-10T10:23:55Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"telegraf-polling-service", Flush Interval:5s
2021-06-10T10:23:55Z D! [agent] Initializing plugins
2021-06-10T10:23:55Z D! [agent] Connecting outputs
2021-06-10T10:23:55Z D! [agent] Attempting connection to [outputs.prometheus_client]
2021-06-10T10:23:55Z I! [outputs.prometheus_client] Listening on http://[::]:9999/metrics
2021-06-10T10:23:55Z D! [agent] Successfully connected to outputs.prometheus_client
2021-06-10T10:23:55Z D! [agent] Attempting connection to [outputs.health]
2021-06-10T10:23:55Z I! [outputs.health] Listening on http://[::]:8888
2021-06-10T10:23:55Z D! [agent] Successfully connected to outputs.health
2021-06-10T10:23:55Z D! [agent] Starting service inputs
2021-06-10T10:23:56Z D! [inputs.prometheus] will scrape metrics from "http://<ip>:9102/metrics"
2021-06-10T10:23:56Z D! [inputs.prometheus] will scrape metrics from "http://<ip>:8000/metrics"
and so on...
Actual behavior:
2021-06-10T10:28:09Z I! Starting Telegraf 1.18.3
2021-06-10T10:28:09Z I! Using config file: /etc/telegraf/telegraf.conf
2021-06-10T10:28:09Z I! Loaded inputs: internal prometheus
2021-06-10T10:28:09Z I! Loaded aggregators:
2021-06-10T10:28:09Z I! Loaded processors:
2021-06-10T10:28:09Z I! Loaded outputs: health prometheus_client
2021-06-10T10:28:09Z I! Tags enabled: host=telegraf-polling-service
2021-06-10T10:28:09Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"telegraf-polling-service", Flush Interval:5s
2021-06-10T10:28:09Z D! [agent] Initializing plugins
2021-06-10T10:28:09Z D! [agent] Connecting outputs
2021-06-10T10:28:09Z D! [agent] Attempting connection to [outputs.prometheus_client]
2021-06-10T10:28:09Z I! [outputs.prometheus_client] Listening on http://[::]:9999/metrics
2021-06-10T10:28:09Z D! [agent] Successfully connected to outputs.prometheus_client
2021-06-10T10:28:09Z D! [agent] Attempting connection to [outputs.health]
2021-06-10T10:28:09Z I! [outputs.health] Listening on http://[::]:8888
2021-06-10T10:28:09Z D! [agent] Successfully connected to outputs.health
2021-06-10T10:28:09Z D! [agent] Starting service inputs
2021-06-10T10:28:14Z D! [outputs.health] Wrote batch of 5 metrics in 3.65µs
2021-06-10T10:28:14Z D! [outputs.health] Buffer fullness: 0 / 100000 metrics
2021-06-10T10:28:14Z D! [outputs.prometheus_client] Wrote batch of 5 metrics in 125.992µs
2021-06-10T10:28:14Z D! [outputs.prometheus_client] Buffer fullness: 0 / 100000 metrics
Additional info:
The only difference between these 2 runs is the version. I simply updated the image tag manually in the deployment to produce these logs. I can confirm that the relevant annotations are present on the other pods as required.
Relevant telegraf.conf:
System info:
Telegraf 1.18.3
Image: telegraf:1.18.3-alpine on k8s 1.19.8-eks-96780e
Expected behavior:
On pod launch with debug log-level enabled, the logs should show a list of pods to scrape metrics from:
Actual behavior:
Additional info:
The only difference between these 2 runs is the version. I simply updated the image tag manually in the deployment to produce these logs. I can confirm that the relevant annotations are present on the other pods as required.