It was found during #23679. Dynamic inputs setup to automatically discover scheduler and controllermanager Pods does not work right now and we get the following error:
2021-01-25T15:36:29.224Z DEBUG application/periodic.go:40 Failed to read configuration, error: could not emit configuration: could not create the AST from the configuration: missing field accessing 'inputs' (source:'/etc/agent.yml')
Agent version:
Daemon: 7.11.0-SNAPSHOT (build: 5691b7c654663ee51f1d347ce02bda332fb300e7 at 2021-01-25 09:48:16 +0000 UTC)
Binary: 7.11.0-SNAPSHOT (build: 5691b7c654663ee51f1d347ce02bda332fb300e7 at 2021-01-25 09:48:16 +0000 UTC)
agent.yml:
id: ef9cc740-5bf0-11eb-8b51-39775155c3f5
revision: 2
outputs:
default:
type: elasticsearch
hosts:
- >-
${ES_HOST}
username: ${ES_USERNAME}
password: ${ES_PASSWORD}
agent:
monitoring:
enabled: true
use_output: default
logs: true
metrics: true
providers.kubernetes:
node: ${NODE_NAME}
scope: node
inputs:
- id: 934ef8aa-ed19-405b-8160-ebf62e3d32f8
name: kubernetes-node-metrics
revision: 1
type: kubernetes/metrics
use_output: default
meta:
package:
name: kubernetes
version: 0.2.8
data_stream:
namespace: default
streams:
- id: >-
kubernetes/metrics-kubernetes.controllermanager-3d50c483-2327-40e7-b3e5-d877d4763fe1
data_stream:
dataset: kubernetes.controllermanager
type: metrics
metricsets:
- controllermanager
hosts:
- '${kubernetes.pod.ip}:10252'
period: 10s
#condition: startsWith(${kubernetes.pod.name}, "kube-controller-manager")
condition: ${kubernetes.pod.labels.component} == 'kube-controller-manager'
- id: >-
kubernetes/metrics-kubernetes.scheduler-3d50c483-2327-40e7-b3e5-d877d4763fe1
data_stream:
dataset: kubernetes.scheduler
type: metrics
metricsets:
- scheduler
hosts:
- '${kubernetes.pod.ip}:10251'
period: 10s
#condition: startsWith(${kubernetes.pod.name}, "kube-scheduler")
condition: ${kubernetes.pod.labels.component} == 'kube-scheduler'
- id: >-
kubernetes/metrics-kubernetes.proxy-3d50c483-2327-40e7-b3e5-d877d4763fe1
data_stream:
dataset: kubernetes.proxy
type: metrics
metricsets:
- proxy
hosts:
- 'localhost:10249'
period: 10s
- id: >-
kubernetes/metrics-kubernetes.container-934ef8aa-ed19-405b-8160-ebf62e3d32f8
data_stream:
dataset: kubernetes.container
type: metrics
metricsets:
- container
add_metadata: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
hosts:
- 'https://${env.NODE_NAME}:10250'
period: 10s
ssl.verification_mode: noneConverting ${NODE_NAME} placeholders to ${env.NODE_NAME} does not fix the problem and even if we remove all other datastream configs and leave only the dynamic one it still gives the error.
In addition, if we remove the dynamic inputs part and have ${env.NODE_NAME} we still get the same error.
Also a condition like condition: startsWith(${kubernetes.pod.name}, "kube-controller-manager") fails and the inspect command gives the following:
condition "startsWith(${kubernetes.pod.name}, "kube-controller-manager")" evaluation failed: startsWith: accepts exactly 2 string arguments; recieved *eql.null and string
Related to #23613
It was found during #23679. Dynamic inputs setup to automatically discover
schedulerandcontrollermanagerPods does not work right now and we get the following error:Agent version:
agent.yml:Converting
${NODE_NAME}placeholders to${env.NODE_NAME}does not fix the problem and even if we remove all other datastream configs and leave only the dynamic one it still gives the error.In addition, if we remove the dynamic inputs part and have
${env.NODE_NAME}we still get the same error.Also a condition like
condition: startsWith(${kubernetes.pod.name}, "kube-controller-manager")fails and theinspectcommand gives the following:Related to #23613