Hi 👋 ,
We have an Agent configuration which is referencing 2 Secrets using the K8S Secret Provider, for example:
streams:
- id: elasticsearch/metrics-elasticsearch.stack_monitoring.ccr
data_stream:
dataset: elasticsearch.stack_monitoring.ccr
type: metrics
metricsets:
- ccr
hosts:
- 'https://${kubernetes.pod.ip}:9200'
headers:
Authorization: Bearer ${kubernetes_secrets.elastic-agent.somesecret1.value}
AnotherHeader: SharedSecret ${kubernetes_secrets.elastic-agent.somesecret2.value}
scope: node
period: 10sWhile investigating a global performance issue in our K8S cluster we observed a fair amount of requests to our K8S API server, with a constant rate of around 4 to 8 GET requests on these 2 Secrets per second. The audit trace suggests that that those requests are made by Agents managed by a DaemonSet on our clusters:
Audit Event Sample
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"auditID": "4a1822f1-6509-45ad-a767-4174f0beded",
"stage": "ResponseComplete",
"requestURI": "/api/v1/namespaces/elastic-agent/secrets/somesecret1",
"verb": "get",
"user": {
"username": "system:serviceaccount:elastic-agent:elastic-agent",
"uid": "7322c0d3-f498-4dae-a7e5-699d899fdfdf",
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:elastic-agent",
"system:authenticated"
],
"extra": {
"authentication.kubernetes.io/pod-name": [
"elastic-agent-b5hff"
],
"authentication.kubernetes.io/pod-uid": [
"64cb7be8-3d16-470f-afe6-c77fd092ffff"
]
}
},
"sourceIPs": [
"192.168.92.12"
],
"userAgent": "Go-http-client/2.0",
"objectRef": {
"resource": "secrets",
"namespace": "elastic-agent",
"name": "somesecret1",
"apiVersion": "v1"
},
"responseStatus": {
"metadata": {},
"code": 200
},
"requestReceivedTimestamp": "2023-09-20T14:55:44.652043Z",
"stageTimestamp": "2023-09-20T14:55:44.655909Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"elastic-agent\" of ClusterRole \"elastic-agent\" to ServiceAccount \"elastic-agent/elastic-agent\""
}
}
I think these Secrets should be cached, as it is usually the case when using the K8S client from the controller runtime for example. It would definitely help in our case.
We tried to have a look at the code and we were wondering if the client calls are coming from here.
Thanks!
For confirmed bugs, please report:
- Version:
docker.elastic.co/beats/elastic-agent:8.8.2
- Operating System: K8S 1.25 on EKS
Hi 👋 ,
We have an Agent configuration which is referencing 2
Secretsusing the K8S Secret Provider, for example:While investigating a global performance issue in our K8S cluster we observed a fair amount of requests to our K8S API server, with a constant rate of around 4 to 8
GETrequests on these 2 Secrets per second. The audit trace suggests that that those requests are made by Agents managed by aDaemonSeton our clusters:Audit Event Sample
{ "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Metadata", "auditID": "4a1822f1-6509-45ad-a767-4174f0beded", "stage": "ResponseComplete", "requestURI": "/api/v1/namespaces/elastic-agent/secrets/somesecret1", "verb": "get", "user": { "username": "system:serviceaccount:elastic-agent:elastic-agent", "uid": "7322c0d3-f498-4dae-a7e5-699d899fdfdf", "groups": [ "system:serviceaccounts", "system:serviceaccounts:elastic-agent", "system:authenticated" ], "extra": { "authentication.kubernetes.io/pod-name": [ "elastic-agent-b5hff" ], "authentication.kubernetes.io/pod-uid": [ "64cb7be8-3d16-470f-afe6-c77fd092ffff" ] } }, "sourceIPs": [ "192.168.92.12" ], "userAgent": "Go-http-client/2.0", "objectRef": { "resource": "secrets", "namespace": "elastic-agent", "name": "somesecret1", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "code": 200 }, "requestReceivedTimestamp": "2023-09-20T14:55:44.652043Z", "stageTimestamp": "2023-09-20T14:55:44.655909Z", "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"elastic-agent\" of ClusterRole \"elastic-agent\" to ServiceAccount \"elastic-agent/elastic-agent\"" } }I think these
Secretsshould be cached, as it is usually the case when using the K8S client from the controller runtime for example. It would definitely help in our case.We tried to have a look at the code and we were wondering if the client calls are coming from here.
Thanks!
For confirmed bugs, please report:
docker.elastic.co/beats/elastic-agent:8.8.2