Skip to content

Cilium Error: "metadata resolver: pod store out-of-date" for Static Pod in Kubernetes #34197

Closed
Bug
#45016
#34389
Closed
Cilium Error: "metadata resolver: pod store out-of-date" for Static Pod in Kubernetes#34197
Bug
#45016
#34389
Assignees
christarazi
Labels
area/agentCilium agent related.area/cniImpacts the Container Networking Interface between Cilium and the orchestrator.area/daemonImpacts operation of the Cilium daemon.kind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and next steps.
@kosticana

Description

@kosticana
kosticana
opened on Aug 6, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Version

higher than v1.16.0 and lower than v1.17.0

What happened?

Hi,
I am encountering an issue with Cilium in my Kubernetes cluster related to a static pod. Here are the details of my setup and the problem.

I installed Kubernetes (version 1.28.11) on an AWS EC2 instance created from an Ubuntu 22.04 AMI.
After setting up the cluster and installing Cilium (version 1.16.0) as the CNI, I deployed a static pod by placing its manifest in /etc/kubernetes/manifests/.
The static pod is running correctly, but cilium status reports an error:

cilium status
   /¯¯\
 /¯¯\__/¯¯\    Cilium:             1 errors
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium-envoy       Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium             Desired: 1, Ready: 1/1, Available: 1/1
Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium             Running: 1
                       cilium-envoy       Running: 1
                       cilium-operator    Running: 1
Cluster Pods:          2/3 managed by Cilium
Helm chart version:    
Image versions         cilium             quay.io/cilium/cilium:v1.16.0@sha256:46ffa4ef3cf6d8885dcc4af5963b0683f7d59daa90d49ed9fb68d3b1627fe058: 1
                       cilium-envoy       quay.io/cilium/cilium-envoy:v1.29.7-39a2a56bbd5b3a591f69dbca51d3e30ef97e0e51@sha256:bd5ff8c66716080028f414ec1cb4f7dc66f40d2fb5a009fff187f4a9b90b566b: 1
                       cilium-operator    quay.io/cilium/operator-generic:v1.16.0@sha256:d6621c11c4e4943bf2998af7febe05be5ed6fdcf812b27ad4388f47022190316: 1
Errors:                cilium             cilium-c9ptw    controller resolve-labels-default/nginx-static-pod-master-node is failing since 7s (7x): metadata resolver: pod store out-of-date

Static pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-static-pod
  namespace: default
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80

./k8s-unmanaged.sh 
Skipping pods with host networking enabled or with status not in Running or Pending phase...
default/nginx-static-pod-master-node

I expected cilium status to report that all pods are managed correctly without any metadata resolver issues.

How can we reproduce the issue?

Steps taken:

I deployed an AWS EC2 instance created from an Ubuntu 22.04 AMI and executed the following steps:

  1. Disabled swap:
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab
  1. Kernel Modules and Sysctl Settings:
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system
  1. Installed Kubernetes Components:
VERSION=1.28.11-1.1
sudo apt-get install -y kubelet=$VERSION kubeadm=$VERSION kubectl=$VERSION
  1. Installed and Configured Docker and Containerd
sudo apt install docker.io
sudo mkdir /etc/containerd
sudo sh -c "containerd config default > /etc/containerd/config.toml"
sudo sed -i 's/ SystemdCgroup = false/ SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd.service
sudo systemctl restart kubelet.service
  1. Initialized Kubernetes Master Node
sudo kubeadm config images pull
sudo kubeadm init --pod-network-cidr=172.48.0.0/12
  1. Installed Cilium
curl -LO https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz
cilium install

Cilium Version

1.16.0

Kernel Version

Linux master-node 6.5.0-1022-aws #22~22.04.1-Ubuntu SMP Fri Jun 14 16:31:00 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

Client Version: v1.28.11
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.12

Regression

No response

Sysdump

No response

Relevant log output

time="2024-08-06T09:07:00Z" level=info msg="Create endpoint request" addressing="&{10.0.0.156 bf3f3452-b967-404f-9397-1cae9c727dc7 default   }" containerID=938071c657629b04d242c686bd897e05704c5bab68243c6880310bc29139321a containerInterface=eth0 datapathConfiguration="&{false false false false false <nil>}" interface=lxc5585b28160dc k8sPodName=default/nginx-static-pod-master-node k8sUID=83ca69a09f5f9c22e393d79814308af4 labels="[]" subsys=daemon sync-build=true
time="2024-08-06T09:07:00Z" level=warning msg="Detected outdated Pod UID during Endpoint creation. Endpoint creation cannot proceed with an outdated Pod store. Attempting to fetch latest Pod." k8sPodName=default/nginx-static-pod-master-node k8sUID=83ca69a09f5f9c22e393d79814308af4 subsys=daemon
time="2024-08-06T09:07:00Z" level=warning msg="Timeout occurred waiting for Pod store, fetching latest Pod via the apiserver." k8sPodName=default/nginx-static-pod-master-node k8sUID=83ca69a09f5f9c22e393d79814308af4 subsys=daemon
time="2024-08-06T09:07:00Z" level=warning msg="Unable to fetch kubernetes labels" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=0 error="pod store outdated" ipv4= ipv6= k8sPodName=/ subsys=api
time="2024-08-06T09:07:00Z" level=info msg="New endpoint" ciliumEndpointName=default/nginx-static-pod-master-node containerID=938071c657 containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=119 ipv4=10.0.0.156 ipv6= k8sPodName=default/nginx-static-pod-master-node subsys=endpoint
time="2024-08-06T09:07:00Z" level=info msg="Resolving identity labels (blocking)" ciliumEndpointName=default/nginx-static-pod-master-node containerID=938071c657 containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=119 identityLabels="reserved:init" ipv4=10.0.0.156 ipv6= k8sPodName=default/nginx-static-pod-master-node subsys=endpoint
time="2024-08-06T09:07:00Z" level=info msg="Identity of endpoint changed" ciliumEndpointName=default/nginx-static-pod-master-node containerID=938071c657 containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=119 identity=5 identityLabels="reserved:init" ipv4=10.0.0.156 ipv6= k8sPodName=default/nginx-static-pod-master-node oldIdentity="no identity" subsys=endpoint
time="2024-08-06T09:07:00Z" level=info msg="Waiting for endpoint to be generated" ciliumEndpointName=default/nginx-static-pod-master-node containerID=938071c657 containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=119 identity=5 ipv4=10.0.0.156 ipv6= k8sPodName=default/nginx-static-pod-master-node subsys=endpoint
time="2024-08-06T09:07:00Z" level=info msg="Program cil_from_container attached to device lxc5585b28160dc using legacy tc" subsys=datapath-loader
time="2024-08-06T09:07:00Z" level=info msg="Reloaded endpoint BPF program" ciliumEndpointName=default/nginx-static-pod-master-node containerID=938071c657 containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=119 identity=5 ipv4=10.0.0.156 ipv6= k8sPodName=default/nginx-static-pod-master-node subsys=endpoint
time="2024-08-06T09:07:00Z" level=info msg="Successful endpoint creation" ciliumEndpointName=default/nginx-static-pod-master-node containerID=938071c657 containerInterface= datapathPolicyRevision=1 desiredPolicyRevision=1 endpointID=119 identity=5 ipv4=10.0.0.156 ipv6= k8sPodName=default/nginx-static-pod-master-node subsys=daemon

Anything else?

No response

Cilium Users Document

  • Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

Labels

area/agentCilium agent related.area/cniImpacts the Container Networking Interface between Cilium and the orchestrator.area/daemonImpacts operation of the Cilium daemon.kind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and next steps.

Type

Bug

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions