-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Host network broken after one of the underlying interfaces of a bond goes down #18706
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
On Equinix Metal the network setup is a bond of two NICs using LACP.
When Cilium is used as CNI for Kubernetes on Flatcar Container Linux, and one of the two NIC interfaces goes down, the host network is broken and remains broken even if the underlying interface goes up again.
The network looks normal but ping 1.1 leads to no outgoing packets visible in tcpdump -i bond0 (ping reports 100% packet loss) and ping 127.0.0.1 leads to ping: sendmsg: Operation not permitted.
We could not restore the network even after terminating the Pods and kubelet on the node, flushing nft and deleting the Cilium interfaces (maybe BPF programs are still loaded and not cleaned up?)
IPv6 is not affected, ping6 2606:4700:4700::1111 works.
Cilium Version
1.9, 1.10, 1.11
Kernel Version
from 5.10.52 to 5.10.84
Kubernetes Version
1.22
Sysdump
🔍 Collecting Kubernetes nodes
failed to create sysdump collector: failed to collect Kubernetes nodes: Get "https://136.144.49.47:6443/api/v1/nodes": dial tcp 136.144.49.47:6443: i/o timeout
Relevant log output
Kernel
Feb 04 15:50:06 kernel: bond0: (slave enp2s0f0np0): link status down for interface, disabling it in 200 ms
Feb 04 15:50:06 kernel: bond0: (slave enp2s0f0np0): link status down for interface, disabling it in 200 ms
Feb 04 15:50:06 kernel: bond0: (slave enp2s0f0np0): link status down for interface, disabling it in 200 ms
Feb 04 15:50:06 kernel: bond0: (slave enp2s0f0np0): link status down for interface, disabling it in 200 ms
Feb 04 15:50:06 kernel: bond0: (slave enp2s0f0np0): invalid new link 1 on slave
Feb 04 15:50:06 kernel: mlx5_core 0000:02:00.0: modify lag map port 1:2 port 2:2
Feb 04 15:50:06 kernel: bond0: (slave enp2s0f0np0): link status definitely down, disabling slave
Feb 04 15:51:09 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): lxc_health: link becomes ready
Feb 04 15:51:10 kernel: lxc_health: Caught tx_queue_len zero misconfig
Feb 04 15:51:33 kernel: mlx5_core 0000:02:00.0 enp2s0f0np0: Link down
Feb 04 15:51:33 kernel: mlx5_core 0000:02:00.0 enp2s0f0np0: Link up
Feb 04 15:51:33 kernel: bond0: (slave enp2s0f0np0): link status up again after 200 ms
Feb 04 15:51:33 kernel: bond0: (slave enp2s0f0np0): link status definitely up, 10000 Mbps full duplex
Feb 04 15:51:35 kernel: mlx5_core 0000:02:00.0: modify lag map port 1:1 port 2:2
Anything else?
Flatcar releases 2905.x.y to 3033.x.y are affected, running systemd from 247 to 249 (maybe relevant because systemd-networkd is used)
Flatcar releases 2764.x.y are not affected (kernel 5.10.43, systemd 247)
Reproduce it by provisioning an Equinix Metal machine with Flatcar Stable (used c3.small.x86).
Ensure it is on the latest version:
update_engine_client -update
sudo rm -f /etc/systemd/system/containerd.service.d/10-use-cgroupfs.conf
sudo sed -i 's/systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller//' /usr/share/oem/grub.cfg
sudo systemctl reboot
Set up a one-node Cilium cluster, using the script contents at the end:
sudo ./install.sh
Then, this action is valid and should not any harm, but now does:
sudo ip link set enp2s0f0np0 down
(and sudo ip link set enp2s0f0np0 up does not help)
The install.sh script used above:
#!/bin/bash
set -xe
systemctl enable --now docker
modprobe br_netfilter
cat <<EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
CNI_VERSION="v0.8.2"
CRICTL_VERSION="v1.17.0"
RELEASE_VERSION="v0.4.0"
DOWNLOAD_DIR=/opt/bin
RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
mkdir -p /opt/cni/bin
mkdir -p /etc/systemd/system/kubelet.service.d
curl() {
command curl -sSfL "$@"
}
curl "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" | tar -C /opt/cni/bin -xz
curl "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz" | tar -C $DOWNLOAD_DIR -xz
curl "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | tee /etc/systemd/system/kubelet.service
curl "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
curl --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-amd64.tar.gz /opt/bin
rm cilium-linux-amd64.tar.gz{,.sha256sum}
chmod +x {kubeadm,kubelet,kubectl}
mv {kubeadm,kubelet,kubectl} $DOWNLOAD_DIR/
systemctl enable --now kubelet
#systemctl status kubelet
cat <<EOF | tee kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
controllerManager:
extraArgs:
flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
networking:
podSubnet: "192.168.254.0/24"
EOF
# For explicit cgroupdriver selection
# ---
# kind: KubeletConfiguration
# apiVersion: kubelet.config.k8s.io/v1beta1
# cgroupDriver: systemd
# For containerd
# apiVersion: kubeadm.k8s.io/v1beta2
# kind: InitConfiguration
# nodeRegistration:
# criSocket: "unix:///run/containerd/containerd.sock
export PATH=$PATH:$DOWNLOAD_DIR
kubeadm config images pull
kubeadm init --config kubeadm-config.yaml
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.9.4/install/kubernetes/quick-install.yaml
kubectl taint nodes --all node-role.kubernetes.io/master-
kubectl get pods -A
kubectl get nodes -o wide
kubectl apply -f https://k8s.io/examples/application/deployment.yaml
kubectl expose deployment.apps/nginx-deployment
curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
tar xzvfC cilium-linux-amd64.tar.gz /opt/bin
rm cilium-linux-amd64.tar.gz{,.sha256sum}
Code of Conduct
- I agree to follow this project's Code of Conduct