test-me-please

test-1.16-netnext

test-1.16-netnext

Looks like this is timing out on waiting for the kubelet to become ready: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-net-next/1445/ I think this is also what happened on the previous run and on the previous PR #17352.

I've tried to reproduce this locally but so far wasn't able to (i.e. the VMs always come up as expected with the new VM image).

test-me-please

test-1.16-netnext

Looks like this is timing out on waiting for the kubelet to become ready: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-net-next/1445/ I think this is also what happened on the previous run and on the previous PR #17352.

I've tried to reproduce this locally but so far wasn't able to (i.e. the VMs always come up as expected with the new VM image).

I was using the dev VM instructions instead of the CI VM instructions (thanks @nbusseneau for pointing this out). Using
K8S_VERSION=1.16 K8S_NODES=1 NETNEXT=1 ./vagrant-local-start.sh I was able to reproduce the VM provisioning failure. It looks like kubelet doesn't come up:

vagrant@k8s1:~$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Thu 2021-09-16 13:21:21 UTC; 86ms ago
     Docs: https://kubernetes.io/docs/home/
  Process: 11042 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 11042 (code=exited, status=255)
vagrant@k8s1:~$ journalctl -xeu kubelet
[...]
Sep 16 13:22:13 k8s1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is RESULT.
Sep 16 13:22:13 k8s1 kubelet[11384]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See htt
Sep 16 13:22:13 k8s1 kubelet[11384]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See http
Sep 16 13:22:13 k8s1 kubelet[11384]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See htt
Sep 16 13:22:13 k8s1 kubelet[11384]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See http
Sep 16 13:22:13 k8s1 kubelet[11384]: I0916 13:22:13.995746   11384 server.go:410] Version: v1.16.15
Sep 16 13:22:13 k8s1 kubelet[11384]: I0916 13:22:13.996551   11384 plugins.go:100] No cloud provider specified.
Sep 16 13:22:13 k8s1 kubelet[11384]: I0916 13:22:13.996614   11384 server.go:773] Client rotation is on, will bootstrap in background
Sep 16 13:22:14 k8s1 kubelet[11384]: I0916 13:22:14.009873   11384 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Sep 16 13:22:14 k8s1 kubelet[11384]: I0916 13:22:14.234620   11384 server.go:636] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Sep 16 13:22:14 k8s1 kubelet[11384]: F0916 13:22:14.235267   11384 server.go:271] failed to run Kubelet: strconv.Atoi: parsing "1\n\x00": invalid syntax
Sep 16 13:22:14 k8s1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 16 13:22:14 k8s1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

The following looks like the culprit for failing to start kubelet:

Sep 16 13:22:14 k8s1 kubelet[11384]: F0916 13:22:14.235267   11384 server.go:271] failed to run Kubelet: strconv.Atoi: parsing "1\n\x00": invalid syntax

Not sure where that 1\n\x00 is coming from. Maybe command line or a config file? Will try to dig further...

Starting kubelet with --v=8 unfortunately doesn't give any further clues.

But it smells a bit like that value could come from a sysctl or sysfs file. And indeed, if I check for uses of strconv.Atoi in code around the last log messages I noticed this:

https://github.com/kubernetes/kubernetes/blob/2adc8d7091e89b6e3ca8d048140618ec89b39369/pkg/kubelet/cm/cpuset/cpuset.go#L258-L297

which gets called from here:

https://github.com/kubernetes/kubernetes/blob/2adc8d7091e89b6e3ca8d048140618ec89b39369/pkg/kubelet/cm/cpumanager/topology/topology.go#L310-L325

Comparing the content of that file for the net-next kernel :

vagrant@k8s1:~$ xxd /sys/devices/system/node/node0/cpulist
00000000: 302d 310a 00                             0-1..

with my development machine running kernel 5.11:

tklauser@orbital:~ $ xxd /sys/devices/system/node/node0/cpulist
00000000: 302d 370a                                0-7.

Notice the trailing null byte 00 in the output on the net-next kernel (the difference in the CPU numbers is expected as these are different systems). This looks like a kernel regression to me.

Notice the trailing null byte 00 in the output on the net-next kernel (the difference in the CPU numbers is expected as these are different systems). This looks like a kernel regression to me.

It's indeed a kernel regression which incidentally breaks kubectl. It looks like the regression was introduced by the series at https://lore.kernel.org/all/20210806110251.560-1-song.bao.hua@hisilicon.com/

I've sent a potential fix upstream: https://lore.kernel.org/lkml/20210916222705.13554-1-tklauser@distanz.ch/T/#u

I've sent a potential fix upstream: https://lore.kernel.org/lkml/20210916222705.13554-1-tklauser@distanz.ch/T/#u

Patch was applied to the driver-core tree and should hopefully hit Linus' tree for 5.15-rc3.

Marking this PR as draft until that is the case and bpf-next picked up the fix as well at which point we'd need to build new net-next VM images to pick up into this PR.

Notice the trailing null byte 00 in the output on the net-next kernel (the difference in the CPU numbers is expected as these are different systems). This looks like a kernel regression to me.

It's indeed a kernel regression which incidentally breaks kubectl. It looks like the regression was introduced by the series at https://lore.kernel.org/all/20210806110251.560-1-song.bao.hua@hisilicon.com/

I've sent a potential fix upstream: https://lore.kernel.org/lkml/20210916222705.13554-1-tklauser@distanz.ch/T/#u

Quick status update: The patch has made it upstream into Linus' tree but hasn't made it into the bpf-next tree yet. In order to unblock this PR and PRs depending on a Vagrant VM bump, I've opened cilium/packer-ci-build#284 which applies said patch manually before building the kernel for the bpf-next kernel image. Once that is approved and merged and a new VM image was built, we can finally move forward with this PR.

/test

Updated to the latest VM image versions (namely to include the kernel patch mentioned above) and included a revert of commit 1c42ae9 as suggested by Paul.

Job 'Cilium-PR-K8s-GKE' failed and has not been observed before, so may be related to your PR:

K8sDatapathConfig Host firewall With native routing

FAIL: Test suite timed out after 3h0m0s

If it is a flake, comment /mlh new-flake Cilium-PR-K8s-GKE so I can create a new GitHub issue to track it.

/test

/test

/test

Job 'Cilium-PR-K8s-GKE' failed and has not been observed before, so may be related to your PR:

K8sDatapathConfig Etcd Check connectivity

FAIL: Pods are not ready in time: timed out waiting for pods with filter  to be ready: 4m0s timeout expired

If it is a flake, comment /mlh new-flake Cilium-PR-K8s-GKE so I can create a new GitHub issue to track it.

/test

test-runtime

Failed to import image: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/303/

Job 'Cilium-PR-K8s-1.20-kernel-4.19' failed and has not been observed before, so may be related to your PR:

K8sDatapathConfig Host firewall With VXLAN

FAIL: Failed to reach 192.168.36.11:80 from testclient-69hxt

If it is a flake, comment /mlh new-flake Cilium-PR-K8s-1.20-kernel-4.19 so I can create a new GitHub issue to track it.

ci-eks

Failed to create cluster: https://github.com/cilium/cilium/runs/3824163280

/test

CI finally passed 🎉 (except for AKS which is disabled) and reviews are in. Marked as ready to merge.