containerd fails on start: "failed to get metadata for stored sandbox" #10848
Description
Description
We observed one of our nodes failing to properly boot. The error is:
Oct 17 13:46:12 containerd[40100]: time="2024-10-17T13:46:12.015594695Z" level=info msg="Start recovering state"
Oct 17 13:46:12 containerd[40100]: time="2024-10-17T13:46:12.015677461Z" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Oct 17 13:46:12 containerd[40100]: time="2024-10-17T13:46:12.016081023Z" level=info msg=serving... address=/run/containerd/containerd.sock
Oct 17 13:46:12 containerd[40100]: time="2024-10-17T13:46:12.062483416Z" level=fatal msg="Failed to run CRI service" error="failed to recover state: failed to get metadata for stored sandbox \"4ea7054a2b00694197040ac0c59ac2552624824adf3df31c292243992c72bfc9\": not found"
Workaround: rm -rf /var/lib/containerd/ then restarting containerd. Note: I previously suggested just removing io.containerd.metadata.v1.bolt/meta.db but removed this based on Thomas's comment below
Steps to reproduce the issue
We've not yet reproduced the issue, it's a one-off so far. I've seen two other references: siderolabs/talos#9496 (from 5 days ago) and #10236 (from May 16th).
I'll update here if I see this again or if I can reproduce it.
Describe the results you received and expected
I expect that containerd is able to recover state without failing, ignoring/deleting any invalid entries in the metadata.
What version of containerd are you using?
2.0.0-rc.4
Any other relevant information
# runc --version
runc version 1.1.14
commit: 2c9f5602f0ba3d9da1c2596322dfc4e156844890
spec: 1.0.2-dev
go: go1.23.0
libseccomp: 2.5.5
# uname -a
Linux hostname 6.6.42-rt37 #1 SMP PREEMPT_RT Thu Sep 12 11:03:43 CEST 2024 x86_64 AMD Ryzen 5 5600X 6-Core Processor AuthenticAMD GNU/Linux
Show configuration if it is related to CRI plugin.
No response