Description

There is a possibility to enter into an error state during RemoveContainer() where an entry for a container exists in container store but not in containerd. When this happens, the RemoveContainer() call from here returns an error and kubelet will continue to retry until it gets a nil error.
To avoid this, we can take containerd's view of containers as source of truth and if an entry for a container doesn't exist in there, then we can attempt to remove the entry for this container from the container store. This will make container store's entries to be consistent with containerd's view.

Steps to reproduce the issue

Describe the results you received and expected

When containerd gets into this state, POD is stuck in TERMINATING state. Due to inconsistency between container and criservice's container store, there is an error returned from https://github.com/containerd/containerd/blob/main/pkg/cri/server/container_remove.go#L48 which causes kubelet to keep retrying RemoveContainer() call until it receives a nil error

What version of containerd are you using?

1.6.14

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response