A number of race conditions exist when pods are terminated early in their lifecycle because components in the kubelet need to know "no running containers" or "containers can't be started from now on" but were relying on outdated state (they don't know whether the pod is setting up, tearing down, or can never be started again).

Only the pod worker knows whether containers are being started for a given pod, which is required to know when a pod is "terminated" (no running containers, none coming). Move that responsibility and podKiller function into the pod workers, and have everything that was killing the pod go into the UpdatePod loop. Have pod workers remain running until the pod is ready to be deleted in etcd (PodResourcesAreReclaimed would return true). Split syncPod into three phases - setup, terminate containers, and cleanup pod - and have transitions between those methods be visible to other components. After this change, to kill a pod you tell the pod worker to UpdatePod({UpdateType: SyncPodKill, Pod: pod}).

Several places in the kubelet were incorrect about whether they were handling terminating (should stop running, might have
containers) or terminated (no running containers) pods. The pod worker exposes methods that allow other loops to know when to set up or tear down resources based on the state of the pod - these methods remove the possibility of race conditions by ensuring a single component is responsible for knowing each pod's allowed state and other components
simply delegate to checking whether they are in the window by UID.

In addition, removing containers now no longer blocks final pod deletion in the API server and are handled as background cleanup.

See https://docs.google.com/document/d/1DvAmqp9CV8i4_zYvNdDWZW-V0FMk1NbZ8BAOvLRD3Ds/edit# for details

TODO:

• Implement context cancellation on syncPod and terminatingPod when necessary
• Add an e2e test like "pod submit and delete" with init containers Add init container pod deletion test #103128
• Add an e2e test that verifies that logs remain and are accessible until the pod object is deleted (set a 15s grace period, write a log message every second)
• Thoroughly review that no regressions in cleanup are present - setup in loops should use the right conditions, teardown should also use them
• Improve pod sync latency by having pod worker track the latency

What type of PR is this?

/kind bug
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fix a number of race conditions in the kubelet when pods are starting up or shutting down that might cause pods to take a long time to shut down.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: