Description

On our nodes running Containerd 1.7.0 we are experiencing issues with Containerd failing with the following error message:

goroutine 110794110 [running]:
    panic: send on closed channel
    	/go/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:295 +0x1e
    github.com/containerd/ttrpc.(*Client).run(0xc0061eed80)
    	/go/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:333 +0x12e
    github.com/containerd/ttrpc.(*Client).receiveLoop(0xc0061eed80)
    	/go/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/stream.go:71 +0xa6
    github.com/containerd/ttrpc.(*stream).receive(0xc0061eed80?, {0x5634d429a300, 0xc0097f9630}, 0x5634d3f49b80?)
    created by github.com/containerd/ttrpc.NewClient
    	/go/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:95 +0x1f6
    created by github.com/containerd/ttrpc.NewClient

This causes the containerd service to restart and the node goes briefly unready during this time.

The problem seems to relate to either a set of pods that are stuck in a terminating state on the node or a set of pods restarting consistently. When cleaning up pods in these conditions we experienced a decrease in Containerd failing with the above panic message.

Steps to reproduce the issue

Not entirely sure of the trigger, potentially pods stuck in terminating or pods restarting.

Describe the results you received and expected

Containerd should be able to handle closed channels gracefully

What version of containerd are you using?

1.7.0

Any other relevant information

Kernel 6.1.21
OS Ubuntu 22.04
Kubernetes 1.25.6

Show configuration if it is related to CRI plugin.

No response