-
Notifications
You must be signed in to change notification settings - Fork 3.9k
Description
Description
Containerd is shipping with KillMode=process in systemd unit, so that shims won't be killed if containerd stops.
Systemd broadcast SIGTERM during the final stage of shutdown, this is the only chance for shim to terminate gracefully.
It seems that shim v2 does not handle SIGTERM/SIGINT at all, and hangs machine reboot/shutdown for 90s, got killed at last.
Shim v1 handles SIGINT/SIGTERM
containerd/cmd/containerd-shim/main_unix.go
Lines 248 to 261 in c7e4747
| case unix.SIGTERM, unix.SIGINT: | |
| go termOnce.Do(func() { | |
| ctx := context.TODO() | |
| if err := server.Shutdown(ctx); err != nil { | |
| logger.WithError(err).Error("failed to shutdown server") | |
| } | |
| // Ensure our child is dead if any | |
| sv.Kill(ctx, &shimapi.KillRequest{ | |
| Signal: uint32(syscall.SIGKILL), | |
| All: true, | |
| }) | |
| sv.Delete(context.Background(), &ptypes.Empty{}) | |
| close(done) | |
| }) |
Shim v2 registers and ignores SIGINT/SIGTERM
containerd/runtime/v2/shim/shim_unix.go
Lines 81 to 87 in c7e4747
| switch s { | |
| case unix.SIGCHLD: | |
| if err := reaper.Reap(); err != nil { | |
| logger.WithError(err).Error("reap exit status") | |
| } | |
| case unix.SIGPIPE: | |
| } |
See also:
- containerd-shim hangs on reboot/shutdown (live restore + runc v2 runtime) moby/moby#41831
- systemd-shutdown hangs on containerd-shim when k3s-agent running k3s-io/k3s#2400
- SIGTERM doesn't kill containerd-shims #386
Steps to reproduce the issue:
- install docker-ce 20.10.x, enable live-restore
docker run -d k8s.gcr.io/pausesudo reboot
Describe the results you received:
The shutdown/reboot process stuck for 90s, due to containerd-shim.
[ OK ] Reached target Shutdown.
[ OK ] Reached target Final Step.
[ OK ] Finished Reboot.
[ OK ] Reached target Reboot.
[ 214.337805] systemd-shutdown[1]: Waiting for process: containerd-shim
Describe the results you expected:
containerd-shim should not interfere with shutdown/reboot.
What version of containerd are you using:
$ containerd --version
containerd containerd.io 1.4.4 05f951a3781f4f2c1911b05e61c160e9c30eaa8e