Pods with exec liveness/readiness show periodic debug logs in the journal #313
Description
Description
Pods having exec liveness or readiness probes show periodic debug logs in the journal with an error.
Impact
This pollutes the journal.
Environment and steps to reproduce
- Set-up: A Lokomotive cluster running on Flatcar 2605.10.0
- Task: Run a pod with an exec liveness or readiness probe
- Action(s): Access the journald log.
- Error: Messages like the following are seen every few seconds:
Jan 06 09:20:44 acela-controller-0 env[871]: time="2021-01-06T09:20:44Z" level=debug msg="process with id:1588989 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.con>
Jan 06 09:20:44 acela-controller-0 env[871]: time="2021-01-06T09:20:44Z" level=debug msg="process with id:1588964 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.con>
Jan 06 09:20:41 acela-controller-0 env[871]: time="2021-01-06T09:20:41Z" level=debug msg="process with id:1588893 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.con>
Jan 06 09:20:41 acela-controller-0 env[871]: time="2021-01-06T09:20:41Z" level=debug msg="process with id:1588873 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.con>
Jan 06 09:20:40 acela-controller-0 env[871]: time="2021-01-06T09:20:40Z" level=debug msg="process with id:1588861 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.con>
Jan 06 09:20:40 acela-controller-0 env[871]: time="2021-01-06T09:20:40Z" level=debug msg="process with id:1588837 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.con>
Expected behavior
Debug logs shouldn't pollute the journald log.
Additional information
In this containerd PR the log message was demoted to Debug because this happens "for all exec processes that use the v1 shim with Docker because Docker deletes the process once it receives the exit event from containerd."
We set shim_debug = true in /run/torcx/unpack/docker/usr/share/containerd/config.toml, perhaps we can set it to false to disable debug logs.