RuntimeOptions are not passed to the shim when user namespace is enabled #11480
Description
Context
When BinaryName is set in the containerd runtime options, it is included in the RuntimeOptions struct. However, there are instances where containerd prioritizes TaskOptions over RuntimeOptions if TaskOptions are present. This behavior results in the dropping of all RuntimeOptions, including BinaryName. I addressed one such instance in this pull request, but Netflix has encountered another similar issue, and it's unclear what the behavior should be when both options are present.
The Bug
When user namespaces are enabled, TaskOptions are populated with IoUid and IoGid (code). This leads containerd to favor TaskOptions over RuntimeOptions during the invocation of shimTask.Create (code). This causes containerd to use the default runc, rather than the one specified in the runtime options. If the user does not have a runc in the the $PATH, then they will encounter this error:
Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/k8s.io/main/log.json: no such file or directory): exec: "runc": executable file not found in $PATH
Conversely, when user namespaces are not enabled, TaskOptions are nil, allowing RuntimeOptions to be preferred, hiding this edge case.
Related Bug Reports
Steps to reproduce the issue
- Install
runcsomewhere not in the $PATH - Configure containerd runtime options
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]to use this runcBinaryName = '/custom-path/to/runc - Create a k8s pod with user namespace enabled
Describe the results you received and expected
When the pod is launched, you will see this error in the events:
Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/k8s.io/main/log.json: no such file or directory): exec: "runc": executable file not found in $PATH
My expectation is that containerd will pass the correct runtime options with the custom BinaryName to the runc runtime shim, but it's not, so it's falling back to use the default runc BinaryName runc, and not finding it the $PATH.
What version of containerd are you using?
v2.0.3
Any other relevant information
No response
Show configuration if it is related to CRI plugin.
version = 3
root = '/mnt/containerd'
state = '/run/containerd'
imports = []
[grpc]
address = '/run/containerd/rk-containerd.sock'
[plugins]
[plugins.'io.containerd.cri.v1.images']
discard_unpacked_layers = true
[plugins.'io.containerd.cri.v1.images'.pinned_images]
sandbox = 'registry.us-east-1.streamingtest.titus.netflix.net:7002/eks/pause:3.5'
[plugins.'io.containerd.cri.v1.runtime']
[plugins.'io.containerd.cri.v1.runtime'.containerd]
default_runtime_name = 'runc'
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
runtime_type = 'io.containerd.runc.v2'
runtime_path = '/apps/nflx-containerd/bin/containerd-shim-runc-v2'
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
SystemdCgroup = true
BinaryName = '/apps/titus-kubelet/bin/runc-wrapper'
[plugins.'io.containerd.cri.v1.runtime'.cni]
bin_dir = '/opt/cni/bin'
conf_dir = '/etc/cni/net.d'
[plugins."io.containerd.nri.v1.nri"]
plugin_registration_timeout = "10s"
# We're doing a lot of titus initializations in NRI that need more time to finish, i.e. pulling images/creating containers
plugin_request_timeout = "120s"