Problem

CDI spec generation mounts IPC sockets (nvidia-persistenced, nvidia-fabricmanager ...) with the ro (read-only) mount option. This breaks nested container runtimes like enroot/pyxis that need to bind-mount these sockets into containers.

When we try to run slurm job with enroot/pyxis on K8s:

pyxis: imported docker image: nvcr.io/nvidia/cuda:12.4.0-base-ubuntu22.04
error: pyxis: container start failed with error code: 1
error: pyxis: printing enroot log file:
error: pyxis:     nvidia-container-cli: mount error: mount operation failed: 
                  /tmp/enroot/data/user-0/pyxis_16.0/run/nvidia-persistenced/socket: operation not permitted
error: pyxis:     [ERROR] /etc/enroot/hooks.d/98-nvidia.sh exited with return code 1

Root Cause

The ro option is inherited from the default mount options in mounts.go, but IPC sockets should not be read-only. This is inconsistent with libnvidia-container which does not use MS_RDONLY for IPC mounts (reference).

Fix

Define IPC-specific mount options in ipc.go that exclude ro, matching libnvidia-container behavior:

var ipcMountOptions = []string{
    "nosuid",
    "nodev", 
    "rbind",
    "rprivate",
    "noexec",
}

Testing

• Unit test updated and passing
• Manually verified on AWS EKS with host-installed NVIDIA drivers
• Confirmed enroot container starts successfully after fix

Additional Context

I tested two AWS EKS clusters with identical GPU Operator versions:

• When GPU Operator manages drivers, nvidia-persistenced runs inside the driver container. The socket is part of the overlay filesystem and it works fine.
• When using host-installed drivers (e.g., AWS AMI), CDI discovers the host socket and mounts it as tmpfs (ro)

This is critical to support SlurmonK8s (https://github.com/SlinkyProject/slurm-operator) with enroot/pyxis on K8s.