Review

Good direction — aligning the validator runtime with the NRI profile. A few issues to address:

Critical

1. Missing /sys and /proc/sys hostPath mounts on tcpxo-daemon — The runtime removes hostNetwork: true but doesn't add the /sys → /hostsysfs and /proc/sys → /hostprocsysfs volume mounts. Without these, RxDM will detect 0/8 GPUs — the container namespace hides the PCI sysfs tree. These mounts are what makes the NRI profile work without hostNetwork. See demos/workloads/training/gke-nccl-test-tcpxo.yaml in PR fix(gke): update TCPXO to NRI profile without hostNetwork #420 for the working config.
2. RxDM v1.0.21 with incompatible flags — The entrypoint uses --uid= --alsologtostderr but v1.0.21 uses the older fastrak_gpumem_manager binary which reports ERROR: Unknown command line flag 'enforce_kernel_ipv6_support' style errors. We validated v1.0.20 works with simplified args (--num_hops=2 --num_nics 8). Note: v1.0.21 is also marked as deprecated in the container registry. Recommend updating to tcpgpudmarxd-dev:v1.0.20.

Medium

3. Capability names should omit CAP_ prefix — The runtime uses CAP_NET_ADMIN, CAP_NET_BIND_SERVICE but the Kubernetes spec expects names without the CAP_ prefix (NET_ADMIN, NET_BIND_SERVICE, IPC_LOCK). GKE's containerd accepts both, but it's not portable and could fail on stricter runtimes or admission controllers.
4. GPU node pool prerequisite — Worth noting in the PR description or runtime comments that the GPU node pool must not have a gVNIC additional network. A gVNIC takes PCI 0000:06:00.0, displacing one of the 8 GPU NICs and causing a 7/8 GPU detection failure. Root cause details in #381.

Looks good

• NRI device annotation generation with dynamic GPU count + tests
• SSH port change from 2222 to default (correct for non-hostNetwork)
• MPI OOB interface comment update (eth0 is control NIC in pod networking)
• Worker capability IPC_LOCK instead of privileged

1. Missing /sys and /proc/sys mounts

These are already present in the runtime (lines 180-183):

• name: nvtcpxo-sys
  mountPath: /hostsysfs
• name: nvtcpxo-proc-sys
  mountPath: /hostprocsysfs
  Backed by hostPath volumes at the bottom of the spec.

2. RxDM v1.0.21 → v1.0.20

Downgraded to v1.0.20 and removed --uid= --alsologtostderr flags. Reverted entrypoint args to --num_hops=2 --num_nics ${GPU_COUNT_PER_NODE}.

3. Drop CAP_ prefix

Fixed — changed to NET_ADMIN, NET_BIND_SERVICE, IPC_LOCK.

4. gVNIC prerequisite

Added comment in runtime header:

  # Prerequisite: GPU node pool must NOT have a gVNIC additional network.
  # A gVNIC network takes a PCI slot, displacing one GPU NIC (7/8 detection).
  # See https://github.com/NVIDIA/aicr/issues/381 for details.

5. Lint (CI)

Fixed prealloc lint — preallocated lines slice with make([]string, 0, gpuCount+3).