1. Quick Debug Information

• OS/Version(e.g. RHEL8.6, Ubuntu22.04): RHEL 8.10
• Kernel Version: 4.18.0-553.el8_10.x86_64
• Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): CRI-O
• K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): K8s
• GPU Operator Version: gpu-operator-v24-3-0, driver version: 535.183.01

2. Issue or feature description

We are trying to seperate nfd out of gpu operator namespace and deploy seperately. We installed GPU operator with precompiled as false and when there is a restart of nfd pod, the driver daemon set is terminated and restarted. When this happens the node label nvidia.com/gpu-driver-upgrade-state is still on upgrade done, due to this the pods are not evicted on the node from which the driver must be installed and the driver stays in init crash back loop off waiting for pods to be evicted.

I tried setting various env parameters like ENABLE_AUTO_DRAIN , DRAIN_USE_FORCE on k8s-driver-manager but no luck.

nfd version: 0.15.4
driver version: 535.183.01

3. Steps to reproduce the issue

1. Install GPU operator with useprecompiled as false
2. Restart nfd of a node
3. Driver daemon set is stuck in init loop off