This commit fixes #1368.

In addition to verifying that DesiredNumberScheduled and NumberMisscheduled are 0 for a DaemonSet, we also verify that the DaemonSet's nodeSelector does not match any nodes before deciding the DaemonSet is stale and deleting it.

Without this change, it is possible for the NVIDIADriver controller to enter an endless loop of creating and deleting a DaemonSet. This occurs when 1) The NVIDIADriver daemonset does not tolerate a taint present on all nodes matching its configured nodeSelector, AND 2) None of the DaemonSet pods have been scheduled yet.

Testing

Reproduce the bug:

Taint all of your nodes

$ kubectl taint nodes <node-name> foo=bar:NoSchedule

Install GPU Operator and tolerate the taint so that all components can be scheduled. Enable the NVIDIADriver CRD but disable the default CR creation.

$ cat values.yaml 
daemonsets:
  tolerations:
    - key: "foo"
      operator: "Exists"
      effect: "NoSchedule"

operator:
  tolerations:
    - key: "foo"
      operator: "Exists"
      effect: "NoSchedule"

node-feature-discovery:
  worker:
    tolerations:
      - key: "foo"
        operator: "Exists"
        effect: "NoSchedule"
  master:
    tolerations:
      - key: "foo"
        operator: "Exists"
        effect: "NoSchedule"
  gc:
    tolerations:
      - key: "foo"
        operator: "Exists"
        effect: "NoSchedule"

driver:
  nvidiaDriverCRD:
    enabled: true
    deployDefaultCR: false

$ helm install --generate-name nvidia/gpu-operator -n gpu-operator -f values.yaml

Deploy the sample NVIDIADriver CR which does not tolerate the taint:

$ kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/heads/main/config/samples/nvidia_v1alpha1_nvidiadriver.yaml

You will observe a driver DaemonSet repeatedly being created and deleted in the gpu-operator namespace. The gpu-operator logs will also confirm this.

Verify the fix:

Follow the same steps to reproduce the bug and verify that a driver DaemonSet is created corresponding to the NVIDIADriver CR but it does not get repeatedly deleted. The DesiredNumberScheduled will remain 0 for the DaemonSet unless you remove the taint for the node(s) or add the toleration to the NVIDIADriver CR.