This PR include using the existing NFS server on the controller to distribute slurm key to nodeset running on GKE. If this is accepted, we can close #4562 as it's no longer needed.

/gcbrun

/gcbrun

I deployed the blueprint from this PR. After a few minutes, gke based nodes went to DOWN state. The slurmd log shows:

CPUs=8 Boards=1 Sockets=1 Cores=8 Threads=1 Memory=64309 TmpDisk=96515 Uptime=67206 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)
[2025-09-25T08:43:49.012] error: Security violation, ping RPC from uid 981
[2025-09-25T08:43:49.012] error: Do you have SlurmUser configured as uid 981?

If we have nodes going down for such reason, than we need our own build of slurmd container with SlurmUser=401

I deployed the blueprint from this PR. After a few minutes, gke based nodes went to DOWN state. The slurmd log shows:

CPUs=8 Boards=1 Sockets=1 Cores=8 Threads=1 Memory=64309 TmpDisk=96515 Uptime=67206 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)
[2025-09-25T08:43:49.012] error: Security violation, ping RPC from uid 981
[2025-09-25T08:43:49.012] error: Do you have SlurmUser configured as uid 981?

If we have nodes going down for such reason, than we need our own build of slurmd container with SlurmUser=401

This was caused by uid mismatch in the public slinky image. A custom-built image with the correct UID is required to fix this. I will open a follow-up PR with instructions for building this image.

From time to time, deployment is failing, pods are stuck in init stage with message:

MountVolume.MountDevice failed for volume "slurm-key-pv" : rpc error: code = DeadlineExceeded desc = context deadline exceeded 
<br class="Apple-interchange-newline">

Also, sometimes ./gcluster deploy has to be run twice, since for the first time below error message appears:

for: "/tmp/608228131kubectl_manifest.yaml": error when patching "/tmp/608228131kubectl_manifest.yaml": PersistentVolume "slurm-key-pv" is invalid: spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
  core.PersistentVolumeSource{

/gcbrun

/gcbrun