To prevent OOMKiller killing random processes on the node it is possible to define MemSpecLimit which reserves some of the memory for the system and limit job memory below what is available on the node.

This example reserves 1024MB of RAM for system on the node:

      - id: debug_nodeset
        source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
        use: [network]
        settings:
          disk_size_gb: 30
          machine_type: n2d-standard-2
          node_conf:
            MemSpecLimit: 1024

But with such definition, running job fails with:

$ srun -p debug hostname
srun: error: Unable to allocate resources: Requested node configuration is not available

Though running job with:

$ srun -p debug --mem 100 hostname

Succeeds, as it requests less memory.

This change subtracts reserved memory from total memory available on the instance before calculating DefMemPerCPU which results in default memory claim within available memory.

This is most visible on 1 CPU nodes, but with larger nodes, at least one CPU may not be available for scheduling due to this.

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #