Skip to content

Numeric Slurm Partition Names Summed in Snakemake Group Jobs #3992

@andynu

Description

@andynu

Snakemake version
v9.16.3 (also checked HEAD of main — not fixed)

Describe the bug
When a group job has 2+ sibling rules, Snakemake sums all integer-typed resources across siblings — including categorical resources like slurm_partition. If the partition name happens to be numeric (e.g. 20), the summed value (40) is passed to sbatch, which fails.

The root cause is a two-step type-information loss:

  1. Coercion 1 — String serialization in profiles.py:29: f"{key}={val}" flattens all values to strings, so a YAML string "20" becomes indistinguishable from the integer 20.
  2. Coercion 2 — Integer parse in resources.py:1041: val = int(val) coerces any numeric-looking string back to int.

Since the value is now an integer, _is_string_resource() classifies it as numeric, and the intralayer merge at resources.py:351 sums it: 20 + 20 = 40.

Quoting in YAML does not help — Coercion 1 destroys the type, and Coercion 2 re-parses it as int regardless.

This affects any resource whose value is a valid integer and is used in a group job with 2+ siblings (e.g. slurm_partition, slurm_account, slurm_cluster, constraint, or any custom resource with a numeric-looking value). Non-grouped jobs are unaffected (no summing with a single rule), and non-numeric string values are unaffected (int() fails and they stay as strings).

Logs

sbatch: error: invalid partition specified: 40

Minimal example

Snakefile:

rule a:
    input: "data.txt"
    output: "a.txt"
    group: "mygroup"
    shell: "cp {input} {output}"

rule b:
    input: "a.txt"
    output: "b.txt"
    group: "mygroup"
    shell: "cp {input} {output}"

profile/config.yaml:

executor: slurm
default-resources:
  slurm_partition: 20
  tasks: 1

Run with:

snakemake --profile profile/

Expected: sbatch ... -p 20
Actual: sbatch ... -p 40

Additional context
Our slurm cluster uses Ubuntu-version-based partition names (18, 20, 26), which is how we encountered this.

I appreciate that snakemake services many different backends, so simply fixing partition is no good. But perhaps we could consider:

  • a) Preserving type information through the serialization round-trip, so only truly numeric values are summed.
  • b) Providing a mechanism to declare certain resource keys as non-summable (e.g. via a CLI flag like --non-summable-resources or via executor plugin metadata).

I would be happy to explore PRs for either approach if deemed suitable.

Workaround: Use non-numeric partition names (e.g. u20 instead of 20).

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions