-
Notifications
You must be signed in to change notification settings - Fork 634
Description
Snakemake version
v9.16.3 (also checked HEAD of main — not fixed)
Describe the bug
When a group job has 2+ sibling rules, Snakemake sums all integer-typed resources across siblings — including categorical resources like slurm_partition. If the partition name happens to be numeric (e.g. 20), the summed value (40) is passed to sbatch, which fails.
The root cause is a two-step type-information loss:
- Coercion 1 — String serialization in
profiles.py:29:f"{key}={val}"flattens all values to strings, so a YAML string"20"becomes indistinguishable from the integer20. - Coercion 2 — Integer parse in
resources.py:1041:val = int(val)coerces any numeric-looking string back toint.
Since the value is now an integer, _is_string_resource() classifies it as numeric, and the intralayer merge at resources.py:351 sums it: 20 + 20 = 40.
Quoting in YAML does not help — Coercion 1 destroys the type, and Coercion 2 re-parses it as int regardless.
This affects any resource whose value is a valid integer and is used in a group job with 2+ siblings (e.g. slurm_partition, slurm_account, slurm_cluster, constraint, or any custom resource with a numeric-looking value). Non-grouped jobs are unaffected (no summing with a single rule), and non-numeric string values are unaffected (int() fails and they stay as strings).
Logs
sbatch: error: invalid partition specified: 40
Minimal example
Snakefile:
rule a:
input: "data.txt"
output: "a.txt"
group: "mygroup"
shell: "cp {input} {output}"
rule b:
input: "a.txt"
output: "b.txt"
group: "mygroup"
shell: "cp {input} {output}"profile/config.yaml:
executor: slurm
default-resources:
slurm_partition: 20
tasks: 1Run with:
snakemake --profile profile/Expected: sbatch ... -p 20
Actual: sbatch ... -p 40
Additional context
Our slurm cluster uses Ubuntu-version-based partition names (18, 20, 26), which is how we encountered this.
I appreciate that snakemake services many different backends, so simply fixing partition is no good. But perhaps we could consider:
- a) Preserving type information through the serialization round-trip, so only truly numeric values are summed.
- b) Providing a mechanism to declare certain resource keys as non-summable (e.g. via a CLI flag like
--non-summable-resourcesor via executor plugin metadata).
I would be happy to explore PRs for either approach if deemed suitable.
Workaround: Use non-numeric partition names (e.g. u20 instead of 20).
Metadata
Metadata
Assignees
Labels
Type
Projects
Status