Skip to content

cannot set tmpdir default resource on SLURM #2423

@fgvieira

Description

@fgvieira

Snakemake version

v7.32.3

Describe the bug

I want to set up different temp folders depending on where the workflow is being run and the amount of data. For example:

  1. when being run locally with small files, I can use the default /tmp
  2. If I am running it locally with a lot of data, then I need to use a bigger temp folder (e.g. /scratch)
  3. When running it through SLURM, it should use the nodes local temp (e.g. /scratch/$SLURM_JOBID).

The 1. and 2. are ok, since I can use --default-resources tmpdir="\"/scratch\"", but for 3. I cannot get it so use the $SLURM_JOBID.

Minimal example

rule all:
    output:
        "temp.txt"
    shell:
        "echo {resources.tmpdir} > {output}"

Running snakemake --jobs 1 --default-resources tmpdir="\"/tmp/scratch\"" works fine (even though \tmp\scratch is not deleted at the end); the same with snakemake --jobs 1 --default-resources tmpdir="\"/tmp/scratch\"" --slurm.

But when using an env variable:

  • snakemake --jobs 1 --default-resources tmpdir="\"/scratch/$SLURM_JOBID\"" --slurm -> /scratch
  • snakemake --jobs 1 --default-resources tmpdir="\"/scratch/\$SLURM_JOBID\"" --slurm -> /scratch/$SLURM_JOBID (literal)
  • snakemake --jobs 1 --default-resources tmpdir='\"/scratch/$SLURM_JOBID\"' --slurm -> Failed to evaluate default resources value '\\\\"/scratch/$SLURM_JOBID\\\\"'.
  • snakemake --jobs 1 --default-resources tmpdir='/scratch/$SLURM_JOBID' -p --slurm -> Failed to evaluate default resources value '/scratch/$SLURM_JOBID'

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions