Skip to content

Failure invoking expand() or collect() rules.<previous_rule>.output with storage provider plugin  #3027

@jns

Description

@jns

Snakemake version
8.18.0

Describe the bug
Snakemake fails to generate a DAG when using expand or collect on the output of a prior rule in combination with a storage provider.

Logs

WorkflowError in file /home/ubuntu/snaketest/Snakefile, line 28:
Flags ({'storage_object': <snakemake_storage_plugin_fs.StorageObject object at 0x7ebd211710a0>}) in file pattern '.snakemake/storage/fs/home/ubuntu/data/sorted_reads/{sample}.bam' given to expand() are invalid. Flags (e.g. temp(), directory()) have to be applied outside of expand (e.g. 'temp(expand("plots/{sample}.pdf", sample=SAMPLES))').

Minimal example
The following example invokes collect() on the output of a prior rule and fails when either the fs or s3 storage providers are specified.

Example Invocation: snakemake -np --default-storage-provider fs --default-storage-prefix /home/ubuntu/data

SAMPLES=["A", "B"]

rule all:
    input:
        "calls/all.vcf"

rule bwa_map:
    input:
        "data/genome.fa",
        "data/samples/{sample}.fastq"
    output:
        "mapped_reads/{sample}.bam"
    shell:
        "bwa mem {input} | samtools view -Sb - > {output}"

rule samtools_sort:
    input:
        "mapped_reads/{sample}.bam"
    output:
        "sorted_reads/{sample}.bam"
    shell:
        "samtools sort -T sorted_reads/{wildcards.sample} "
        "-O bam {input} > {output}"

rule bcftools_call:
    input:
        fa="data/genome.fa",
        bam=collect(rules.samtools_sort.output, sample=SAMPLES)
    output:
        "calls/all.vcf"
    shell:
        "bcftools mpileup -f {input.fa} {input.bam} | "
        "bcftools call -mv - > {output}"

Additional context
None

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstale

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions