Skip to content

Temp files get deleted before time #2732

@fgvieira

Description

@fgvieira

Snakemake version

v8.5.3

Describe the bug

Temp files are being deleted when they will still be needed. In the example below, the problem seems to be that temp file A.txt is removed after rule b1, but it is still needed by rule a1.

Logs

Assuming unrestricted shared filesystem usage for local execution.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job      count
-----  -------
a            1
all          1
b1           1
b2           1
total        4

Select jobs to execute...
Execute 1 jobs...

[Wed Mar  6 10:04:25 2024]
localrule a:
    output: A.txt
    jobid: 3
    reason: Forced execution
    wildcards: name=A
    resources: tmpdir=/tmp

[Wed Mar  6 10:04:25 2024]
Finished job 3.
1 of 4 steps (25%) done
Select jobs to execute...
Execute 1 jobs...

[Wed Mar  6 10:04:25 2024]
localrule b1:
    input: A.txt
    output: A.b1.out
    jobid: 2
    reason: Forced execution
    resources: tmpdir=/tmp

[Wed Mar  6 10:04:25 2024]
Finished job 2.
2 of 4 steps (50%) done
Removing temporary output A.txt.
Select jobs to execute...
Execute 1 jobs...

[Wed Mar  6 10:04:25 2024]
localcheckpoint b2:
    input: A.b1.out
    output: A.b2.out
    jobid: 1
    reason: Forced execution
    resources: tmpdir=/tmp
DAG of jobs will be updated after completion.

[Wed Mar  6 10:04:25 2024]
Finished job 1.
3 of 4 steps (75%) done
Select jobs to execute...
Execute 1 jobs...

[Wed Mar  6 10:04:25 2024]
localrule a1:
    input: A.txt
    output: A.a1.out
    jobid: 6
    reason: Forced execution
    wildcards: name=A
    resources: tmpdir=/tmp

Waiting at most 5 seconds for missing files.
[...]
    raise IOError(
OSError: Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
A.txt

Minimal example

rule a:
    output:
        temp("{name}.txt"),
    shell:
        "touch {output}"

rule a1:
    input:
        "{name}.txt",
    output:
        temp("{name}.a1.out"),
    shell:
        "touch {output}"

rule b1:
    input:
        "A.txt",
    output:
        temp("A.b1.out"),
    shell:
        "touch {output}"

checkpoint b2:
    input:
        "A.b1.out"
    output:
        "A.b2.out"
    shell:
        "echo A > {output}"

def in_func1(w):
    file = checkpoints.b2.get().output[0]
    return open(file, "r").read().strip()

rule all:
    input:
        lambda w: expand(rules.a1.output, name=in_func1(w)),
    default_target: True

Additional context

Maybe related to #2464

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions