Skip to content

Allow specification of when temp files are deleted #2464

@BEFH

Description

@BEFH

Is your feature request related to a problem? Please describe.
If a file may be used after a checkpoint, but is not in the initial DAG, it is deleted once the last rule requiring the file in the original DAG completes. The file then needs to be recreated after the checkpoint. If the pipeline then fails or the DAG recalculated again, the pipleine must start from the beginning because the modification date on that file is newer than the intermediate files. Additionally, the rule creating the temp file must be executed multiple times.

Describe the solution you'd like
Add an option to the temp() function allowing the files to be deleted once the whole pipeline completes, rather than once the file is no longer required by another rule.

Describe alternatives you've considered
One alternative is not use the temp() function, but this is problematic for large files. Another option is to require the file in the last rule to run, but that adds a fictional dependency to the DAG.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions