-
Notifications
You must be signed in to change notification settings - Fork 634
Description
Is your feature request related to a problem? Please describe.
If a file may be used after a checkpoint, but is not in the initial DAG, it is deleted once the last rule requiring the file in the original DAG completes. The file then needs to be recreated after the checkpoint. If the pipeline then fails or the DAG recalculated again, the pipleine must start from the beginning because the modification date on that file is newer than the intermediate files. Additionally, the rule creating the temp file must be executed multiple times.
Describe the solution you'd like
Add an option to the temp() function allowing the files to be deleted once the whole pipeline completes, rather than once the file is no longer required by another rule.
Describe alternatives you've considered
One alternative is not use the temp() function, but this is problematic for large files. Another option is to require the file in the last rule to run, but that adds a fictional dependency to the DAG.