-
Notifications
You must be signed in to change notification settings - Fork 634
Description
Is your feature request related to a problem? Please describe.
For large numbers of jobs (mainly due to large datasets with many intermediate files), the DAG building can run for very long times. We have seen multiple hours in some instances. At the same time, it is very important to do a --dryrun on such large workflows, to avoid unintended reruns of large parts of a workflow, for example after changing some parameters. However, having to do the DAG building twice, once for the --dryrun and once for the actual run, even if the --dryrun looks fine, can be a considerable slow-down of data analyses.
Describe the solution you'd like
I think, the optimal solution would be to turn the --dryrun default into asking for confirmation to actually run at the end of every --dryrun. However, this is not backwards compatible and would for example break existing continuous integration tests that rely on a --dryrun to run through without interaction. Thus, a new --confirm-with-dryrun option for this behavior might be the better solution, because it is backwards compatible and the wording of the option is probably a bit clearer.
Describe alternatives you've considered
Additional context
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Status