Skip to content

allow for directly launching a run, if a --dryrun looked good, without restarting DAG building #3330

@dlaehnemann

Description

@dlaehnemann

Is your feature request related to a problem? Please describe.
For large numbers of jobs (mainly due to large datasets with many intermediate files), the DAG building can run for very long times. We have seen multiple hours in some instances. At the same time, it is very important to do a --dryrun on such large workflows, to avoid unintended reruns of large parts of a workflow, for example after changing some parameters. However, having to do the DAG building twice, once for the --dryrun and once for the actual run, even if the --dryrun looks fine, can be a considerable slow-down of data analyses.

Describe the solution you'd like
I think, the optimal solution would be to turn the --dryrun default into asking for confirmation to actually run at the end of every --dryrun. However, this is not backwards compatible and would for example break existing continuous integration tests that rely on a --dryrun to run through without interaction. Thus, a new --confirm-with-dryrun option for this behavior might be the better solution, because it is backwards compatible and the wording of the option is probably a bit clearer.

Describe alternatives you've considered

Additional context

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

Status

Done

Status

In progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions