Skip to content

Slow DAG building and job completion due to large file checksumming #2949

@laf070810

Description

@laf070810

I have several TB data in my DAG, and file checksumming happens in both DAG building and jobs completion stage, causing the DAG building and job completion during running be extremely slow (more than 30mins or even 1h). See the diff in https://github.com/laf070810/snakemake for the affected parts.

I suggest adding some file size checking to avoid large file checksumming, or adding a config option to completely disable file checksumming.

Related to #2920

Metadata

Metadata

Labels

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions