I have a rather complex workflow with 750 samples and roughly 18.000 jobs, at first snakemake runs just fine but then after around 4.000 jobs it suddenly freezes and upon restart it hangs with "Select jobs to execute..." for 24h, after that I terminated it. The initial DAG building takes roughly 2-3 minutes, though.

When I run snakemake (v5.32.0 and v5.32.1) with the --verbose option, I get tons of lines similar to this one:

Cbc0010I After 600 nodes, 304 on tree, -52534.791 best solution, best possible -52538.194 (7.08 seconds

I tried to delete the .snakemake folder in the hope that something went riot there, but that wasn't the case, unfortunately. To me it seems that the CBC MILP Solver somehow does not converge, and it keeps going and going to bring the best and the best possible solution closer together!?

After terminating a (shorter) run, I get this verbose output

Result - User ctrl-cuser ctrl-c

Objective value:                52534.79114334
Upper bound:                    52538.202
Gap:                            -0.00
Enumerated nodes:               186926
Total iterations:               1807277
Time (CPU seconds):             1181.97
Time (Wallclock seconds):       1188.11

I believe it is a bug in the current version, I downgraded Snakemake back to version 5.24, it created then the DAG within 10 minutes and started to run the pipeline. So, apparently there is some bug with the latest version?!

Unfortunately I cannot create a minimal example to reproduce the problem, as it just happens with this dataset on which I run the pipeline (with a smaller dataset it runs just fine)