Issue with continuously generated input #3696
Description
Snakemake version
9.9.0
Describe the bug
--queue-input-wait-time argument has no effect on the DAG update time (which seems to be set to10s as default).
Logs
Command run:
snakemake --cores 1 --queue-input-wait-time 1
Printed logs:
Assuming unrestricted shared filesystem usage.
host: MININT-BN1SRSN
Building DAG of jobs...
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job count
-------- -------
all 1
generate 1
total 2
Updating jobs with queue input...
Select jobs to execute...
Execute 1 jobs...
[Thu Aug 14 11:07:57 2025]
localrule generate:
output: test0.txt
jobid: 1
reason: Missing output files: test0.txt
wildcards: i=0
resources: tmpdir=C:\Users\DAS02\AppData\Local\Temp
[Thu Aug 14 11:07:57 2025]
Finished jobid: 1 (Rule: generate)
1 of 2 steps (50%) done
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Waiting for queue input...
Updating jobs with queue input...
Select jobs to execute...
Execute 1 jobs...
[Thu Aug 14 11:08:07 2025]
localrule generate:
output: test2.txt
jobid: 4
reason: Missing output files: test2.txt
wildcards: i=2
resources: tmpdir=C:\Users\DAS02\AppData\Local\Temp
[Thu Aug 14 11:08:07 2025]
Finished jobid: 4 (Rule: generate)
2 of 11 steps (18%) done
Select jobs to execute...
Execute 1 jobs...
[Thu Aug 14 11:08:07 2025]
localrule generate:
output: test9.txt
jobid: 11
reason: Missing output files: test9.txt
wildcards: i=9
resources: tmpdir=C:\Users\DAS02\AppData\Local\Temp
[Thu Aug 14 11:08:07 2025]
Finished jobid: 11 (Rule: generate)
3 of 11 steps (27%) done
Select jobs to execute...
Execute 1 jobs...
[Thu Aug 14 11:08:07 2025]
localrule generate:
output: test4.txt
jobid: 6
reason: Missing output files: test4.txt
wildcards: i=4
resources: tmpdir=C:\Users\DAS02\AppData\Local\Temp
[Thu Aug 14 11:08:07 2025]
Finished jobid: 6 (Rule: generate)
4 of 11 steps (36%) done
Select jobs to execute...
Execute 1 jobs...
[Thu Aug 14 11:08:07 2025]
localrule generate:
output: test5.txt
jobid: 7
reason: Missing output files: test5.txt
wildcards: i=5
resources: tmpdir=C:\Users\DAS02\AppData\Local\Temp
[Thu Aug 14 11:08:07 2025]
Finished jobid: 7 (Rule: generate)
5 of 11 steps (45%) done
Select jobs to execute...
Execute 1 jobs...
[Thu Aug 14 11:08:07 2025]
localrule generate:
output: test6.txt
jobid: 8
reason: Missing output files: test6.txt
wildcards: i=6
resources: tmpdir=C:\Users\DAS02\AppData\Local\Temp
[Thu Aug 14 11:08:07 2025]
Finished jobid: 8 (Rule: generate)
6 of 11 steps (55%) done
Select jobs to execute...
Execute 1 jobs...
[Thu Aug 14 11:08:07 2025]
localrule generate:
output: test7.txt
jobid: 9
reason: Missing output files: test7.txt
wildcards: i=7
resources: tmpdir=C:\Users\DAS02\AppData\Local\Temp
[Thu Aug 14 11:08:07 2025]
Finished jobid: 9 (Rule: generate)
7 of 11 steps (64%) done
Select jobs to execute...
Execute 1 jobs...
[Thu Aug 14 11:08:07 2025]
localrule generate:
output: test8.txt
jobid: 10
reason: Missing output files: test8.txt
wildcards: i=8
resources: tmpdir=C:\Users\DAS02\AppData\Local\Temp
[Thu Aug 14 11:08:07 2025]
Finished jobid: 10 (Rule: generate)
8 of 11 steps (73%) done
Select jobs to execute...
Execute 1 jobs...
[Thu Aug 14 11:08:07 2025]
localrule generate:
output: test3.txt
jobid: 5
reason: Missing output files: test3.txt
wildcards: i=3
resources: tmpdir=C:\Users\DAS02\AppData\Local\Temp
[Thu Aug 14 11:08:07 2025]
Finished jobid: 5 (Rule: generate)
9 of 11 steps (82%) done
Select jobs to execute...
Execute 1 jobs...
[Thu Aug 14 11:08:07 2025]
localrule generate:
output: test1.txt
jobid: 3
reason: Missing output files: test1.txt
wildcards: i=1
resources: tmpdir=C:\Users\DAS02\AppData\Local\Temp
[Thu Aug 14 11:08:07 2025]
Finished jobid: 3 (Rule: generate)
10 of 11 steps (91%) done
Select jobs to execute...
Execute 1 jobs...
[Thu Aug 14 11:08:07 2025]
localrule all:
input: test0.txt, test1.txt, test2.txt, test3.txt, test4.txt, test5.txt, test6.txt, test7.txt, test8.txt, test9.txt
jobid: 0
reason: Input files updated by another job: test6.txt, test1.txt, test5.txt, test4.txt, test2.txt, test8.txt, test9.txt, test3.txt, test7.txt
resources: tmpdir=C:\Users\DAS02\AppData\Local\Temp
[Thu Aug 14 11:08:07 2025]
Finished jobid: 0 (Rule: all)
11 of 11 steps (100%) done
Complete log(s): D:\Data\AdemSaglam\testing\tmp\.snakemake\log\2025-08-14T110757.155362.snakemake.log
Minimal example
import threading, queue, time
# the finish sentinel
finish_sentinel = object()
# a synchronized queue for the input files
all_results = queue.Queue()
# a thread that fills the queue with input files to be considered
def update_results():
try:
for i in range(10):
all_results.put(f"test{i}.txt")
time.sleep(1)
all_results.put(finish_sentinel)
all_results.join()
except (KeyboardInterrupt, SystemExit):
return
update_thread = threading.Thread(target=update_results)
update_thread.start()
# target rule which will be continuously updated until the queue is finished
rule all:
input:
from_queue(all_results, finish_sentinel=finish_sentinel),
# job that generates the requested input files
rule generate:
output:
"test{i}.txt"
shell:
"echo {wildcards.i} > {output}"
Additional context
The script above generates 10 txt files. The input is updated every second.
The expected behaviour is that at every second a new txt file is generated.
The observed behaviour is that the first file is generated, the next 9 are generated at the same time at the 10th second (which can be seen in the logs).
If one increases the number range of i to 20, then the following will happen (logs not copied here):
- The first file will be generated in the first second
- The next 9 files will be generated at the 10th second
- The last 10 files will be generated at the 20th second.
This behaviour indicates that the parameter--queue-input-wait-timedoes not overwrite the default behaviour of updating the DAG every 10 seconds.