clock skew error with grouped rules #3165
Description
There is a rule download which downloads a FASTQ. The output of that rule is marked temp(). Then, there are three rules which can run in parallel, which all use this FASTQ. After those rules finish, the FASTQ is removed. However, I get this clock skew error ~25% of the time...
Importantly, I don't see this error if I run without the grouping
Here's the last bit of the log
5 of 5 steps (100%) done
Storing output in storage.
+ Exception Group Traceback (most recent call last):
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/cli.py", line 2100, in args_to_api
| dag_api.execute_workflow(
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/api.py", line 595, in execute_workflow
| workflow.execute(
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/workflow.py", line 1264, in execute
| raise e
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/workflow.py", line 1260, in execute
| success = self.scheduler.schedule()
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/scheduler.py", line 194, in schedule
| self._finish_jobs()
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/scheduler.py", line 401, in _finish_jobs
| async_run(postprocess())
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/common/__init__.py", line 89, in async_run
| return asyncio.run(coroutine)
| ^^^^^^^^^^^^^^^^^^^^^^
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/asyncio/runners.py", line 194, in run
| return runner.run(main)
| ^^^^^^^^^^^^^^^^
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/asyncio/runners.py", line 118, in run
| return self._loop.run_until_complete(task)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
| return future.result()
| ^^^^^^^^^^^^^^^
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/scheduler.py", line 363, in postprocess
| await job.postprocess(
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/jobs.py", line 1508, in postprocess
| async with asyncio.TaskGroup() as tg:
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/asyncio/taskgroups.py", line 145, in __aexit__
| raise me from None
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/jobs.py", line 1148, in postprocess
| await self.dag.check_and_touch_output(
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/dag.py", line 730, in check_and_touch_output
| await self.check_output_mtime(job, expanded_output)
| File "/home/mihall/sw/miniforge3/envs/gsize/lib/python3.12/site-packages/snakemake/dag.py", line 752, in check_output_mtime
| raise WorkflowError(
| snakemake_interface_common.exceptions.WorkflowError: Output /data/scratch/projects/punim2009/lrge/paper/results/estimates/genomescope/SRR/216/303/SRR21630359/SRR21630359.size has older modification time (2024-10-22 11:20:32.226016) than input /data/scratch/projects/punim2009/lrge/paper/results/downloads/SRR/216/303/SRR21630359/SRR21630359.fastq.gz (2024-10-22 11:20:32.969082). This could indicate a clock skew problem in your network and would trigger a rerun of this job in the next execution and should therefore be fixed on system level. System time: 2024-10-22 11:20:32.975679 (rule estimate_genomescope, line 156, /data/scratch/projects/punim2009/lrge/paper/workflow/rules/estimate.smk)
Originally posted by @mbhall88 in #3097 (comment)
Metadata
Metadata
Assignees
Labels
No labels