-
Notifications
You must be signed in to change notification settings - Fork 634
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Snakemake version: 8.24.0 and 8.25.2
Describe the bug
snakemake --cleanup-metadata <path> crashes with an AssertionError in snakemake/persistence.py", line 702, in _record_path
Logs
$ snakemake --cleanup-metadata foobar.txt
Traceback (most recent call last):
File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/cli.py", line 2128, in args_to_api
dag_api.cleanup_metadata(args.cleanup_metadata)
File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/api.py", line 604, in _handle_no_exec
return method(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/api.py", line 677, in cleanup_metadata
self.workflow_api._workflow.cleanup_metadata(paths)
File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/workflow.py", line 841, in cleanup_metadata
success = self.persistence.cleanup_metadata(path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/persistence.py", line 228, in cleanup_metadata
return self._delete_record(self._metadata_path, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/persistence.py", line 644, in _delete_record
recpath = self._record_path(subject, id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/persistence.py", line 702, in _record_path
assert isinstance(id, _IOFile)
^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
Minimal example
- Test setup:
$ mkvirtualenv test
$ pip install snakemake
$ cat Snakefile
rule foobar:
input:
"foo.txt",
"bar.txt"
output:
"foobar.txt"
shell:
"sleep 100 && cat {input} > {output}"
- Run the workflow and interrupt the run (using
<Ctrl-C>):
$ snakemake
Assuming unrestricted shared filesystem usage.
host: pc1011901129
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 6
Rules claiming more threads will be scaled down.
Job stats:
job count
------ -------
foobar 1
total 1
Select jobs to execute...
Execute 1 jobs...
[Fri Nov 8 10:46:59 2024]
localrule foobar:
input: foo.txt, bar.txt
output: foobar.txt
jobid: 0
reason: Missing output files: foobar.txt
resources: tmpdir=/tmp
^CTerminating processes on user request, this might take some time.
[Fri Nov 8 10:47:00 2024]
Error in rule foobar:
jobid: 0
input: foo.txt, bar.txt
output: foobar.txt
shell:
sleep 100 && cat foo.txt bar.txt > foobar.txt
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Complete log: .snakemake/log/2024-11-08T104659.843222.snakemake.log
WorkflowError:
At least one job did not complete successfully.
- Generate the output yourself:
$ cat foo.txt bar.txt > foobar.txt
- Try to run Snakemake again:
$ snakemake
Assuming unrestricted shared filesystem usage.
host: pc1011901129
Building DAG of jobs...
IncompleteFilesException:
The files below seem to be incomplete. If you are sure that certain files are not incomplete, mark them as complete with
snakemake --cleanup-metadata <filenames>
To re-generate the files rerun your command with the --rerun-incomplete flag.
Incomplete files:
foobar.txt
- Try to clean up the metadata:
$ snakemake --cleanup-metadata foobar.txt
- This will produce the error in the logs above.
Additional context
-
I faced this issue when I tried to copy over some pre-computed data from an older workflow that produces the same initial files after realizing re-computing the files would take too long.
-
One can bypass the problem by removing the files in
.snakemake/incomplete, see Error when cleaning up metadata when using workdir in snakefile #2933.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Type
Projects
Status
Done