Skip to content

snakemake --cleanup-metadata appears broken in 8.24.0 and 8.25.2 #3193

@daknuett

Description

@daknuett

Snakemake version: 8.24.0 and 8.25.2

Describe the bug
snakemake --cleanup-metadata <path> crashes with an AssertionError in snakemake/persistence.py", line 702, in _record_path

Logs

$ snakemake --cleanup-metadata foobar.txt
Traceback (most recent call last):

  File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/cli.py", line 2128, in args_to_api
    dag_api.cleanup_metadata(args.cleanup_metadata)

  File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/api.py", line 604, in _handle_no_exec
    return method(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/api.py", line 677, in cleanup_metadata
    self.workflow_api._workflow.cleanup_metadata(paths)

  File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/workflow.py", line 841, in cleanup_metadata
    success = self.persistence.cleanup_metadata(path)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/persistence.py", line 228, in cleanup_metadata
    return self._delete_record(self._metadata_path, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/persistence.py", line 644, in _delete_record
    recpath = self._record_path(subject, id)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "$HOME/.virtualenvs/test/lib/python3.12/site-packages/snakemake/persistence.py", line 702, in _record_path
    assert isinstance(id, _IOFile)
           ^^^^^^^^^^^^^^^^^^^^^^^

AssertionError

Minimal example

  • Test setup:
$ mkvirtualenv test
$ pip install snakemake
$ cat Snakefile 
rule foobar:
    input:
        "foo.txt",
        "bar.txt"
    output:
        "foobar.txt"
    shell:
        "sleep 100 && cat {input} > {output}"

  • Run the workflow and interrupt the run (using <Ctrl-C>):
$ snakemake
Assuming unrestricted shared filesystem usage.
host: pc1011901129
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 6
Rules claiming more threads will be scaled down.
Job stats:
job       count
------  -------
foobar        1
total         1

Select jobs to execute...
Execute 1 jobs...

[Fri Nov  8 10:46:59 2024]
localrule foobar:
    input: foo.txt, bar.txt
    output: foobar.txt
    jobid: 0
    reason: Missing output files: foobar.txt
    resources: tmpdir=/tmp

^CTerminating processes on user request, this might take some time.
[Fri Nov  8 10:47:00 2024]
Error in rule foobar:
    jobid: 0
    input: foo.txt, bar.txt
    output: foobar.txt
    shell:
        sleep 100 && cat foo.txt bar.txt > foobar.txt
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Complete log: .snakemake/log/2024-11-08T104659.843222.snakemake.log
WorkflowError:
At least one job did not complete successfully.
  • Generate the output yourself:
 $  cat foo.txt bar.txt > foobar.txt
  • Try to run Snakemake again:
$ snakemake
Assuming unrestricted shared filesystem usage.
host: pc1011901129
Building DAG of jobs...
IncompleteFilesException:
The files below seem to be incomplete. If you are sure that certain files are not incomplete, mark them as complete with

    snakemake --cleanup-metadata <filenames>

To re-generate the files rerun your command with the --rerun-incomplete flag.
Incomplete files:
foobar.txt
  • Try to clean up the metadata:
$ snakemake --cleanup-metadata foobar.txt
  • This will produce the error in the logs above.

Additional context

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions