Skip to content

fix: reduce number of tests leaving temporary files behind#4033

Merged
johanneskoester merged 9 commits intosnakemake:mainfrom
edbennett:reduce-temporary-litter
Mar 11, 2026
Merged

fix: reduce number of tests leaving temporary files behind#4033
johanneskoester merged 9 commits intosnakemake:mainfrom
edbennett:reduce-temporary-litter

Conversation

@edbennett
Copy link
Copy Markdown
Contributor

@edbennett edbennett commented Mar 10, 2026

Currently, many tests in the default test suite fail to clean up after themselves. On CI this doesn't really matter as the VM will be destroyed once the run completes, but it's mildly annoying for local development.

This PR currently cleans up the low-hanging fruit: either removing unneeded cleanup=False options presumably left over from debugging, or adding explicit cleanup.

A few cases are harder to deal with: the following tests cause the call to run() to raise an exception:

  • test_github_issue1158
  • test_ambiguousruleexception
  • test_wildcard_annotatedstrings

In this case, run() never reaches the cleanup code, and the calling function doesn't (and can't) know the temporary path used to be able to clean up.

I can see a few options on how to work around this:

  1. Ugly hack: make an extra call to run() but requesting a no-op target to generate the temporary directory, then pass this in to the current call to run() to perform the test; after generating the error, then remove the temporary directory.
  2. Move the logic in run() that creates the temporary directory to a separate helper function, and call this explicitly from the error-producing tests, then proceed as in (1) above.
  3. Refactor the temporary directory functionality to be a class that manages the temporary directory, and deletes it when the instance is destroyed—this could be just directly using tempfile.TemporaryDirectory, or reimplementing similar functionality.

@johanneskoester Do you have a preference (one of these three, or something else entirely)?

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

In principle one could add a test case that tests do not leave litter, but having tests for your tests seems a little excessive.

Summary by CodeRabbit

  • Tests
    • Added a reusable temp-directory helper to support both in-place and isolated test modes.
    • Standardized temp-dir resolution and result-path handling across tests, with clearer behavior when running without a temp dir.
    • Restored and unified cleanup behavior (including platform-aware handling), ensuring temporary artifacts are removed on success and failure.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 10, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a public helper prepare_tmpdir to tests, refactors run/tmpdir/no_tmpdir handling and Snakefile/path resolution in tests/common.py, and updates multiple tests to use prepare_tmpdir or capture tmpdir and perform explicit, platform-aware temporary-directory cleanup.

Changes

Cohort / File(s) Summary
Test harness / tmpdir helper
tests/common.py
Adds prepare_tmpdir(source_path, snakefile_parent_dirname=None, dest_path=None); reorganizes tmpdir vs. no_tmpdir control flow, Snakefile resolution, print_tree/path references, target file path handling, Windows-specific path logic, and preserves cleanup semantics with early return for no_tmpdir.
Tests updated to use tmpdir helper and explicit cleanup
tests/tests.py
Imports prepare_tmpdir; updates tests to call prepare_tmpdir(...) or capture tmpdir from run(...); reintroduces explicit shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS) cleanup in many tests; removes some no_tmpdir=True usages and adds platform-aware chmod/removal where needed.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 7.69% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: reducing temporary files left behind by tests through cleanup improvements.
Description check ✅ Passed The description clearly explains the problem, the approach taken, unresolved cases, and explicitly addresses all QC checklist items with justifications.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
tests/common.py (2)

516-520: ⚠️ Potential issue | 🔴 Critical

Bug: tmpdir is undefined when no_tmpdir=True and execution fails.

When no_tmpdir=True, the tmpdir variable is never assigned. If the workflow execution fails (success=False), line 517 calls print_tree(tmpdir, ...) which will raise a NameError or pass None causing unexpected behavior.

🐛 Proposed fix
     else:
         if not success:
             if snakemake_api is not None and exception is not None:
                 snakemake_api.print_exception(exception)
-            print("Workdir:")
-            print_tree(tmpdir, exclude=".snakemake/conda")
+            if not no_tmpdir:
+                print("Workdir:")
+                print_tree(tmpdir, exclude=".snakemake/conda")
             if exception is not None:
                 raise exception
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/common.py` around lines 516 - 520, The code calls print_tree(tmpdir,
...) unconditionally but tmpdir is never assigned when no_tmpdir=True, causing a
NameError on failure; update the function to initialize tmpdir = None at the top
(or ensure it's always assigned), and guard the print_tree call with a check
like "if tmpdir is not None" or "if not no_tmpdir and tmpdir is not None" before
printing, keeping the existing exception handling and assert logic intact;
reference variables/functions: tmpdir, no_tmpdir, print_tree, exception,
success.

529-530: ⚠️ Potential issue | 🔴 Critical

Bug: tmpdir undefined in result checking when no_tmpdir=True.

When no_tmpdir=True, the tmpdir variable is not assigned, but lines 529-530 use it to construct targetfile. If check_results=True (which is the default when shouldfail=False), this will fail with a NameError.

The result checking should use the original path when no_tmpdir=True.

🐛 Proposed fix
     if check_results:
+        workdir = path if no_tmpdir else tmpdir
         for resultfile in get_expected_files(results_dir):
             if resultfile in [".gitignore", ".gitkeep"] or not os.path.isfile(
                 os.path.join(results_dir, resultfile)
             ):
                 # this means tests cannot use directories as output files
                 continue
-            targetfile = join(tmpdir, resultfile)
+            targetfile = join(workdir, resultfile)
             expectedfile = join(results_dir, resultfile)

And similarly update line 533 and 536 to use workdir instead of tmpdir.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/common.py` around lines 529 - 530, The code builds targetfile and
expectedfile using tmpdir even when no_tmpdir=True (so tmpdir is undefined);
change the logic in the result-checking block that computes
targetfile/expectedfile (and the other uses of tmpdir around the same area) to
use workdir (or the original path variable) when no_tmpdir is true — i.e.,
compute targetfile = join(workdir, resultfile) (or join(path, resultfile)
depending on how workdir/path are used earlier) and leave expectedfile =
join(results_dir, resultfile) as-is; update the other occurrences mentioned (the
lines referenced near the targetfile/expectedfile computation and the later
checks that currently reference tmpdir) to use workdir/path so no NameError
occurs when no_tmpdir=True.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@tests/common.py`:
- Around line 516-520: The code calls print_tree(tmpdir, ...) unconditionally
but tmpdir is never assigned when no_tmpdir=True, causing a NameError on
failure; update the function to initialize tmpdir = None at the top (or ensure
it's always assigned), and guard the print_tree call with a check like "if
tmpdir is not None" or "if not no_tmpdir and tmpdir is not None" before
printing, keeping the existing exception handling and assert logic intact;
reference variables/functions: tmpdir, no_tmpdir, print_tree, exception,
success.
- Around line 529-530: The code builds targetfile and expectedfile using tmpdir
even when no_tmpdir=True (so tmpdir is undefined); change the logic in the
result-checking block that computes targetfile/expectedfile (and the other uses
of tmpdir around the same area) to use workdir (or the original path variable)
when no_tmpdir is true — i.e., compute targetfile = join(workdir, resultfile)
(or join(path, resultfile) depending on how workdir/path are used earlier) and
leave expectedfile = join(results_dir, resultfile) as-is; update the other
occurrences mentioned (the lines referenced near the targetfile/expectedfile
computation and the later checks that currently reference tmpdir) to use
workdir/path so no NameError occurs when no_tmpdir=True.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ffee7b94-c34a-4846-8f66-0fe2969025e5

📥 Commits

Reviewing files that changed from the base of the PR and between b88171c and dbd8b5e.

📒 Files selected for processing (2)
  • tests/common.py
  • tests/tests.py

@johanneskoester
Copy link
Copy Markdown
Contributor

I can see a few options on how to work around this:

1. Ugly hack: make an extra call to `run()` but requesting a no-op target to generate the temporary directory, then pass this in to the current call to `run()` to perform the test; after generating the error, then remove the temporary directory.

2. Move the logic in `run()` that creates the temporary directory to a separate helper function, and call this explicitly from the error-producing tests, then proceed as in (1) above.

3. Refactor the temporary directory functionality to be a class that manages the temporary directory, and deletes it when the instance is destroyed—this could be just directly using `tempfile.TemporaryDirectory`, or reimplementing similar functionality.

I think best is to create the tempdir outside of run in these cases, using the tempfile.TemporaryDirectory context manager, and pass it to run via run(..., tmpdir=...).

Copy link
Copy Markdown
Contributor

@johanneskoester johanneskoester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Just see my comment above about using the TemporaryDirectory context manager in case of expected exceptions.

@edbennett edbennett moved this to In progress in Snakemake Hackathon 2026 Mar 11, 2026
…r by default

- factor out test directory setup into helper function
- add ability to pass in an existing empty directory
- use TemporaryDirectory in cases that xfail due to a decorator
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/common.py`:
- Around line 145-154: Replace the use of the private
tempfile._get_candidate_names() + os.mkdir split with a single atomic
tempfile.mkdtemp() call: when dest_path is falsy, ensure
snakefile_parent_dirname is set (as currently done), then call tempfile.mkdtemp
with a prefix like f"snakemake-{snakefile_parent_dirname}-" and
dir=tempfile.gettempdir() to create and return the directory atomically (assign
result to dest_path) and remove the separate os.mkdir call; update references to
tmpdir accordingly.

In `@tests/tests.py`:
- Around line 2890-2897: Replace the try/except that only removes tmpdir on
AmbiguousRuleException with a try/finally around the call to run(path,
tmpdir=tmpdir) so that shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS) is always
executed; keep the except AmbiguousRuleException branch only if you need to
return early (or convert it into a conditional after run), and apply the same
change to the analogous block around the other test at the region covering lines
2931-2935 (i.e., ensure the run(path, tmpdir=tmpdir) invocation is wrapped in
try/finally and tmpdir is removed in the finally block).
- Around line 1785-1792: The test creates a temporary dir (tmpdir = run(...))
then does shell(...) and shutil.rmtree(tmpdir) but if shell(...) raises the
rmtree is skipped; wrap the post-run operations in a try/finally: after tmpdir
is assigned call shell(...) in the try block and ensure shutil.rmtree(tmpdir) is
executed in the finally block (guarding existence/permission errors as needed).
Update the test surrounding symbols tmpdir, run(...), shell(...), and
shutil.rmtree to perform cleanup reliably even on failures.
- Around line 2037-2040: The TemporaryDirectory usage is wrong: change the
prefix to use the resolved Path variable (use path.name, not dpath.name), call
prepare_tmpdir with the temporary directory returned (prepare_tmpdir(tmpdir)
instead of prepare_tmpdir(path) so you prepare the same folder you pass to run),
and pass tmpdir (a str) directly to run as tmpdir=tmpdir (do not use
tmpdir.name). Update the lines referencing dpath, prepare_tmpdir, and run
accordingly.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1ca609b0-93e0-42db-b2fb-132aec8a0658

📥 Commits

Reviewing files that changed from the base of the PR and between 1745c58 and cb1c401.

📒 Files selected for processing (2)
  • tests/common.py
  • tests/tests.py

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tests/tests.py (1)

470-491: ⚠️ Potential issue | 🟠 Major

Move these new manual cleanups into finally.

The added rmtree(...) calls still only run on the happy path. If any intermediate run(...), os.utime(...), or assertion fails, the tmpdir is left behind anyway, which is exactly the litter this PR is trying to remove.

Also applies to: 2563-2568, 2600-2607

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/tests.py` around lines 470 - 491, The test test_script_python currently
calls shutil.rmtree(tmpdir, ...) only on the happy path; wrap the setup and all
intermediate operations (the run(...) calls, os.utime(...), and assertions) in a
try/finally and move the final cleanup call shutil.rmtree(tmpdir,
ignore_errors=ON_WINDOWS) into the finally block so the temporary directory is
removed even if a run, os.utime, or assertion raises; apply the same change
pattern to the other test blocks noted (the similar rmtree calls around lines
2563-2568 and 2600-2607) and ensure tmpdir is defined before the try so finally
can always reference it.
♻️ Duplicate comments (3)
tests/tests.py (3)

2037-2040: ⚠️ Potential issue | 🔴 Critical

Use path.name for the TemporaryDirectory prefix.

dpath is the imported helper function, so dpath.name will raise AttributeError before this test reaches run(). This should use the resolved Path from Line 2037 instead.

🔧 Proposed fix
-    with tempfile.TemporaryDirectory(prefix=f"snakemake-{dpath.name}") as tmpdir:
+    with tempfile.TemporaryDirectory(prefix=f"snakemake-{path.name}-") as tmpdir:
#!/bin/bash
python - <<'PY'
from pathlib import Path

def dpath(arg):
    return Path(arg)

path = dpath("test_github_issue1158")
print("path.name:", path.name)
print("hasattr(dpath, 'name'):", hasattr(dpath, "name"))
PY
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/tests.py` around lines 2037 - 2040, The TemporaryDirectory prefix is
incorrectly using the helper function name dpath (which has no .name) causing
AttributeError; change the prefix to use the resolved Path variable path (i.e.,
use path.name) in the TemporaryDirectory call so the prefix becomes
f"snakemake-{path.name}"; update the line that constructs TemporaryDirectory
(where TemporaryDirectory(prefix=f"snakemake-{dpath.name}") is used) and leave
prepare_tmpdir(path, dest_path=tmpdir) and run(path, ...) unchanged.

1785-1792: ⚠️ Potential issue | 🟠 Major

Put the post-run cleanup in finally.

If Line 1791 fails, Line 1792 never runs and this test still leaves litter behind.

🔧 Proposed fix
     tmpdir = run(
         dpath("test_github_issue640"),
         targets=["Output/FileWithRights"],
         executor="dryrun",
         cleanup=False,
     )
-    shell(f"chmod -R u+rwX {tmpdir}")
-    shutil.rmtree(tmpdir)
+    try:
+        shell(f"chmod -R u+rwX {tmpdir}")
+    finally:
+        shutil.rmtree(tmpdir)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/tests.py` around lines 1785 - 1792, The test currently assigns tmpdir
from run(...), runs shell("chmod ...") and then calls shutil.rmtree(tmpdir) but
if the test fails before the rmtree call the temp directory is left behind; wrap
the run and shell calls in a try/finally and move the shutil.rmtree(tmpdir) into
the finally block so removal always runs; update the block surrounding the
tmpdir variable (the run(...) call and the subsequent shell(...) chmod) to use
try: ... finally: shutil.rmtree(tmpdir) and guard removal for a non-None tmpdir
where necessary.

2890-2898: ⚠️ Potential issue | 🟠 Major

Guarantee tmpdir cleanup with finally in both exception-path tests.

These tests still remove tmpdir only when the expected control flow succeeds. An unexpected exception, or a failing pytest.raises(...), will skip cleanup and leave the temporary directory behind.

🔧 Proposed fix
 def test_ambiguousruleexception():
     path = dpath("test_ambiguousruleexception")
     tmpdir = prepare_tmpdir(path)
     try:
-        run(path, tmpdir=tmpdir)
-    except AmbiguousRuleException:
-        shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS)
-        return
-    shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS)
-    raise AssertionError("This is an ambiguous case! Should have raised an error...")
+        with pytest.raises(AmbiguousRuleException):
+            run(path, tmpdir=tmpdir)
+    finally:
+        shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS)
 
 def test_wildcard_annotatedstrings():
     path = dpath("test_wildcard_annotatedstrings")
     tmpdir = prepare_tmpdir(path, path.name)
-    with pytest.raises(WorkflowError, match=r"unpack\(\) is not allowed with params"):
-        run(path, targets=["test.out"], tmpdir=tmpdir)
-    shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS)
+    try:
+        with pytest.raises(
+            WorkflowError, match=r"unpack\(\) is not allowed with params"
+        ):
+            run(path, targets=["test.out"], tmpdir=tmpdir)
+    finally:
+        shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS)

Also applies to: 2931-2935

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/tests.py` around lines 2890 - 2898, The tmpdir cleanup currently only
happens inside the try/except and is skipped on unexpected failures; wrap the
run(...) call and its except branch in a try/finally so shutil.rmtree(tmpdir,
ignore_errors=ON_WINDOWS) is executed unconditionally—update the test block that
uses dpath("test_ambiguousruleexception"), prepare_tmpdir, run, and
AmbiguousRuleException to perform cleanup in a finally, and apply the same
change to the duplicate test block (the one around lines 2931-2935) to guarantee
tmpdir is always removed.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@tests/tests.py`:
- Around line 470-491: The test test_script_python currently calls
shutil.rmtree(tmpdir, ...) only on the happy path; wrap the setup and all
intermediate operations (the run(...) calls, os.utime(...), and assertions) in a
try/finally and move the final cleanup call shutil.rmtree(tmpdir,
ignore_errors=ON_WINDOWS) into the finally block so the temporary directory is
removed even if a run, os.utime, or assertion raises; apply the same change
pattern to the other test blocks noted (the similar rmtree calls around lines
2563-2568 and 2600-2607) and ensure tmpdir is defined before the try so finally
can always reference it.

---

Duplicate comments:
In `@tests/tests.py`:
- Around line 2037-2040: The TemporaryDirectory prefix is incorrectly using the
helper function name dpath (which has no .name) causing AttributeError; change
the prefix to use the resolved Path variable path (i.e., use path.name) in the
TemporaryDirectory call so the prefix becomes f"snakemake-{path.name}"; update
the line that constructs TemporaryDirectory (where
TemporaryDirectory(prefix=f"snakemake-{dpath.name}") is used) and leave
prepare_tmpdir(path, dest_path=tmpdir) and run(path, ...) unchanged.
- Around line 1785-1792: The test currently assigns tmpdir from run(...), runs
shell("chmod ...") and then calls shutil.rmtree(tmpdir) but if the test fails
before the rmtree call the temp directory is left behind; wrap the run and shell
calls in a try/finally and move the shutil.rmtree(tmpdir) into the finally block
so removal always runs; update the block surrounding the tmpdir variable (the
run(...) call and the subsequent shell(...) chmod) to use try: ... finally:
shutil.rmtree(tmpdir) and guard removal for a non-None tmpdir where necessary.
- Around line 2890-2898: The tmpdir cleanup currently only happens inside the
try/except and is skipped on unexpected failures; wrap the run(...) call and its
except branch in a try/finally so shutil.rmtree(tmpdir,
ignore_errors=ON_WINDOWS) is executed unconditionally—update the test block that
uses dpath("test_ambiguousruleexception"), prepare_tmpdir, run, and
AmbiguousRuleException to perform cleanup in a finally, and apply the same
change to the duplicate test block (the one around lines 2931-2935) to guarantee
tmpdir is always removed.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 77e0328c-37d0-4f8e-b479-3988324cd3a0

📥 Commits

Reviewing files that changed from the base of the PR and between cb1c401 and ba470af.

📒 Files selected for processing (1)
  • tests/tests.py

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tests/tests.py (1)

471-491: ⚠️ Potential issue | 🟠 Major

Move these tmpdir cleanups into finally.

These tests still only call rmtree() on the success path. If any intermediate run() or assertion fails, the tmpdir survives, which undercuts the goal of this PR.

Also applies to: 2564-2568, 2603-2607

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/tests.py` around lines 471 - 491, The tmpdir teardown is only performed
on the success path; wrap the test body that creates and uses tmpdir (the calls
to run(..., tmpdir=tmpdir) and assertions that reference outfile_path and
outfile_timestamp_orig) in a try/finally and move the shutil.rmtree(tmpdir,
ignore_errors=ON_WINDOWS) call into the finally block so the temporary directory
is always removed even if a run() or assertion fails; apply the same change to
the other occurrences noted around the blocks that use tmpdir (lines referenced
by the runs at the top and the ones at 2564-2568 and 2603-2607).
♻️ Duplicate comments (4)
tests/tests.py (4)

2931-2936: ⚠️ Potential issue | 🟠 Major

Cleanup still needs a finally block here.

If pytest.raises(...) fails or run() raises a different exception, shutil.rmtree() is skipped and this test still leaves litter behind.

Suggested fix
 def test_wildcard_annotatedstrings():
     path = dpath("test_wildcard_annotatedstrings")
     tmpdir = prepare_tmpdir(path, path.name)
-    with pytest.raises(WorkflowError, match=r"unpack\(\) is not allowed with params"):
-        run(path, targets=["test.out"], tmpdir=tmpdir)
-    shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS)
+    try:
+        with pytest.raises(
+            WorkflowError, match=r"unpack\(\) is not allowed with params"
+        ):
+            run(path, targets=["test.out"], tmpdir=tmpdir)
+    finally:
+        shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/tests.py` around lines 2931 - 2936, The test
test_wildcard_annotatedstrings leaves the temporary directory behind if
pytest.raises or run() fails; wrap the run/assertion and the rmtree call in a
try/finally so shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS) always executes.
Locate the test_wildcard_annotatedstrings function (uses prepare_tmpdir,
pytest.raises and run) and move the run(...) call into a try block and perform
the cleanup in the finally block to guarantee removal of tmpdir.

1785-1792: ⚠️ Potential issue | 🟠 Major

Keep the chmod cleanup path in finally.

If shell(...) raises here, the rmtree() is skipped and this test still leaves litter behind. Let finally own cleanup, and tolerate deletion failures there because chmod is exactly what makes the tree removable.

Suggested fix
     tmpdir = run(
         dpath("test_github_issue640"),
         targets=["Output/FileWithRights"],
         executor="dryrun",
         cleanup=False,
     )
-    shell(f"chmod -R u+rwX {tmpdir}")
-    shutil.rmtree(tmpdir)
+    try:
+        shell(f"chmod -R u+rwX {tmpdir}")
+    finally:
+        shutil.rmtree(tmpdir, ignore_errors=True)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/tests.py` around lines 1785 - 1792, The temporary-directory cleanup
currently calls shell("chmod -R u+rwX {tmpdir}") and then shutil.rmtree(tmpdir)
inline, which can skip removal if shell() raises; move both the chmod and
shutil.rmtree call into a finally block so cleanup always runs, applying chmod
before attempting rmtree; wrap shutil.rmtree(tmpdir) in a tolerant try/except
(catch Exception) to ignore removal failures (since chmod is what enables
deletion) and reference the existing tmpdir variable and the calls to run(),
shell(), and shutil.rmtree() when making the change.

2890-2899: ⚠️ Potential issue | 🔴 Critical

Don't delete the same tmpdir in both except and finally.

On the expected AmbiguousRuleException path, the first rmtree() removes the directory and the finally block immediately tries to remove it again. On non-Windows that second call raises FileNotFoundError, so the test fails for the wrong reason.

Suggested fix
 def test_ambiguousruleexception():
     path = dpath("test_ambiguousruleexception")
     tmpdir = prepare_tmpdir(path)
     try:
-        run(path, tmpdir=tmpdir)
-    except AmbiguousRuleException:
-        shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS)
-        return
+        with pytest.raises(AmbiguousRuleException):
+            run(path, tmpdir=tmpdir)
     finally:
         shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS)
-    raise AssertionError("This is an ambiguous case! Should have raised an error...")
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/tests.py` around lines 2890 - 2899, The test currently calls
shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS) in both the except
AmbiguousRuleException handler and the finally block, which can cause a
FileNotFoundError on non-Windows; remove the redundant removal from the except
block so cleanup is performed only in the finally block (leave the finally's
shutil.rmtree as the single cleanup), keeping the try/run(path, tmpdir=tmpdir)
call and the except AmbiguousRuleException return behavior intact (references:
tmpdir, run(), AmbiguousRuleException, finally, shutil.rmtree, ON_WINDOWS).

2037-2040: ⚠️ Potential issue | 🔴 Critical

Use path.name here, not dpath.name.

dpath is the helper function, so dpath.name raises before the test body runs. The prefix needs to come from the resolved path object instead.

#!/bin/bash
sed -n '2037,2040p' tests/tests.py
rg -n -A4 '^def dpath' tests/common.py
python - <<'PY'
from pathlib import Path
def dpath(_): return Path("test_github_issue1158")
print("function_has_name_attr:", hasattr(dpath, "name"))
print("resolved_path_name:", dpath("x").name)
PY
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/tests.py` around lines 2037 - 2040, The test constructs a
TemporaryDirectory prefix using the helper function name (dpath.name) which is
incorrect because dpath is a function; change the prefix to use the resolved
Path object's name (path.name) instead — update the TemporaryDirectory call in
the test around run(path, cluster="./qsub.py", tmpdir=tmpdir) so it uses
tempfile.TemporaryDirectory(prefix=f"snakemake-{path.name}") (referencing the
dpath() helper and the local variable path to locate the change).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@tests/tests.py`:
- Around line 471-491: The tmpdir teardown is only performed on the success
path; wrap the test body that creates and uses tmpdir (the calls to run(...,
tmpdir=tmpdir) and assertions that reference outfile_path and
outfile_timestamp_orig) in a try/finally and move the shutil.rmtree(tmpdir,
ignore_errors=ON_WINDOWS) call into the finally block so the temporary directory
is always removed even if a run() or assertion fails; apply the same change to
the other occurrences noted around the blocks that use tmpdir (lines referenced
by the runs at the top and the ones at 2564-2568 and 2603-2607).

---

Duplicate comments:
In `@tests/tests.py`:
- Around line 2931-2936: The test test_wildcard_annotatedstrings leaves the
temporary directory behind if pytest.raises or run() fails; wrap the
run/assertion and the rmtree call in a try/finally so shutil.rmtree(tmpdir,
ignore_errors=ON_WINDOWS) always executes. Locate the
test_wildcard_annotatedstrings function (uses prepare_tmpdir, pytest.raises and
run) and move the run(...) call into a try block and perform the cleanup in the
finally block to guarantee removal of tmpdir.
- Around line 1785-1792: The temporary-directory cleanup currently calls
shell("chmod -R u+rwX {tmpdir}") and then shutil.rmtree(tmpdir) inline, which
can skip removal if shell() raises; move both the chmod and shutil.rmtree call
into a finally block so cleanup always runs, applying chmod before attempting
rmtree; wrap shutil.rmtree(tmpdir) in a tolerant try/except (catch Exception) to
ignore removal failures (since chmod is what enables deletion) and reference the
existing tmpdir variable and the calls to run(), shell(), and shutil.rmtree()
when making the change.
- Around line 2890-2899: The test currently calls shutil.rmtree(tmpdir,
ignore_errors=ON_WINDOWS) in both the except AmbiguousRuleException handler and
the finally block, which can cause a FileNotFoundError on non-Windows; remove
the redundant removal from the except block so cleanup is performed only in the
finally block (leave the finally's shutil.rmtree as the single cleanup), keeping
the try/run(path, tmpdir=tmpdir) call and the except AmbiguousRuleException
return behavior intact (references: tmpdir, run(), AmbiguousRuleException,
finally, shutil.rmtree, ON_WINDOWS).
- Around line 2037-2040: The test constructs a TemporaryDirectory prefix using
the helper function name (dpath.name) which is incorrect because dpath is a
function; change the prefix to use the resolved Path object's name (path.name)
instead — update the TemporaryDirectory call in the test around run(path,
cluster="./qsub.py", tmpdir=tmpdir) so it uses
tempfile.TemporaryDirectory(prefix=f"snakemake-{path.name}") (referencing the
dpath() helper and the local variable path to locate the change).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 67d555ee-6460-48e8-95b2-c192685ad1c4

📥 Commits

Reviewing files that changed from the base of the PR and between ba470af and b5b3ac8.

📒 Files selected for processing (1)
  • tests/tests.py

@johanneskoester johanneskoester merged commit a3a1c97 into snakemake:main Mar 11, 2026
59 checks passed
@github-project-automation github-project-automation bot moved this from In progress to Done in Snakemake Hackathon 2026 Mar 11, 2026
LKress pushed a commit to LKress/snakemake that referenced this pull request Mar 11, 2026
…#4033)

Currently, many tests in the default test suite fail to clean up after
themselves. On CI this doesn't really matter as the VM will be destroyed
once the run completes, but it's mildly annoying for local development.

This PR currently cleans up the low-hanging fruit: either removing
unneeded `cleanup=False` options presumably left over from debugging, or
adding explicit cleanup.

A few cases are harder to deal with: the following tests cause the call
to `run()` to raise an exception:

- `test_github_issue1158`
- `test_ambiguousruleexception`
- `test_wildcard_annotatedstrings`

In this case, `run()` never reaches the cleanup code, and the calling
function doesn't (and can't) know the temporary path used to be able to
clean up.

I can see a few options on how to work around this:

1. Ugly hack: make an extra call to `run()` but requesting a no-op
target to generate the temporary directory, then pass this in to the
current call to `run()` to perform the test; after generating the error,
then remove the temporary directory.
2. Move the logic in `run()` that creates the temporary directory to a
separate helper function, and call this explicitly from the
error-producing tests, then proceed as in (1) above.
3. Refactor the temporary directory functionality to be a class that
manages the temporary directory, and deletes it when the instance is
destroyed—this could be just directly using
`tempfile.TemporaryDirectory`, or reimplementing similar functionality.

@johanneskoester Do you have a preference (one of these three, or
something else entirely)?

### QC
<!-- Make sure that you can tick the boxes below. -->

* [ ] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [X] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).

In principle one could add a test case that tests do not leave litter,
but having tests for your tests seems a little excessive.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added a reusable temp-directory helper to support both in-place and
isolated test modes.
* Standardized temp-dir resolution and result-path handling across
tests, with clearer behavior when running without a temp dir.
* Restored and unified cleanup behavior (including platform-aware
handling), ensuring temporary artifacts are removed on success and
failure.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
johanneskoester pushed a commit that referenced this pull request Mar 13, 2026
🤖 I have created a release *beep* *boop*
---


##
[9.17.0](v9.16.3...v9.17.0)
(2026-03-13)


### Features

* Allow storing snakemake metadata in files or databases
([#4012](#4012))
([dd75f31](dd75f31))
* Allow to specify comparison command per-unit test
([#3956](#3956))
([b88171c](b88171c))
* job table orderd topological when run is started
([#4018](#4018))
([75cf506](75cf506))
* lambda functions for priority in rules
([#3253](#3253))
([d2aa226](d2aa226))
* Make on... directive of modules accessible
([#4050](#4050))
([e9f2e1c](e9f2e1c))


### Bug Fixes

* adjust conda tests to not fail on apple silicon; fix
[#4040](#4040)
([#4049](#4049))
([f5b0142](f5b0142))
* allow "--containerize apptainer" to output apptainer format instead of
dockerfile ([#4030](#4030))
([f5cac30](f5cac30))
* apptainer command not recognized when singularity is absent
([#4010](#4010))
([b8162e2](b8162e2))
* capture stderr when tests fail
([#3995](#3995))
([97d74ba](97d74ba))
* **docs:** make Data-dependent conditional execution a complete example
([#4043](#4043))
([3a1d7f2](3a1d7f2))
* don't build the DAG when running unlock. Fixes
[#4000](#4000) and
[#198](#198)
([#4007](#4007))
([acf79fd](acf79fd))
* Ensure pixi tasks may be run as advertised
([#4046](#4046))
([88253c2](88253c2))
* fix checkpoint handling corner cases
([#3870](#3870) and
[#3559](#3559))
([#4015](#4015))
([63f4257](63f4257))
* issue 3642
([#4054](#4054))
([76e6fc2](76e6fc2))
* issue 3815
([#4026](#4026))
([b0eec96](b0eec96))
* logging None in shellcmd context causes error
([#4064](#4064))
([d0652cd](d0652cd))
* lookup function returns default value for empty DataFrame queries
([#4056](#4056))
([f71de97](f71de97))
* make `cache: omit-software` a rule specific property
([#4085](#4085))
([034a9e7](034a9e7))
* reduce number of tests leaving temporary files behind
([#4033](#4033))
([a3a1c97](a3a1c97))
* regression in dynamic resource handling
([#4038](#4038))
([f2c554a](f2c554a))
* somewhat shorter announce message
([#4080](#4080))
([57efc71](57efc71))


### Performance Improvements

* switch reretry with tenacity; decouple container classes (with Python
3.7 compat for old scripts) from rest of the codebase (enabling moving
to newer python versions)
([#4032](#4032))
([ffb19e7](ffb19e7))


### Documentation

* Add AI-assisted contributions policy to contributing guidelines
([#4051](#4051))
([dd70526](dd70526))
* **codebase:** Update & simplify plugin architecture section
([#4052](#4052))
([176cf63](176cf63))
* Correct workflow.source_path() description in documentation
([#4036](#4036))
([45883c5](45883c5))
* fixed wrong code example for collect() function
([#4037](#4037))
([5c85ed8](5c85ed8))
* Minor docs improvements
([#4089](#4089))
([29ea226](29ea226))
* switch to sphinx_design for tabs
([#3976](#3976))
([9674614](9674614))
* typo in the migration table breaking a pip install command
([#4024](#4024))
([66f9dda](66f9dda))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants