feat: keep temp files in case they are an input of target rules that that don't have their own output#3536
Merged
johanneskoester merged 3 commits intomainfrom Apr 16, 2025
Merged
Conversation
…that don't have their own output
Contributor
📝 Walkthrough## Walkthrough
The changes introduce enhancements to the DAG class in the Snakemake workflow engine, specifically by tracking derived target files to refine the logic for determining whether temporary files are still needed. The `derived_targetfiles` attribute is added and populated during DAG initialization. The logic in `is_needed_tempfile` is updated to consider these derived target files, preventing their premature removal. Additionally, a new test workflow and corresponding test case are added to verify the correct handling of temporary files that are also final targets.
## Changes
| File(s) | Change Summary |
|-------------------------------------------|----------------------------------------------------------------------------------------------------------------|
| src/snakemake/dag.py | Added `derived_targetfiles` attribute to the DAG class; updated initialization and tempfile necessity logic. |
| tests/test_temp_and_all_input/Snakefile | Introduced a workflow with a temp output that is also a final target, using two rules: `all` and `a`. |
| tests/tests.py | Added a test function `test_temp_and_all_input` to run the new workflow and verify tempfile handling. |
## Sequence Diagram(s)
```mermaid
sequenceDiagram
participant User
participant Snakemake
participant DAG
participant Job
User->>Snakemake: Run workflow
Snakemake->>DAG: Initialize DAG (init)
DAG->>DAG: Set derived_targetfiles
loop For each job
Snakemake->>DAG: Check if tempfile is needed (is_needed_tempfile)
alt Tempfile is needed by downstream job or in derived_targetfiles
DAG-->>Snakemake: Return True
else
DAG-->>Snakemake: Return False
end
end
Snakemake->>Job: Execute jobs as per workflow
Snakemake->>DAG: Remove temp files if not needed |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
src/snakemake/dag.py(3 hunks)tests/test_temp_and_all_input/Snakefile(1 hunks)tests/tests.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest type annotations for functions that are defined inside of functions or methods. Do not suggest type annotation of the `s...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
tests/tests.pysrc/snakemake/dag.py
🧬 Code Graph Analysis (1)
tests/tests.py (1)
tests/common.py (2)
run(151-495)dpath(32-35)
⏰ Context from checks skipped due to timeout of 90000ms (36)
- GitHub Check: tests (10, macos-latest, py312)
- GitHub Check: tests (10, windows-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (9, windows-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (8, windows-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (7, windows-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (6, windows-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (5, windows-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (4, macos-latest, py312)
- GitHub Check: tests (4, windows-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (3, macos-latest, py312)
- GitHub Check: tests (3, windows-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (2, macos-latest, py312)
- GitHub Check: tests (2, windows-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (1, macos-latest, py312)
- GitHub Check: tests (1, windows-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: apidocs
- GitHub Check: tests (1, ubuntu-latest, py311)
🔇 Additional comments (5)
src/snakemake/dag.py (2)
141-141: Added new attribute to track derived target files.This new attribute helps manage temporary files that are inputs to rules without outputs.
917-924: Improved logic to prevent premature removal of temporary files.The feature now correctly preserves temporary files that are inputs to target rules without outputs by checking if the tempfile is included in the derived_targetfiles set.
tests/tests.py (1)
2333-2335: LGTM! This test verifies the new feature for handling temp files.This test function validates the behavior mentioned in the PR objectives: ensuring that temporary files are retained when they serve as inputs to target rules without their own outputs. The test runs a simple workflow defined in the "test_temp_and_all_input" directory.
tests/test_temp_and_all_input/Snakefile (2)
1-4: LGTM! Appropriate target rule for testing temp file handling.This rule correctly defines a target with no outputs that depends on a temporary file. It's the perfect setup to test the feature mentioned in the PR: keeping temporary files when they are inputs to target rules without outputs.
5-9: LGTM! Well-defined rule with a temporary output.This rule appropriately uses the
temp()flag to mark "foo.txt" as a temporary file. The simple shell command is sufficient for creating the test file. This configuration is crucial for testing the new feature that prevents the premature removal of temporary files when they serve as inputs to target rules without outputs.
johanneskoester
pushed a commit
that referenced
this pull request
Apr 16, 2025
🤖 I have created a release *beep* *boop* --- ## [9.3.0](v9.2.1...v9.3.0) (2025-04-16) ### Features * keep temp files in case they are an input of target rules that that don't have their own output ([#3536](#3536)) ([a2caa0b](a2caa0b)) * make shutil available in resource expressions (i.e. for --default-resources and --set-resources), can be used to e.g. dynamically decide about the tmpdir to be used based on shutil.disk_usage(system_tmpdir) ([#3535](#3535)) ([aeab67f](aeab67f)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
2 tasks
2 tasks
kjohnsen
pushed a commit
to kjohnsen/snakemake
that referenced
this pull request
Dec 15, 2025
…that don't have their own output (snakemake#3536) <!--Add a description of your PR here--> ### QC <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Improved handling of temporary files to prevent premature removal when they are needed as final outputs. - **Tests** - Added a new workflow and corresponding test to verify correct behavior when temporary files are also specified as final targets. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
kjohnsen
pushed a commit
to kjohnsen/snakemake
that referenced
this pull request
Dec 15, 2025
🤖 I have created a release *beep* *boop* --- ## [9.3.0](snakemake/snakemake@v9.2.1...v9.3.0) (2025-04-16) ### Features * keep temp files in case they are an input of target rules that that don't have their own output ([snakemake#3536](snakemake#3536)) ([a2caa0b](snakemake@a2caa0b)) * make shutil available in resource expressions (i.e. for --default-resources and --set-resources), can be used to e.g. dynamically decide about the tmpdir to be used based on shutil.disk_usage(system_tmpdir) ([snakemake#3535](snakemake#3535)) ([aeab67f](snakemake@aeab67f)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
QC
docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).Summary by CodeRabbit