Skip to content

fix: checkpoints inside modules are overwritten #3359

Merged
johanneskoester merged 12 commits intosnakemake:mainfrom
Hocnonsense:module_checkpoints-1
Mar 14, 2025
Merged

fix: checkpoints inside modules are overwritten #3359
johanneskoester merged 12 commits intosnakemake:mainfrom
Hocnonsense:module_checkpoints-1

Conversation

@Hocnonsense
Copy link
Copy Markdown
Contributor

@Hocnonsense Hocnonsense commented Mar 10, 2025

update of #2843, may merge to there

Summary by CodeRabbit

  • New Features

    • Introduced a modular workflow with distinct processing stages and clear output checkpoints.
    • Added new modules and rules for enhanced workflow capabilities.
  • Tests

    • Expanded the test suite with additional module checkpoint coverage for enhanced workflow integrity.
  • Refactor

    • Optimized checkpoint management to improve modularity and stability in workflow execution.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 10, 2025

📝 Walkthrough

Walkthrough

This pull request introduces a new Snakemake workflow for module checkpoints along with supporting expected result files, tests, and internal modifications. A new workflow is defined in two Snakefiles for module-specific checkpointing, along with corresponding expected outputs for different modules. In addition, a new test function is added. The checkpoint mechanism has been refactored by updating the Checkpoints class to use a private attribute with a property accessor and by introducing a new CheckpointsProxy. The WorkflowModifier now initializes the checkpoints global using this new namespace.

Changes

File(s) Change Summary
tests/test_module_checkpoint/Snakefile New Snakefile defining a workflow with two modules (test_a and test_b), each using a separate configuration/prefix, printing checkpoints IDs, asserting configuration, and defining a rule all to aggregate outputs.
tests/test_module_checkpoint/expected-results/... New expected result files added:
- a/post/a.txt: Contains "post_a".
- a/somestep/a.txt: Added line "AAA".
- b/alt/b.txt: Contains "alt_b".
- b/somestep/b.txt: Contains "BBB".
tests/test_module_checkpoint/module-test/Snakefile New Snakefile defining a workflow with a checkpoint rule (somestep), rules for intermediate steps (intermediate and alt_intermediate), an aggregate_input function to determine the correct input based on file content, and a rule aggregate to finalize the aggregated output.
tests/tests.py New test function test_module_checkpoint() added to extend test coverage for module checkpoints.
src/snakemake/checkpoints.py Updated Checkpoints class to implement a singleton pattern: the created_output attribute is now a private set with a property accessor, a new method spawn_new_namespace() has been added, and a new CheckpointsProxy class (subclassing Checkpoints) has been introduced. The register method now includes type hints and simplified logic, and the get method’s condition was updated.
src/snakemake/modules.py In WorkflowModifier.__init__, the global checkpoints variable is now reassigned using the new spawn_new_namespace() method to create a namespace proxy.
src/snakemake/dag.py Updated update_checkpoint_outputs method to explicitly clear and update the created_output attribute without changing its overall functionality.

Sequence Diagram(s)

sequenceDiagram
    participant WM as WorkflowModifier
    participant CP as Checkpoints
    participant CPP as CheckpointsProxy
    participant ST as Module-Test Snakefile
    participant AG as Aggregate Rule

    WM->>CP: Call spawn_new_namespace()
    CP-->>CPP: Return new CheckpointsProxy instance
    Note over WM, CPP: Global "checkpoints" updated
    ST->>ST: Execute checkpoint "somestep" rule
    ST->>ST: Run aggregate_input() to read "somestep/{sample}.txt"
    alt Content is "AAA"
        ST->>ST: Execute rule "intermediate" (post output)
    else Otherwise
        ST->>ST: Execute rule "alt_intermediate" (alt output)
    end
    ST->>AG: Execute rule "aggregate" using determined input
    AG-->>ST: Produce aggregated output
Loading

Suggested reviewers

  • johanneskoester

Tip

⚡🧪 Multi-step agentic review comment chat (experimental)
  • We're introducing multi-step agentic chat in review comments. This experimental feature enhances review discussions with the CodeRabbit agentic chat by enabling advanced interactions, including the ability to create pull requests directly from comments.
    - To enable this feature, set early_access to true under in the settings.

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1b89a0c and b48540f.

📒 Files selected for processing (1)
  • src/snakemake/dag.py (1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • src/snakemake/dag.py
⏰ Context from checks skipped due to timeout of 90000ms (31)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (7, windows-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (6, windows-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (1, windows-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py311)
  • GitHub Check: apidocs
🔇 Additional comments (1)
src/snakemake/dag.py (1)

308-312: Improved checkpoint output handling for modules

This change properly implements the pattern needed for the new CheckpointsProxy class. By clearing and then updating the set in two separate steps, it ensures that checkpoint outputs from modules won't be overwritten, which aligns with the PR's objective.

✨ Finishing Touches
  • 📝 Generate Docstrings

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
snakemake/checkpoints.py (1)

19-38: Good implementation of the wrapper class

The ModuleCheckpoints class effectively delegates property access to the parent Checkpoints instance through getters and setters, ensuring that both future_output and created_output are properly synchronized between module and parent contexts.

Consider adding a brief docstring to explain the purpose of this class, something like:

"""A wrapper around Checkpoints for use in modules to ensure checkpoints are properly tracked in both module and parent contexts."""
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d6b5af0 and 01a9502.

📒 Files selected for processing (2)
  • snakemake/checkpoints.py (1 hunks)
  • tests/tests.py (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/tests.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • snakemake/checkpoints.py
⏰ Context from checks skipped due to timeout of 90000ms (29)
  • GitHub Check: testing (10, 3.12, bash)
  • GitHub Check: testing (10, 3.11, bash)
  • GitHub Check: testing (9, 3.12, bash)
  • GitHub Check: testing (8, 3.12, bash)
  • GitHub Check: testing (8, 3.11, bash)
  • GitHub Check: testing (7, 3.12, bash)
  • GitHub Check: testing (7, 3.11, bash)
  • GitHub Check: testing (6, 3.12, bash)
  • GitHub Check: testing (6, 3.11, bash)
  • GitHub Check: testing-windows (10)
  • GitHub Check: testing (5, 3.12, bash)
  • GitHub Check: testing-windows (9)
  • GitHub Check: testing (5, 3.11, bash)
  • GitHub Check: testing-windows (8)
  • GitHub Check: testing (4, 3.12, bash)
  • GitHub Check: testing-windows (7)
  • GitHub Check: testing (4, 3.11, bash)
  • GitHub Check: testing-windows (6)
  • GitHub Check: testing (3, 3.12, bash)
  • GitHub Check: testing-windows (5)
  • GitHub Check: testing (3, 3.11, bash)
  • GitHub Check: testing-windows (4)
  • GitHub Check: testing (2, 3.12, bash)
  • GitHub Check: testing-windows (3)
  • GitHub Check: testing (2, 3.11, bash)
  • GitHub Check: testing-windows (2)
  • GitHub Check: testing (1, 3.12, bash)
  • GitHub Check: testing-windows (1)
  • GitHub Check: testing (1, 3.11, bash)
🔇 Additional comments (2)
snakemake/checkpoints.py (2)

15-17: Implementation looks good!

The module_use method is a clean way to create a ModuleCheckpoints wrapper around the current Checkpoints instance, enabling proper checkpoint handling in modules.


43-44: Fix for the overwriting issue looks correct

This change ensures checkpoint registration happens in both the current instance and the parent instance, effectively fixing the issue where checkpoints inside modules were being overwritten.

@sonarqubecloud
Copy link
Copy Markdown

@Hocnonsense Hocnonsense force-pushed the module_checkpoints-1 branch from 79310b4 to e9f8022 Compare March 13, 2025 10:49
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
src/snakemake/checkpoints.py (3)

10-13: Improve docstring grammar.
The phrase "This never go into snakefile" could be corrected for clarity. Consider:

-    """A singleton object in a workflow.
-    Created_output can be accessed by checkpoint rules in or out modules.
-    This never go into snakefile, so no rules name will be set to it.
-    """
+    """
+    A singleton object in a workflow.
+    The created_output set can be accessed by checkpoint rules in or out of modules.
+    This never goes into a snakefile, so no rule names will be set on it.
+    """

36-38: Consider verifying parent is not None.
The constructor accepts a parent: Checkpoints. Adding a small check or assertion that parent is not None might prevent cryptic bugs if misused.


47-47: Potential naming collisions when using setattr.
While dynamically assigning attributes is handy, it can overwrite existing methods or attributes if rule.name conflicts. You might consider raising an error if rule.name already exists on self.

 def register(self, rule: "Rule", fallback_name=None):
     checkpoint = Checkpoint(rule, self)
     if fallback_name:
         setattr(self, fallback_name, checkpoint)
-    setattr(self, rule.name, checkpoint)
+    if hasattr(self, rule.name):
+        raise WorkflowError(f"Attribute '{rule.name}' already exists on this checkpoint proxy.")
+    else:
+        setattr(self, rule.name, checkpoint)
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 79310b4 and e9f8022.

📒 Files selected for processing (10)
  • src/snakemake/checkpoints.py (2 hunks)
  • src/snakemake/dag.py (1 hunks)
  • src/snakemake/modules.py (1 hunks)
  • tests/test_module_checkpoint/Snakefile (1 hunks)
  • tests/test_module_checkpoint/expected-results/a/post/a.txt (1 hunks)
  • tests/test_module_checkpoint/expected-results/a/somestep/a.txt (1 hunks)
  • tests/test_module_checkpoint/expected-results/b/alt/b.txt (1 hunks)
  • tests/test_module_checkpoint/expected-results/b/somestep/b.txt (1 hunks)
  • tests/test_module_checkpoint/module-test/Snakefile (1 hunks)
  • tests/tests.py (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • tests/test_module_checkpoint/expected-results/a/somestep/a.txt
🚧 Files skipped from review as they are similar to previous changes (6)
  • tests/test_module_checkpoint/expected-results/a/post/a.txt
  • tests/test_module_checkpoint/expected-results/b/somestep/b.txt
  • tests/test_module_checkpoint/expected-results/b/alt/b.txt
  • tests/tests.py
  • tests/test_module_checkpoint/Snakefile
  • tests/test_module_checkpoint/module-test/Snakefile
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • src/snakemake/dag.py
  • src/snakemake/modules.py
  • src/snakemake/checkpoints.py
⏰ Context from checks skipped due to timeout of 90000ms (41)
  • GitHub Check: tests (10, windows-latest, py312, bash)
  • GitHub Check: tests (10, windows-latest, py311, bash)
  • GitHub Check: tests (10, ubuntu-latest, py312, bash)
  • GitHub Check: tests (10, ubuntu-latest, py311, bash)
  • GitHub Check: tests (9, windows-latest, py312, bash)
  • GitHub Check: tests (9, windows-latest, py311, bash)
  • GitHub Check: tests (9, ubuntu-latest, py312, bash)
  • GitHub Check: tests (9, ubuntu-latest, py311, bash)
  • GitHub Check: tests (8, windows-latest, py312, bash)
  • GitHub Check: tests (8, windows-latest, py311, bash)
  • GitHub Check: tests (8, ubuntu-latest, py312, bash)
  • GitHub Check: tests (8, ubuntu-latest, py311, bash)
  • GitHub Check: tests (7, windows-latest, py312, bash)
  • GitHub Check: tests (7, windows-latest, py311, bash)
  • GitHub Check: tests (7, ubuntu-latest, py312, bash)
  • GitHub Check: tests (7, ubuntu-latest, py311, bash)
  • GitHub Check: tests (6, windows-latest, py312, bash)
  • GitHub Check: tests (6, windows-latest, py311, bash)
  • GitHub Check: tests (6, ubuntu-latest, py312, bash)
  • GitHub Check: tests (6, ubuntu-latest, py311, bash)
  • GitHub Check: tests (5, windows-latest, py312, bash)
  • GitHub Check: tests (5, windows-latest, py311, bash)
  • GitHub Check: tests (5, ubuntu-latest, py312, bash)
  • GitHub Check: tests (5, ubuntu-latest, py311, bash)
  • GitHub Check: tests (4, windows-latest, py312, bash)
  • GitHub Check: tests (4, windows-latest, py311, bash)
  • GitHub Check: tests (4, ubuntu-latest, py312, bash)
  • GitHub Check: tests (4, ubuntu-latest, py311, bash)
  • GitHub Check: tests (3, windows-latest, py312, bash)
  • GitHub Check: tests (3, windows-latest, py311, bash)
  • GitHub Check: tests (3, ubuntu-latest, py312, bash)
  • GitHub Check: tests (3, ubuntu-latest, py311, bash)
  • GitHub Check: tests (2, windows-latest, py312, bash)
  • GitHub Check: tests (2, windows-latest, py311, bash)
  • GitHub Check: tests (2, ubuntu-latest, py312, bash)
  • GitHub Check: tests (2, ubuntu-latest, py311, bash)
  • GitHub Check: tests (1, windows-latest, py312, bash)
  • GitHub Check: tests (1, windows-latest, py311, bash)
  • GitHub Check: tests (1, ubuntu-latest, py312, bash)
  • GitHub Check: tests (1, ubuntu-latest, py311, bash)
  • GitHub Check: apidocs
🔇 Additional comments (8)
src/snakemake/modules.py (1)

191-191: Ensure non-null checkpoints reference.
Assigning self.globals["checkpoints"] to the result of module_use() is a solid approach to create a distinct namespace for modules. However, verify that self.globals["checkpoints"] is non-null before calling module_use() to avoid potential attribute errors if the workflow initialization changes upstream.

src/snakemake/dag.py (1)

280-284: Looks good – avoids overwriting checkpoint outputs.
Changing from assignment to using update() preserves previously recorded outputs. Confirm that workflow.checkpoints.created_output is always a set-like object for seamless calls to update().

src/snakemake/checkpoints.py (6)

17-18: Initialization of the private set is efficient.
Storing _created_output as a set helps avoid duplicates and ensures O(1) containment checks. No issues noted.


19-25: Property and module-use mechanism simplifies usage.
Exposing _created_output via a property is a clean approach, and the module_use() method facilitating a CheckpointsProxy is well-structured.


28-33: New proxy class structure is clear.
Creating CheckpointsProxy to differentiate module-specific checkpoints while sharing the same output state is a good design to ensure modular integrity.


39-41: Property forwarding approach is concise.
Forwarding access to parent.created_output ensures a consistent set of created outputs across proxies.


43-43: Type-annotated register method enhances clarity.
Specifying rule: "Rule" makes the signature more descriptive without impacting runtime.


65-67: Check partial vs. complete creation.
Subtracting from created_output works for full coverage, but if only some outputs are created, it still raises IncompleteCheckpointException. Consider a scenario allowing partial completions, if relevant to your workflow.

@johanneskoester johanneskoester enabled auto-merge (squash) March 14, 2025 14:03
@Hocnonsense Hocnonsense disabled auto-merge March 14, 2025 14:20
@johanneskoester johanneskoester enabled auto-merge (squash) March 14, 2025 14:29
@johanneskoester johanneskoester merged commit fba3ac7 into snakemake:main Mar 14, 2025
49 checks passed
@github-project-automation github-project-automation bot moved this from In review to Done in Snakemake Hackathon March 2025 Mar 14, 2025
@Hocnonsense Hocnonsense deleted the module_checkpoints-1 branch March 14, 2025 15:00
johanneskoester pushed a commit that referenced this pull request Mar 14, 2025
🤖 I have created a release *beep* *boop*
---


##
[9.0.0](v8.30.0...v9.0.0)
(2025-03-14)


### ⚠ BREAKING CHANGES

* Logging refactor & add LoggerPluginInterface
([#3107](#3107))

### Features

* [#3412](#3412) - keep
shadow folder of failed job if --keep-incomplete flag is set.
([#3430](#3430))
([22978c3](22978c3))
* add flag --report-after-run to automatically generate the report after
a successfull workflow run
([#3428](#3428))
([b0a7f03](b0a7f03))
* add flatten function to IO utils
([#3424](#3424))
([67fa392](67fa392))
* add helper functions to parse input files
([#2918](#2918))
([63e45a7](63e45a7))
* Add option to print redacted file names
([#3089](#3089))
([ba4d264](ba4d264))
* add support for validation of polars dataframe and lazyframe
([#3262](#3262))
([c7473a6](c7473a6))
* added support for rendering dag with mermaid js
([#3409](#3409))
([7bf8381](7bf8381))
* adding --replace-workflow-config to fully replace workflow configs
(from config: directive) with --configfile, instead of merging them
([#3381](#3381))
([47504a0](47504a0))
* Dynamic module name
([#3401](#3401))
([024dc32](024dc32))
* Enable saving and reloading IOCache object
([#3386](#3386))
([c935953](c935953))
* files added in rule params with workflow.source_path will be available
in used containers
([#3385](#3385))
([a6e45bf](a6e45bf))
* Fix keep_local in storage directive and more freedom over remote
retrieval behaviour
([#3410](#3410))
([67b4739](67b4739))
* inherit parameters of use rule and extend/replace individual items
them when using 'with' directive
([#3365](#3365))
([93e4b92](93e4b92))
* Logging refactor & add LoggerPluginInterface
([#3107](#3107))
([86f1d6e](86f1d6e))
* Maximal file size for checksums
([#3368](#3368))
([b039f8a](b039f8a))
* Modernize package configuration using Pixi
([#3369](#3369))
([77992d8](77992d8))
* multiext support for named input/output
([#3372](#3372))
([05e1378](05e1378))
* optionally auto-group jobs via temp files in case of remote execution
([#3378](#3378))
([cc9bba2](cc9bba2))


### Bug Fixes

* `--delete-all-output` ignores `--dry-run`
([#3265](#3265))
([23fef82](23fef82))
* 3342 faster touch runs and warning messages for non-existing files
([#3398](#3398))
([cd9c3c3](cd9c3c3))
* add default value to max-jobs-per-timespan
([#3043](#3043))
([2959abe](2959abe))
* checkpoints inside modules are overwritten
([#3359](#3359))
([fba3ac7](fba3ac7))
* Convert Path to IOFile
([#3405](#3405))
([c58684c](c58684c))
* Do not perform storage object cleanup with --keep-storage-local-copies
set ([#3358](#3358))
([9a6d14b](9a6d14b))
* edgecases of source deployment in case of remote execution
([#3396](#3396))
([5da13be](5da13be))
* enhance error message formatting for strict DAG-building mode
([#3376](#3376))
([a1c39ee](a1c39ee))
* fix bug in checkpoint handling that led to exceptions in case
checkpoint output was missing upon rerun
([#3423](#3423))
([8cf4a2f](8cf4a2f))
* force check all required outputs
([#3341](#3341))
([495a4e7](495a4e7))
* group job formatting
([#3442](#3442))
([f0b10a3](f0b10a3))
* in remote jobs, upload storage in topological order such that
modification dates are preserved (e.g. in case of group jobs)
([#3377](#3377))
([eace08f](eace08f))
* only skip eval when resource depends on input
([#3374](#3374))
([4574c92](4574c92))
* Prevent execution of conda in apptainer when not explicitly requested
in software deployment method
([#3388](#3388))
([c43c5c0](c43c5c0))
* print filenames with quotes around them in RuleException
([#3269](#3269))
([6baeda5](6baeda5))
* Re-evaluation of free resources
([#3399](#3399))
([6371293](6371293))
* ReadTheDocs layout issue due to src directory change
([#3419](#3419))
([695b127](695b127))
* robustly escaping quotes in generated bash scripts (v2)
([#3297](#3297))
([#3389](#3389))
([58720bd](58720bd))
* Show apptainer image URL in snakemake report
([#3407](#3407))
([45f0450](45f0450))
* Update ReadTheDocs configuration for documentation build to use Pixi
([#3433](#3433))
([3f227a6](3f227a6))


### Documentation

* Add pixi setup instructions to general use tutorial
([#3382](#3382))
([115e81b](115e81b))
* fix contribution section heading levels, fix docs testing setup order
([#3360](#3360))
([051dc53](051dc53))
* fix link to github.com/snakemake/poetry-snakemake-plugin
([#3436](#3436))
([ec6d97c](ec6d97c))
* fix quoting
([#3394](#3394))
([b40f599](b40f599))
* fix rerun-triggers default
([#3403](#3403))
([4430e23](4430e23))
* fix typo 'safe' -&gt; 'save'
([#3384](#3384))
([7755861](7755861))
* mention code formatting in the contribution section
([#3431](#3431))
([e8682b7](e8682b7))
* remove duplicated 'functions'.
([#3356](#3356))
([7c595db](7c595db))
* update broken links documentation
([#3437](#3437))
([e3d0d88](e3d0d88))
* Updating contributing guidelines with new pixi dev setup
([#3415](#3415))
([8e95a12](8e95a12))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: snakemake-bot <snakemake-bot-admin@googlegroups.com>
kjohnsen pushed a commit to kjohnsen/snakemake that referenced this pull request Dec 15, 2025
update of snakemake#2843, may merge to there

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Introduced a modular workflow with distinct processing stages and
clear output checkpoints.
  - Added new modules and rules for enhanced workflow capabilities.
  
- **Tests**
- Expanded the test suite with additional module checkpoint coverage for
enhanced workflow integrity.

- **Refactor**
- Optimized checkpoint management to improve modularity and stability in
workflow execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: fgvieira <1151762+fgvieira@users.noreply.github.com>
Co-authored-by: Johannes Koester <johannes.koester@uni-due.de>
kjohnsen pushed a commit to kjohnsen/snakemake that referenced this pull request Dec 15, 2025
🤖 I have created a release *beep* *boop*
---


##
[9.0.0](snakemake/snakemake@v8.30.0...v9.0.0)
(2025-03-14)


### ⚠ BREAKING CHANGES

* Logging refactor & add LoggerPluginInterface
([snakemake#3107](snakemake#3107))

### Features

* [snakemake#3412](snakemake#3412) - keep
shadow folder of failed job if --keep-incomplete flag is set.
([snakemake#3430](snakemake#3430))
([22978c3](snakemake@22978c3))
* add flag --report-after-run to automatically generate the report after
a successfull workflow run
([snakemake#3428](snakemake#3428))
([b0a7f03](snakemake@b0a7f03))
* add flatten function to IO utils
([snakemake#3424](snakemake#3424))
([67fa392](snakemake@67fa392))
* add helper functions to parse input files
([snakemake#2918](snakemake#2918))
([63e45a7](snakemake@63e45a7))
* Add option to print redacted file names
([snakemake#3089](snakemake#3089))
([ba4d264](snakemake@ba4d264))
* add support for validation of polars dataframe and lazyframe
([snakemake#3262](snakemake#3262))
([c7473a6](snakemake@c7473a6))
* added support for rendering dag with mermaid js
([snakemake#3409](snakemake#3409))
([7bf8381](snakemake@7bf8381))
* adding --replace-workflow-config to fully replace workflow configs
(from config: directive) with --configfile, instead of merging them
([snakemake#3381](snakemake#3381))
([47504a0](snakemake@47504a0))
* Dynamic module name
([snakemake#3401](snakemake#3401))
([024dc32](snakemake@024dc32))
* Enable saving and reloading IOCache object
([snakemake#3386](snakemake#3386))
([c935953](snakemake@c935953))
* files added in rule params with workflow.source_path will be available
in used containers
([snakemake#3385](snakemake#3385))
([a6e45bf](snakemake@a6e45bf))
* Fix keep_local in storage directive and more freedom over remote
retrieval behaviour
([snakemake#3410](snakemake#3410))
([67b4739](snakemake@67b4739))
* inherit parameters of use rule and extend/replace individual items
them when using 'with' directive
([snakemake#3365](snakemake#3365))
([93e4b92](snakemake@93e4b92))
* Logging refactor & add LoggerPluginInterface
([snakemake#3107](snakemake#3107))
([86f1d6e](snakemake@86f1d6e))
* Maximal file size for checksums
([snakemake#3368](snakemake#3368))
([b039f8a](snakemake@b039f8a))
* Modernize package configuration using Pixi
([snakemake#3369](snakemake#3369))
([77992d8](snakemake@77992d8))
* multiext support for named input/output
([snakemake#3372](snakemake#3372))
([05e1378](snakemake@05e1378))
* optionally auto-group jobs via temp files in case of remote execution
([snakemake#3378](snakemake#3378))
([cc9bba2](snakemake@cc9bba2))


### Bug Fixes

* `--delete-all-output` ignores `--dry-run`
([snakemake#3265](snakemake#3265))
([23fef82](snakemake@23fef82))
* 3342 faster touch runs and warning messages for non-existing files
([snakemake#3398](snakemake#3398))
([cd9c3c3](snakemake@cd9c3c3))
* add default value to max-jobs-per-timespan
([snakemake#3043](snakemake#3043))
([2959abe](snakemake@2959abe))
* checkpoints inside modules are overwritten
([snakemake#3359](snakemake#3359))
([fba3ac7](snakemake@fba3ac7))
* Convert Path to IOFile
([snakemake#3405](snakemake#3405))
([c58684c](snakemake@c58684c))
* Do not perform storage object cleanup with --keep-storage-local-copies
set ([snakemake#3358](snakemake#3358))
([9a6d14b](snakemake@9a6d14b))
* edgecases of source deployment in case of remote execution
([snakemake#3396](snakemake#3396))
([5da13be](snakemake@5da13be))
* enhance error message formatting for strict DAG-building mode
([snakemake#3376](snakemake#3376))
([a1c39ee](snakemake@a1c39ee))
* fix bug in checkpoint handling that led to exceptions in case
checkpoint output was missing upon rerun
([snakemake#3423](snakemake#3423))
([8cf4a2f](snakemake@8cf4a2f))
* force check all required outputs
([snakemake#3341](snakemake#3341))
([495a4e7](snakemake@495a4e7))
* group job formatting
([snakemake#3442](snakemake#3442))
([f0b10a3](snakemake@f0b10a3))
* in remote jobs, upload storage in topological order such that
modification dates are preserved (e.g. in case of group jobs)
([snakemake#3377](snakemake#3377))
([eace08f](snakemake@eace08f))
* only skip eval when resource depends on input
([snakemake#3374](snakemake#3374))
([4574c92](snakemake@4574c92))
* Prevent execution of conda in apptainer when not explicitly requested
in software deployment method
([snakemake#3388](snakemake#3388))
([c43c5c0](snakemake@c43c5c0))
* print filenames with quotes around them in RuleException
([snakemake#3269](snakemake#3269))
([6baeda5](snakemake@6baeda5))
* Re-evaluation of free resources
([snakemake#3399](snakemake#3399))
([6371293](snakemake@6371293))
* ReadTheDocs layout issue due to src directory change
([snakemake#3419](snakemake#3419))
([695b127](snakemake@695b127))
* robustly escaping quotes in generated bash scripts (v2)
([snakemake#3297](snakemake#3297))
([snakemake#3389](snakemake#3389))
([58720bd](snakemake@58720bd))
* Show apptainer image URL in snakemake report
([snakemake#3407](snakemake#3407))
([45f0450](snakemake@45f0450))
* Update ReadTheDocs configuration for documentation build to use Pixi
([snakemake#3433](snakemake#3433))
([3f227a6](snakemake@3f227a6))


### Documentation

* Add pixi setup instructions to general use tutorial
([snakemake#3382](snakemake#3382))
([115e81b](snakemake@115e81b))
* fix contribution section heading levels, fix docs testing setup order
([snakemake#3360](snakemake#3360))
([051dc53](snakemake@051dc53))
* fix link to github.com/snakemake/poetry-snakemake-plugin
([snakemake#3436](snakemake#3436))
([ec6d97c](snakemake@ec6d97c))
* fix quoting
([snakemake#3394](snakemake#3394))
([b40f599](snakemake@b40f599))
* fix rerun-triggers default
([snakemake#3403](snakemake#3403))
([4430e23](snakemake@4430e23))
* fix typo 'safe' -&gt; 'save'
([snakemake#3384](snakemake#3384))
([7755861](snakemake@7755861))
* mention code formatting in the contribution section
([snakemake#3431](snakemake#3431))
([e8682b7](snakemake@e8682b7))
* remove duplicated 'functions'.
([snakemake#3356](snakemake#3356))
([7c595db](snakemake@7c595db))
* update broken links documentation
([snakemake#3437](snakemake#3437))
([e3d0d88](snakemake@e3d0d88))
* Updating contributing guidelines with new pixi dev setup
([snakemake#3415](snakemake#3415))
([8e95a12](snakemake@8e95a12))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: snakemake-bot <snakemake-bot-admin@googlegroups.com>
@Hocnonsense Hocnonsense self-assigned this Mar 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

3 participants