feat: Make on... directive of modules accessible#4050
feat: Make on... directive of modules accessible#4050johanneskoester merged 7 commits intosnakemake:mainfrom
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughPopulate global hook bindings with partials that inject a log argument and make internal hook assignments conditional on whether a registering modifier is the main Snakefile; add WorkflowModifier.is_main_snakefile(), module lifecycle tests, module Snakefiles/configs, and docs clarifying hook scoping. Changes
Sequence Diagram(s)sequenceDiagram
participant Modifier as WorkflowModifier
participant Workflow as Workflow.register_hooks
participant Logger as logger_manager
participant Globals as Module\ Globals
participant Runtime as Runtime\ (hook invocation)
Modifier->>Workflow: register onstart/onsuccess/onerror
Workflow->>Logger: get_logfile()
Logger-->>Workflow: logfile path
Workflow->>Globals: set global binding to partial(hook, log=logfile)
alt modifier.is_main_snakefile() == True
Workflow->>Workflow: set internal _onstart/_onsuccess/_onerror
else modifier is module
note right of Workflow: do not override internal handlers
end
Runtime->>Globals: call onstart/onsuccess/onerror (partial injects log)
Globals-->>Runtime: invoke actual hook with log argument
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/snakemake/workflow.py (1)
1691-1704: Bind the exported module hooks positionally, not bylog=keyword.The runtime path invokes lifecycle hooks as
func(logfile)positionally, but these wrappers prefilllogby name. That makesmodule.onstart()/module.onsuccess()/module.onerror()stricter than normal hook execution and can break any hook registered with a differently named first parameter. Prefer positional binding here so the exported callable matches the existing contract.Suggested change
def onstart(self, func): """Register onstart function.""" self._onstart = func - self.globals["onstart"] = partial(func, log=self.logger_manager.get_logfile()) + self.globals["onstart"] = partial(func, self.logger_manager.get_logfile()) def onsuccess(self, func): """Register onsuccess function.""" self._onsuccess = func - self.globals["onsuccess"] = partial(func, log=self.logger_manager.get_logfile()) + self.globals["onsuccess"] = partial(func, self.logger_manager.get_logfile()) def onerror(self, func): """Register onerror function.""" self._onerror = func - self.globals["onerror"] = partial(func, log=self.logger_manager.get_logfile()) + self.globals["onerror"] = partial(func, self.logger_manager.get_logfile())🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/snakemake/workflow.py` around lines 1691 - 1704, The exported hook wrappers (onstart, onsuccess, onerror) bind the logfile using a keyword argument (`log=`) which breaks positional calling by the runtime; change the partial bindings stored in self.globals["onstart"]/["onsuccess"]/["onerror"] to bind the logfile positionally (e.g., partial(func, self.logger_manager.get_logfile())) so the callable signature matches the runtime contract while still storing the original function in self._onstart/_onsuccess/_onerror.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@tests/tests.py`:
- Around line 2943-2955: The test function
test_module_onstart_not_in_main_snakefile leaves the temporary dir because it
calls run(..., cleanup=False) but never cleans it up; wrap the assertions that
inspect Path(path)/"onstart_module1.log" and Path(path)/"onstart_module2.log" in
a try/finally block and in the finally call the existing cleanup helper (e.g.,
cleanup_dir(path)) to remove the tmpdir, keeping run(..., cleanup=False) so the
files can be checked but ensuring cleanup afterward.
---
Nitpick comments:
In `@src/snakemake/workflow.py`:
- Around line 1691-1704: The exported hook wrappers (onstart, onsuccess,
onerror) bind the logfile using a keyword argument (`log=`) which breaks
positional calling by the runtime; change the partial bindings stored in
self.globals["onstart"]/["onsuccess"]/["onerror"] to bind the logfile
positionally (e.g., partial(func, self.logger_manager.get_logfile())) so the
callable signature matches the runtime contract while still storing the original
function in self._onstart/_onsuccess/_onerror.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: a4a44568-5fca-4c49-9837-c5257f0b2425
⛔ Files ignored due to path filters (10)
tests/test_module_onerror/expected-results/onerror_module1.logis excluded by!**/*.logtests/test_module_onerror/expected-results/onerror_module2.logis excluded by!**/*.logtests/test_module_onerror/expected-results/onstart_module1.logis excluded by!**/*.logtests/test_module_onerror/expected-results/onstart_module2.logis excluded by!**/*.logtests/test_module_onstart_not_in_main_snakefile/expected-results/onsuccess_module1.logis excluded by!**/*.logtests/test_module_onstart_not_in_main_snakefile/expected-results/onsuccess_module2.logis excluded by!**/*.logtests/test_module_onstart_onsuccess/expected-results/onstart_module1.logis excluded by!**/*.logtests/test_module_onstart_onsuccess/expected-results/onstart_module2.logis excluded by!**/*.logtests/test_module_onstart_onsuccess/expected-results/onsuccess_module1.logis excluded by!**/*.logtests/test_module_onstart_onsuccess/expected-results/onsuccess_module2.logis excluded by!**/*.log
📒 Files selected for processing (14)
src/snakemake/workflow.pytests/test_module_onerror/Snakefiletests/test_module_onerror/config/config.yamltests/test_module_onerror/module1/Snakefiletests/test_module_onerror/module2/Snakefiletests/test_module_onstart_not_in_main_snakefile/Snakefiletests/test_module_onstart_not_in_main_snakefile/config/config.yamltests/test_module_onstart_not_in_main_snakefile/module1/Snakefiletests/test_module_onstart_not_in_main_snakefile/module2/Snakefiletests/test_module_onstart_onsuccess/Snakefiletests/test_module_onstart_onsuccess/config/config.yamltests/test_module_onstart_onsuccess/module1/Snakefiletests/test_module_onstart_onsuccess/module2/Snakefiletests/tests.py
16b59a4 to
44ac727
Compare
* Allows referencing the onsuccess, onstart and onerror directive of modules
* Description of onstart, onsuccess, onerror behaviour in combination with module usage
44ac727 to
67df80e
Compare
There was a problem hiding this comment.
♻️ Duplicate comments (1)
tests/tests.py (1)
3120-3132:⚠️ Potential issue | 🟡 MinorRetained tmpdir never gets cleaned up.
Line 3122 disables cleanup so this test can inspect the generated files, but
pathis never removed afterward. That leaves temp dirs behind across repeated runs; please wrap the assertions in a cleanup path as done in similar tests in this file.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/tests.py` around lines 3120 - 3132, The temporary directory returned by run(...) in test_module_onstart_not_in_main_snakefile is left behind because cleanup=False and no teardown is performed; wrap the assertions in a try/finally (or context-style) block so that after asserting the two Path(path)/"onstart_moduleX.log" files do not exist you remove the temp dir (e.g., with shutil.rmtree or Path(path).rmdir equivalents) in the finally clause; reference the test function test_module_onstart_not_in_main_snakefile, the run(...) call that returns path, and the Path(path) usage to locate where to add the cleanup.
🧹 Nitpick comments (1)
tests/test_module_onstart_not_in_main_snakefile/module1/Snakefile (1)
3-9: Make the lifecycle fixtures hermetic.This rule never reads its HTTP input, so the new hook tests can fail on GitHub/network availability before they ever exercise the lifecycle behavior. A local dummy input—or no input at all—would make the failure mode deterministic. The same pattern is repeated in the other new module hook fixtures.
♻️ One simple local-only variant
rule a: - input: - storage.http("https://raw.githubusercontent.com/snakemake/snakemake/main/LICENSE.md"), output: "results/test.out", shell: "echo {config[test]} > {output[0]}"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/test_module_onstart_not_in_main_snakefile/module1/Snakefile` around lines 3 - 9, The rule "a" currently declares an external HTTP input (storage.http(...)) which isn't actually read and introduces network flakiness; update the rule to be hermetic by replacing the HTTP input with a local dummy file (e.g., create or reference a small local fixture file like "inputs/dummy.txt") or remove the input entirely if not needed, and ensure the shell command still writes to {output[0]}; apply the same change to the other module hook fixture Snakefiles that use storage.http so all lifecycle fixtures are local-only and deterministic.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@tests/tests.py`:
- Around line 3120-3132: The temporary directory returned by run(...) in
test_module_onstart_not_in_main_snakefile is left behind because cleanup=False
and no teardown is performed; wrap the assertions in a try/finally (or
context-style) block so that after asserting the two
Path(path)/"onstart_moduleX.log" files do not exist you remove the temp dir
(e.g., with shutil.rmtree or Path(path).rmdir equivalents) in the finally
clause; reference the test function test_module_onstart_not_in_main_snakefile,
the run(...) call that returns path, and the Path(path) usage to locate where to
add the cleanup.
---
Nitpick comments:
In `@tests/test_module_onstart_not_in_main_snakefile/module1/Snakefile`:
- Around line 3-9: The rule "a" currently declares an external HTTP input
(storage.http(...)) which isn't actually read and introduces network flakiness;
update the rule to be hermetic by replacing the HTTP input with a local dummy
file (e.g., create or reference a small local fixture file like
"inputs/dummy.txt") or remove the input entirely if not needed, and ensure the
shell command still writes to {output[0]}; apply the same change to the other
module hook fixture Snakefiles that use storage.http so all lifecycle fixtures
are local-only and deterministic.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: bf583cde-e951-4261-9921-66a293e472c0
⛔ Files ignored due to path filters (10)
tests/test_module_onerror/expected-results/onerror_module1.logis excluded by!**/*.logtests/test_module_onerror/expected-results/onerror_module2.logis excluded by!**/*.logtests/test_module_onerror/expected-results/onstart_module1.logis excluded by!**/*.logtests/test_module_onerror/expected-results/onstart_module2.logis excluded by!**/*.logtests/test_module_onstart_not_in_main_snakefile/expected-results/onsuccess_module1.logis excluded by!**/*.logtests/test_module_onstart_not_in_main_snakefile/expected-results/onsuccess_module2.logis excluded by!**/*.logtests/test_module_onstart_onsuccess/expected-results/onstart_module1.logis excluded by!**/*.logtests/test_module_onstart_onsuccess/expected-results/onstart_module2.logis excluded by!**/*.logtests/test_module_onstart_onsuccess/expected-results/onsuccess_module1.logis excluded by!**/*.logtests/test_module_onstart_onsuccess/expected-results/onsuccess_module2.logis excluded by!**/*.log
📒 Files selected for processing (16)
docs/snakefiles/rules.rstsrc/snakemake/modules.pysrc/snakemake/workflow.pytests/test_module_onerror/Snakefiletests/test_module_onerror/config/config.yamltests/test_module_onerror/module1/Snakefiletests/test_module_onerror/module2/Snakefiletests/test_module_onstart_not_in_main_snakefile/Snakefiletests/test_module_onstart_not_in_main_snakefile/config/config.yamltests/test_module_onstart_not_in_main_snakefile/module1/Snakefiletests/test_module_onstart_not_in_main_snakefile/module2/Snakefiletests/test_module_onstart_onsuccess/Snakefiletests/test_module_onstart_onsuccess/config/config.yamltests/test_module_onstart_onsuccess/module1/Snakefiletests/test_module_onstart_onsuccess/module2/Snakefiletests/tests.py
🚧 Files skipped from review as they are similar to previous changes (6)
- tests/test_module_onstart_onsuccess/module1/Snakefile
- tests/test_module_onstart_onsuccess/config/config.yaml
- tests/test_module_onstart_not_in_main_snakefile/config/config.yaml
- tests/test_module_onstart_not_in_main_snakefile/module2/Snakefile
- tests/test_module_onerror/module2/Snakefile
- tests/test_module_onerror/config/config.yaml
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@tests/tests.py`:
- Around line 3120-3134: The test test_module_onstart_not_in_main_snakefile
references undefined prepare_tmpdir and uses path before assignment; replace the
prepare_tmpdir block with the established pattern: call path =
run(dpath("test_module_onstart_not_in_main_snakefile"), check_results=True) (do
not pass tmpdir), then manually clean up the returned path after assertions with
shutil.rmtree(path) (use ON_WINDOWS if platform-specific handling is needed).
Also add the missing imports: import shutil and from snakemake.common import
ON_WINDOWS (or import ON_WINDOWS from tests.common if that pattern exists).
There was a problem hiding this comment.
♻️ Duplicate comments (1)
tests/tests.py (1)
3142-3156:⚠️ Potential issue | 🔴 Critical
pathis referenced before assignment – test will raiseNameError.On line 3144,
pathis passed toprepare_tmpdir()but isn't defined until line 3145. The static analysis tool also flagged this (F821). Looking at similar tests (e.g.,test_github_issue1158at line 2116),prepare_tmpdir()should receive the test directory path directly.🐛 Proposed fix
def test_module_onstart_not_in_main_snakefile(): # check that onstart is not executed, if not in the main snakefile - with prepare_tmpdir(path) as tmpdir: + with prepare_tmpdir(dpath("test_module_onstart_not_in_main_snakefile")) as tmpdir: path = run( dpath("test_module_onstart_not_in_main_snakefile"), check_results=True, cleanup=False, tmpdir=tmpdir, ) assert not ( Path(path) / "onstart_module1.log" ).exists(), "onstart should not be executed for module1" assert not ( Path(path) / "onstart_module2.log" ).exists(), "onstart should not be executed for module2"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/tests.py` around lines 3142 - 3156, The test test_module_onstart_not_in_main_snakefile references path before assignment; change the prepare_tmpdir call to receive the test directory spec directly and keep the run(...) assignment inside the context. Concretely, call prepare_tmpdir(dpath("test_module_onstart_not_in_main_snakefile")) instead of prepare_tmpdir(path) and then set path = run(dpath("test_module_onstart_not_in_main_snakefile"), check_results=True, cleanup=False, tmpdir=tmpdir) inside the with-block so prepare_tmpdir and run use the correct directory values.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@tests/tests.py`:
- Around line 3142-3156: The test test_module_onstart_not_in_main_snakefile
references path before assignment; change the prepare_tmpdir call to receive the
test directory spec directly and keep the run(...) assignment inside the
context. Concretely, call
prepare_tmpdir(dpath("test_module_onstart_not_in_main_snakefile")) instead of
prepare_tmpdir(path) and then set path =
run(dpath("test_module_onstart_not_in_main_snakefile"), check_results=True,
cleanup=False, tmpdir=tmpdir) inside the with-block so prepare_tmpdir and run
use the correct directory values.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 14a6bb58-7d5f-4065-9f83-fd2b6be0372f
📒 Files selected for processing (2)
docs/snakefiles/rules.rsttests/tests.py
🚧 Files skipped from review as they are similar to previous changes (1)
- docs/snakefiles/rules.rst
🤖 I have created a release *beep* *boop* --- ## [9.17.0](v9.16.3...v9.17.0) (2026-03-13) ### Features * Allow storing snakemake metadata in files or databases ([#4012](#4012)) ([dd75f31](dd75f31)) * Allow to specify comparison command per-unit test ([#3956](#3956)) ([b88171c](b88171c)) * job table orderd topological when run is started ([#4018](#4018)) ([75cf506](75cf506)) * lambda functions for priority in rules ([#3253](#3253)) ([d2aa226](d2aa226)) * Make on... directive of modules accessible ([#4050](#4050)) ([e9f2e1c](e9f2e1c)) ### Bug Fixes * adjust conda tests to not fail on apple silicon; fix [#4040](#4040) ([#4049](#4049)) ([f5b0142](f5b0142)) * allow "--containerize apptainer" to output apptainer format instead of dockerfile ([#4030](#4030)) ([f5cac30](f5cac30)) * apptainer command not recognized when singularity is absent ([#4010](#4010)) ([b8162e2](b8162e2)) * capture stderr when tests fail ([#3995](#3995)) ([97d74ba](97d74ba)) * **docs:** make Data-dependent conditional execution a complete example ([#4043](#4043)) ([3a1d7f2](3a1d7f2)) * don't build the DAG when running unlock. Fixes [#4000](#4000) and [#198](#198) ([#4007](#4007)) ([acf79fd](acf79fd)) * Ensure pixi tasks may be run as advertised ([#4046](#4046)) ([88253c2](88253c2)) * fix checkpoint handling corner cases ([#3870](#3870) and [#3559](#3559)) ([#4015](#4015)) ([63f4257](63f4257)) * issue 3642 ([#4054](#4054)) ([76e6fc2](76e6fc2)) * issue 3815 ([#4026](#4026)) ([b0eec96](b0eec96)) * logging None in shellcmd context causes error ([#4064](#4064)) ([d0652cd](d0652cd)) * lookup function returns default value for empty DataFrame queries ([#4056](#4056)) ([f71de97](f71de97)) * make `cache: omit-software` a rule specific property ([#4085](#4085)) ([034a9e7](034a9e7)) * reduce number of tests leaving temporary files behind ([#4033](#4033)) ([a3a1c97](a3a1c97)) * regression in dynamic resource handling ([#4038](#4038)) ([f2c554a](f2c554a)) * somewhat shorter announce message ([#4080](#4080)) ([57efc71](57efc71)) ### Performance Improvements * switch reretry with tenacity; decouple container classes (with Python 3.7 compat for old scripts) from rest of the codebase (enabling moving to newer python versions) ([#4032](#4032)) ([ffb19e7](ffb19e7)) ### Documentation * Add AI-assisted contributions policy to contributing guidelines ([#4051](#4051)) ([dd70526](dd70526)) * **codebase:** Update & simplify plugin architecture section ([#4052](#4052)) ([176cf63](176cf63)) * Correct workflow.source_path() description in documentation ([#4036](#4036)) ([45883c5](45883c5)) * fixed wrong code example for collect() function ([#4037](#4037)) ([5c85ed8](5c85ed8)) * Minor docs improvements ([#4089](#4089)) ([29ea226](29ea226)) * switch to sphinx_design for tabs ([#3976](#3976)) ([9674614](9674614)) * typo in the migration table breaking a pip install command ([#4024](#4024)) ([66f9dda](66f9dda)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This PR implements the functionality to access the onstart, onsuccess and onerror directives of modules from the main workflow.
Additionally, it corrects the expected behaviour, that inherited on... directives of the last imported module overwrite non-existing on... directives of the main workflow.
Closes #3920
QC
docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).Summary by CodeRabbit
New Features
Behavior
Tests
Documentation