fix: standardize handling of resources#3421
Conversation
📝 WalkthroughWalkthroughThis pull request introduces a comprehensive refactoring of Snakemake's resource management system. It replaces legacy Changes
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120+ minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ab20f60 to
f98ac7b
Compare
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 1
🧹 Nitpick comments (22)
src/snakemake/workflow.py (1)
166-166: Validate the implications of assigning resources by reference.
self.global_resources = self.resource_settings.resourcescreates a direct reference rather than a copy. If unintentional shared state is not desired later, consider using a copy; otherwise, this is fine if continued synchronization is intended.- self.global_resources = self.resource_settings.resources + self.global_resources = deepcopy(self.resource_settings.resources)src/snakemake/rules.py (1)
1027-1031: Consider chaining the original exception.According to B904, raising a new exception in an
exceptblock can be improved by chaining the old exception (raise ... from err). If the original context is unnecessary, you could also usefrom None.-except ResourceValidationError: - raise WorkflowError( +except ResourceValidationError as err: + raise WorkflowError(🧰 Tools
🪛 Ruff (0.8.2)
1027-1031: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
src/snakemake/cli.py (1)
183-202: Consider removing obsolete code.The commented-out code appears fully replaced by the new resource parsing logic. To keep the codebase clean, consider removing it rather than commenting it out.
- # (Old commented-out code here) - # ... + # Remove this entire commented section to avoid confusiontests/common.py (1)
442-446: Clarity in test failure logic.The new checks handle boolean vs. exception usage correctly. The nested condition can be slightly confusing but remains logically sound.
src/snakemake/settings/types.py (1)
353-354: Default resource mode.Setting
Resources.default("bare")is likely correct for minimal resource usage. Ensure it’s intentional versus"full"elsewhere for consistency.src/snakemake/jobs.py (1)
122-132: Use the local variablevfor consistency.
Within the dictionary comprehension, you already unpack(k, v). For clarity and readability, prefer referencingvinstead ofself.resources[k].- res_dict = { - k: v - for k, v in self.resources.items() - if not self.dag.workflow.resource_scopes.is_local(k) - and not isinstance(self.resources[k], TBDString) - } + res_dict = { + k: v + for k, v in self.resources.items() + if not self.dag.workflow.resource_scopes.is_local(k) + and not isinstance(v, TBDString) + }tests/tests.py (4)
800-801: Clarify removed assertion snippet.The commented-out assertion in line 801 might still be relevant or helpful for debugging. Either remove it entirely or convert it into an explanatory comment if it contains important context:
-# assert {res: resources[res] for res in expected_resources} == expected_resources +# If needed in debugging, consider providing some clarifying context here
1054-1058: Eliminate ambiguous variable name and optimize comparison.Instead of
landnot l == "\n", rename the loop variable to something descriptive and use!=for clarity. Example fix:- lines = [l for l in f.readlines() if not l == "\n"] + lines = [line_content for line_content in f.readlines() if line_content != "\n"]🧰 Tools
🪛 Ruff (0.8.2)
1054-1054: Ambiguous variable name:
l(E741)
1054-1054: Use
l != "\n"instead ofnot l == "\n"Replace with
!=operator(SIM201)
1131-1131: Remove or refine explanatory comment."Implicitly tests the normalization of mem_mb and disk_mb..." can be clarified further or removed if self-explanatory. Comment clarity aids future maintainers.
1148-1148: Consider verifying all suffixed forms.You assert combined resource fields (
mem=60000,disk=3000). Adding checks for_mibor subtle edge cases would ensure that normalizations remain robust across future code changes.src/snakemake/resources.py (12)
2-29: Remove unused imports.Imports such as
TypeVar,UserDict,Dict,Generic,Tuple,Union,isinappear unused and can be removed to keep the code clean.Apply an example diff:
- from ast import TypeVar - from collections import UserDict, defaultdict + from collections import defaultdict ... - from typing import TYPE_CHECKING, Any, Callable, Dict, ... + from typing import TYPE_CHECKING, Any, Callable, ... ... - from numpy import isin🧰 Tools
🪛 Ruff (0.8.2)
2-2:
ast.TypeVarimported but unusedRemove unused import:
ast.TypeVar(F401)
3-3:
collections.UserDictimported but unusedRemove unused import:
collections.UserDict(F401)
16-16:
typing.Dictimported but unusedRemove unused import
(F401)
17-17:
typing.Genericimported but unusedRemove unused import
(F401)
23-23:
typing.Tupleimported but unusedRemove unused import
(F401)
25-25:
typing.Unionimported but unusedRemove unused import
(F401)
29-29:
numpy.isinimported but unusedRemove unused import:
numpy.isin(F401)
135-137: Proper initialization oftotal_resourcesandblocks.Using
defaultdict(int)fortotal_resourcesplus an ordinary list forblocksis consistent. Confirm that these data structures are reset for each new group job evaluation to avoid stale state if the method is reused.
141-144: Check for empty sibling blocks.
block_resources,job_resources, andpipe_resourcesare all set up well. Provide test coverage for empty sibling sets or no pipe groups to confirm stable handling of degenerate cases.
190-190: Centralize local variable naming.
resource_types = list(set(it.chain(*job_resources)))is clear though a bit nested. Consider a short comment clarifying it merges all resource keys from multiple jobs for readability.
219-219: Validatesortbyusage.Bringing in
sortbyfrom the constraints ensures layered scheduling. Consider logging or debugging prints if resource layering leads to unexpected group job expansions.
386-390: Clarify_proportionand_highest_proportiondocstrings.These helper functions are crucial for layering logic. Add docstrings or comments to highlight that
0is returned if there is no constraint.
394-394: Initializingrows.
rows: list[list[tuple[int, ...]]] = [[]]works for the layering approach. Include a quick comment clarifying why you start with a single empty row rather than an empty list for better maintainability.
431-433: Warn on single-resource overshoot.Exceeding constraints in pipe jobs is flagged here. The current messaging is succinct. If possible, mention the job or pipe group name for easier debugging.
596-596: Check type comparisons.Instead of
if type(self_val) != type(other_val):, preferif not isinstance(self_val, type(other_val)):for clarity and best practice:-if type(self_val) != type(other_val): +if not isinstance(self_val, type(other_val)):🧰 Tools
🪛 Ruff (0.8.2)
596-596: Use
isandis notfor type comparisons, orisinstance()for isinstance checks(E721)
754-765: Raise exceptions with explicit cause.Pythonic style recommends chaining exceptions for clarity:
-except Exception as e: - raise WorkflowError(...) +except Exception as e: + raise WorkflowError(...) from e🧰 Tools
🪛 Ruff (0.8.2)
754-765: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
805-805: Use 'is None' checks.Line 805 compares
if mapping == None:; preferif mapping is None:to comply with Python style.🧰 Tools
🪛 Ruff (0.8.2)
805-805: Comparison to
Noneshould becond is NoneReplace with
cond is None(E711)
856-856: Consider using exception chaining.When re-raising a caught exception, do:
raise ValueError("...") from err🧰 Tools
🪛 Ruff (0.8.2)
856-856: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
🛑 Comments failed to post (1)
src/snakemake/resources.py (1)
293-299: 🛠️ Refactor suggestion
Avoid mutable defaults in function parameters.
skip: list[str] = []andmethods: dict[str, Callable[[list[int]], int]] = {}can introduce subtle bugs. UseNoneand initialize inside the function:-def _merge_resource_dict( - cls, - resources: list[dict[str, str | int]], - skip: list[str] = [], - methods: dict[str, Callable[[list[int]], int]] = {}, - default_method: Callable[[list[int]], int] = max, -): +def _merge_resource_dict( + cls, + resources: list[dict[str, str | int]], + skip: Optional[list[str]] = None, + methods: Optional[dict[str, Callable[[list[int]], int]]] = None, + default_method: Callable[[list[int]], int] = max, +): + if skip is None: + skip = [] + if methods is None: + methods = {}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def _merge_resource_dict( cls, resources: list[dict[str, str | int]], skip: Optional[list[str]] = None, methods: Optional[dict[str, Callable[[list[int]], int]]] = None, default_method: Callable[[list[int]], int] = max, ): if skip is None: skip = [] if methods is None: methods = {}🧰 Tools
🪛 Ruff (0.8.2)
296-296: Do not use mutable data structures for argument defaults
Replace with
None; initialize within function(B006)
297-297: Do not use mutable data structures for argument defaults
Replace with
None; initialize within function(B006)
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (12)
src/snakemake/workflow.py (2)
1764-1777: Enhanced threads handling with proper error handling.The try-except block catches
ResourceValidationErrorwith a clear error message. The conditional logic for thread assignment is well-structured and clear.However, there's a potential improvement for the exception handling:
- except ResourceValidationError: - raise RuleException( + except ResourceValidationError as err: + raise RuleException( "Threads value has to be an integer, float, or a callable.", rule=rule, - ) + ) from errAdding
from errto the exception would preserve the exception chain for better debugging.🧰 Tools
🪛 Ruff (0.8.2)
1773-1776: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
1778-1791: Improved resource validation.The updated resource handling code properly validates resources and provides clear error messages. The error message includes the type of the problematic resource.
Similarly to the previous comment, consider preserving the exception chain:
- raise RuleException( + raise RuleException( "Resources values have to be integers, strings, or callables " f"(functions): type({err.args[0]} : {err.args[1]}) == " f"{type(err.args[1])}", rule=rule, - ) + ) from err🧰 Tools
🪛 Ruff (0.8.2)
1785-1790: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
src/snakemake/rules.py (2)
1031-1058: Improved resource evaluation with clear error handling.The new
evaluatefunction is a significant improvement:
- It properly handles resource evaluation with optional skipping
- Provides clear error messages for validation failures
- Uses the new resource types consistently
The error handling is comprehensive, capturing both
ResourceValidationErrorand general exceptions.Consider preserving the exception chain for clearer debugging in these exception handlers:
- except ResourceValidationError: - raise WorkflowError( + except ResourceValidationError as err: + raise WorkflowError( f"Resource {resource} is neither int, float (would be rounded to " "nearest int), str, or None.", rule=self, - ) + ) from errAnd similarly for the
BaseExceptionhandler.🧰 Tools
🪛 Ruff (0.8.2)
1049-1053: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
1068-1094: Robust resource evaluation and constraint checking.This section properly evaluates resources, handles None values, and applies constraints from global resources with appropriate type checking. The error message for type mismatches is clear and informative.
Again, consider preserving the exception chain:
- except ResourceConstraintError: + except ResourceConstraintError as err: global_val = global_res.value global_type = ( "an int" if isinstance(global_val, int) else type(global_val) ) - raise WorkflowError( + raise WorkflowError( f"Resource {resource} is of type {type(val.value).__name__} but " f"global resource constraint defines {global_type} with value " f"{global_val}. Resources with the same name need to have the same " "types (int, float, or str are allowed).", rule=self, - ) + ) from err🧰 Tools
🪛 Ruff (0.8.2)
1086-1092: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
tests/tests.py (1)
1136-1136: Fix spelling error in commentThere's a typo in the comment: "Implicitely" should be "Implicitly".
- # Implicitely tests the normalization of mem_mb and disk_mb to mem and disk + # Implicitly tests the normalization of mem_mb and disk_mb to mem and disk🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 1136-1136:
Implicitely ==> Implicitly🪛 GitHub Actions: Codespell
[error] 1136-1136: codespell: Implicitely ==> Implicitly
src/snakemake/resources.py (7)
1-29: Clean up unused importsSeveral imports are declared but never used in the code, which can lead to maintainability issues.
from __future__ import annotations -from ast import TypeVar from collections import UserDict, defaultdict import copy from dataclasses import dataclass from humanfriendly import InvalidTimespan, InvalidSize, parse_size, parse_timespan import itertools as it import operator as op import re import tempfile import math from typing import ( TYPE_CHECKING, Any, Callable, - Dict, - Generic, Iterator, Literal, Mapping, Self, Sequence, - Tuple, TypeAlias, - Union, cast, ) -from numpy import isin🧰 Tools
🪛 Ruff (0.8.2)
2-2:
ast.TypeVarimported but unusedRemove unused import:
ast.TypeVar(F401)
3-3:
collections.UserDictimported but unusedRemove unused import:
collections.UserDict(F401)
16-16:
typing.Dictimported but unusedRemove unused import
(F401)
17-17:
typing.Genericimported but unusedRemove unused import
(F401)
23-23:
typing.Tupleimported but unusedRemove unused import
(F401)
25-25:
typing.Unionimported but unusedRemove unused import
(F401)
29-29:
numpy.isinimported but unusedRemove unused import:
numpy.isin(F401)
525-538: Replace if-else with ternary in repr methodA small optimization opportunity in the repr method.
def __repr__(self): - if self.is_evaluable(): - value = "function(...)" - else: - value = self._value + value = "function(...)" if self.is_evaluable() else self._value return f'Resource("{self.name}", {value})'🧰 Tools
🪛 Ruff (0.8.2)
526-529: Use ternary operator
value = "function(...)" if self.is_evaluable() else self._valueinstead ofif-else-blockReplace
if-else-block withvalue = "function(...)" if self.is_evaluable() else self._value(SIM108)
566-577: Fix typo in docstringThere's a typo in the docstring for the evaluate method.
def evaluate(self, *args: Any, **kwargs: Any): """Evaluate the resource with given args. This method ensures the returned resource is correctly returned in a - new ``Resource`` class, validated, and standardized. Resources should thus - only be evaulated via this method. + new ``Resource`` class, validated, and standardized. Resources should thus + only be evaluated via this method. """🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 571-571:
evaulated ==> evaluated
674-720: Fix typo in docstringThere's a typo in the docstring for the from_cli_expression method.
@classmethod def from_cli_expression( cls, name: str, value: str, *, with_threads_arg: bool = True ): """Create a new evaluable resource based on a python expression. Threads can optionally be included in the expression environment Arguments ========= name: name of the resource value: - python expression to be evaulated + python expression to be evaluated with_threads_arg: boolean If True, include ``threads`` as an argument in the returned function """🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 687-687:
evaulated ==> evaluated
898-965: Fix typo in docstringThere's a typo in the docstring for the parse method.
@classmethod def parse( cls, exprs: list[str], *, allow_expressions: bool = False, only_positive_integers: bool = False, defaults: None | Literal["bare"] | Literal["full"] = None, ): """Parse a series of CLI-style string resource assigments. - """ + """Parse a series of CLI-style string resource assignments. + """🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 907-907:
assigments ==> assignments
487-487: Fix typo in docstringThere's a typo in the docstring for the Resource class.
"""Standardized representation of a resource name and value. In addition to ``int`` and ``str``, the standard resource types, ``Resource`` can store ``None``, signifying an unset resource, and a callable, signifying a resource to be calculated in the context of a rule. Via the ``.value`` property, ``Resource`` -ensures a resource has been computed before its value is retreived. +ensures a resource has been computed before its value is retrieved. """🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 487-487:
retreived ==> retrieved
492-492: Fix typo in docstringThere's a typo in the docstring for the Resource class.
The class additionally handles input validation and standardization. ``float``s get rounded into ``int``. If the resource name corresponds to a size resource ("disk" and "mem") or a timespan resource ("runtime"), strings values -will be intepreted as a human readable resource and converted to an integer of +will be interpreted as a human readable resource and converted to an integer of appropriate denomination (Mb for size, hr for timespan).🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 492-492:
intepreted ==> interpreted
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (18)
src/snakemake/api.py(2 hunks)src/snakemake/cli.py(5 hunks)src/snakemake/common/__init__.py(1 hunks)src/snakemake/exceptions.py(2 hunks)src/snakemake/io.py(5 hunks)src/snakemake/jobs.py(5 hunks)src/snakemake/logging.py(2 hunks)src/snakemake/resources.py(16 hunks)src/snakemake/rules.py(6 hunks)src/snakemake/scheduler.py(5 hunks)src/snakemake/script/__init__.py(1 hunks)src/snakemake/settings/types.py(2 hunks)src/snakemake/workflow.py(7 hunks)tests/common.py(6 hunks)tests/test_all_temp/Snakefile(0 hunks)tests/test_default_resources/Snakefile(1 hunks)tests/test_group_jobs_resources/Snakefile(2 hunks)tests/tests.py(17 hunks)
💤 Files with no reviewable changes (1)
- tests/test_all_temp/Snakefile
🚧 Files skipped from review as they are similar to previous changes (10)
- tests/test_default_resources/Snakefile
- tests/test_group_jobs_resources/Snakefile
- src/snakemake/script/init.py
- src/snakemake/common/init.py
- src/snakemake/logging.py
- src/snakemake/api.py
- src/snakemake/scheduler.py
- tests/common.py
- src/snakemake/io.py
- src/snakemake/jobs.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
src/snakemake/workflow.pysrc/snakemake/exceptions.pysrc/snakemake/rules.pysrc/snakemake/settings/types.pysrc/snakemake/resources.pytests/tests.pysrc/snakemake/cli.py
🪛 Ruff (0.8.2)
src/snakemake/workflow.py
1773-1776: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
1785-1790: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/rules.py
1049-1053: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
1086-1092: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/resources.py
2-2: ast.TypeVar imported but unused
Remove unused import: ast.TypeVar
(F401)
3-3: collections.UserDict imported but unused
Remove unused import: collections.UserDict
(F401)
16-16: typing.Dict imported but unused
Remove unused import
(F401)
17-17: typing.Generic imported but unused
Remove unused import
(F401)
23-23: typing.Tuple imported but unused
Remove unused import
(F401)
25-25: typing.Union imported but unused
Remove unused import
(F401)
29-29: numpy.isin imported but unused
Remove unused import: numpy.isin
(F401)
296-296: Do not use mutable data structures for argument defaults
Replace with None; initialize within function
(B006)
297-297: Do not use mutable data structures for argument defaults
Replace with None; initialize within function
(B006)
526-529: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block
Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value
(SIM108)
596-596: Use is and is not for type comparisons, or isinstance() for isinstance checks
(E721)
754-765: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
805-805: Comparison to None should be cond is None
Replace with cond is None
(E711)
856-856: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
870-870: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
tests/tests.py
1059-1059: Ambiguous variable name: l
(E741)
1059-1059: Use l != "\n" instead of not l == "\n"
Replace with != operator
(SIM201)
1085-1085: Ambiguous variable name: l
(E741)
1085-1085: Use l != "\n" instead of not l == "\n"
Replace with != operator
(SIM201)
🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py
[failure] 487-487:
retreived ==> retrieved
[failure] 492-492:
intepreted ==> interpreted
[failure] 571-571:
evaulated ==> evaluated
[failure] 687-687:
evaulated ==> evaluated
[failure] 907-907:
assigments ==> assignments
tests/tests.py
[failure] 1136-1136:
Implicitely ==> Implicitly
🪛 GitHub Actions: Codespell
tests/tests.py
[error] 1136-1136: codespell: Implicitely ==> Implicitly
⏰ Context from checks skipped due to timeout of 90000ms (5)
- GitHub Check: tests (9, windows-latest, py311, bash)
- GitHub Check: tests (7, windows-latest, py312, bash)
- GitHub Check: tests (7, ubuntu-latest, py312, bash)
- GitHub Check: tests (4, windows-latest, py312, bash)
- GitHub Check: tests (4, ubuntu-latest, py312, bash)
🔇 Additional comments (73)
src/snakemake/exceptions.py (4)
10-10: Update to type imports is appropriate.Adding
Sequencefrom typing module is necessary for proper type annotation in the new exception classes.
609-610: Good addition of ResourceValidationError class.This custom exception type helps distinguish resource validation errors from other type errors, making error handling more specific.
613-614: ResourceConstraintError provides clear error categorization.This exception helps distinguish resource constraint violations from general type errors, improving error clarity.
617-624: Well-structured ResourceDuplicationError with helpful error message.This exception properly captures duplicate resource information and constructs a clear error message, making debugging easier for users.
src/snakemake/workflow.py (7)
58-61: Appropriate import of new ResourceValidationError class.Adding the import for the new exception class is necessary for proper error handling in resource validation.
118-118: Updated import for Resources components.Using the new
Resourcesimport path is consistent with the architectural changes made to the resource handling system.
166-166: Simplified global_resources initialization.Directly assigning
self.resource_settings.resourcesis more straightforward than creating a new dictionary. This reflects the shift to using theResourcesclass as a container.
569-569: Updated cores retrieval to use the value property.This change is consistent with the new
Resourcesclass structure, which encapsulates resource values with additional metadata.
573-573: Updated nodes retrieval to use the value property.Similar to the cores update, this change correctly accesses the encapsulated value from the
Resourcesclass.
1758-1761: Improved resource initialization.The change from deepcopy to simple copy for default resources is appropriate, likely improving performance. When no default resources are provided, properly initializing with an empty
Resourcesobject maintains type consistency.
1793-1794: Simplified resource overwrite logic.Using the update method of the
Resourcesclass provides a cleaner approach to applying rule-specific resource overwrites.src/snakemake/rules.py (6)
28-28: Updated import for ResourceList.Using
ResourceListfrom the io module is appropriate. This reflects the architectural changes in resource handling.
48-53: Consolidated resource imports.Importing the necessary resource-related classes and exceptions from the new resource module improves code organization and makes dependencies explicit.
74-76: Added typing for workflow import.Adding type checking for the Workflow import helps with static type checking and IDE support.
102-102: Updated resources type annotation.Changing the type annotation to
Resources | Noneproperly reflects the new resource implementation.
1059-1066: Well-implemented cores resource constraint.The cores resource is properly evaluated and constrained against the maximum threads setting. The type check ensures thread counts are integers.
1096-1102: Proper handling of memory and disk resources.This adds support for human-readable resources with automatic conversion between units (MB and MiB). The final return of a
ResourceListmaintains type consistency.src/snakemake/settings/types.py (4)
33-33: Updated import for Resources class.Using the new
Resourcesimport is consistent with the architectural changes in resource handling.
344-344: Improved resource type and initialization.Changing the
resourcesattribute to use theResourcesclass and providing a default factory ensures proper initialization and type consistency.
348-349: Updated resource type annotations.The updated type annotations for
overwrite_resourcesanddefault_resourcesreflect the architectural changes, ensuring type safety throughout the codebase.
353-353: Use of Resources.default factory method.Using the static
defaultmethod of theResourcesclass to create default resources is cleaner than direct instantiation and likely encapsulates initialization logic.src/snakemake/cli.py (3)
156-181: Improved resource parsing with cleaner implementationThe function has been completely rewritten to leverage the new Resources class parser, resulting in a more concise and maintainable implementation. The new approach correctly collects multiple assignments for each rule and processes them together, which aligns with the PR objective to centralize resource handling.
@@ Original function parsed key-value pairs and evaluated expressions manually + New implementation uses Resources.parse() and handles expressions consistently
515-517: Updated default value and parser for resources argumentThe default value for
--resourcesis now a Resources object instead of an empty dictionary, and the parser uses the new Resources.parser_factory() method. This is consistent with the centralized resource handling approach.
594-596: Standardized default resources parsingThe default resources parser now uses the Resources.parser_factory with parameters for defaults and expression allowance, ensuring consistent resource handling throughout the codebase.
tests/tests.py (7)
805-806: Simplified assertion function for resourcesThe
assert_resourcesfunction has been simplified to directly compare the entire resources dictionary with expected values, improving clarity and making test failures more informative.
846-870: Added test for MiB resource specificationsNew test verifies that resources can be provided in mebibytes (MiB) rather than just megabytes (MB), which is part of the PR objective to support both mem_mb and mem_mib variants.
873-897: Added test for human-readable resource specificationsNew test ensures that global resources can be specified in human-readable formats (like "60 Gb"), fulfilling the PR objective to support human-readable resources in various contexts.
900-920: Added validation test for resource duplicationThis parametrized test ensures that specifying both prefixed and unprefixed variants of the same resource (e.g., both "mem" and "mem_mb") raises an appropriate error, improving resource specification clarity.
922-945: Added validation tests for suffixed resource formatsThese tests verify that suffixed resources (mem_mb, mem_mib, etc.) cannot be provided in human-readable format in constraints or declarations, enforcing consistent resource specifications.
1153-1154: Updated assertion to include normalized resourcesThe assertion now correctly includes the normalized resource names (mem, disk) along with the original resource names, ensuring consistency with the new resource normalization behavior.
1386-1415: Added tests for human-readable and mebibyte resource specificationsThese new tests verify that default resources can be specified in both human-readable format and mebibytes, supporting the PR's goal of standardizing resource handling across different formats.
src/snakemake/resources.py (42)
55-63: Good job adding proper type annotations to GroupResources.basic_layeredThe changes to the function signature with proper type annotations improve type safety and make the code more self-documenting. The use of
Resourcestype instead of a generic dictionary is a good improvement.
135-145: Improved variable annotations enhance code readabilityThe added type annotations for variables improve the readability and type safety of the code.
190-202: Good improvement in type safety with proper castingThe use of
cast()to properly type the variables improves type safety and makes the code more robust.
205-207: Resources class property access pattern is consistentUsing the
.get()method with.valueproperty access ensures consistent handling of resources.
218-220: Types are properly passed to _get_layers methodThe changes properly pass value list to the _get_layers method, improving type consistency.
256-259: Type annotations added to _get_saturated_resource_error methodThe method signature now has proper type annotations, which improves code readability and type safety.
274-276: Type annotation for _is_string_resource improves clarityAdding proper type annotations for parameters helps document the expected input types.
300-325: Improved type safety in _merge_resource_dict methodThe addition of type annotations for variables in the method body improves type safety and code readability.
328-333: Enhanced type annotations for _merge_resource_layerThe detailed type annotations provide better documentation and compile-time checking of the method's parameters.
347-349: Type annotations added to _check_constraint methodProper type annotations for parameters improve code readability and type safety.
368-373: Well-defined type annotations for _get_layers methodThe method signature now has clear type annotations, which improves code maintainability.
386-391: Good use of local function type annotationsThe inner functions _proportion and _highest_proportion have proper type annotations, enhancing code clarity.
393-400: Type annotation for rows and resource_groups improves clarityAdding proper type annotations helps clarify the data structures used in the algorithm.
430-433: Fixed type annotation for too_high variableThe precise type annotation for the too_high list makes the code more maintainable.
447-474: Well-implemented helper functions for unit conversionThe helper functions
evaluable_from_mb_to_mibandevaluable_from_mib_to_mbare well-implemented and handle type checking properly.
476-479: Good use of type alias and constantsThe
ValidResourcetype alias andSizedResourcesset make the code more maintainable and self-documenting.
481-524: Resource class is well-designed with comprehensive validationThe Resource class provides standardized representation with proper validation and human-readable resource handling.
🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 487-487:
retreived ==> retrieved
[failure] 492-492:
intepreted ==> interpreted
539-551: Good implementation of the value propertyThe value property correctly ensures that only evaluated resources are accessible, preventing potential errors.
552-565: Well-designed is_evaluable and _get_evaluator methodsThese methods provide a clean API for determining if a resource needs evaluation and retrieving the evaluator function.
603-629: Well-implemented to_mib conversion methodThe to_mib method properly handles different resource types and provides appropriate error messages.
630-656: Well-implemented to_mb conversion methodThe to_mb method properly handles different resource types and provides appropriate error messages.
657-673: Clean implementation of without_suffix and with_name methodsThese utility methods provide a clean way to manipulate resource names and handle suffix conversions.
721-767: Good error handling in cli_evaluator methodThe method properly handles different error cases with descriptive error messages.
🧰 Tools
🪛 Ruff (0.8.2)
754-765: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
768-777: Well-implemented _parse_human_friendly methodThe method properly handles parsing human-readable resource values for both sized resources and runtime.
779-803: Resources class is well-designed with clear documentationThe class provides a standardized container for resources with proper documentation of its functionality.
812-822: Good implementation of _normalize_sizes methodThe method properly handles size resource normalization and raises appropriate errors for duplications.
823-836: Good implementation of mapping methodsThe basic mapping methods are well-implemented with proper type checking.
837-842: Good implementation of setitem methodThe method properly handles different input types.
843-850: Well-implemented args propertyThe property correctly filters non-evaluable, non-None resources for argument representation.
851-857: Good error handling in decode_arg methodThe method properly handles invalid input formats with a descriptive error message.
🧰 Tools
🪛 Ruff (0.8.2)
856-856: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
858-878: Good implementation of default methodThe method provides a clean way to initialize resources with default values.
🧰 Tools
🪛 Ruff (0.8.2)
870-870: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
879-897: Well-designed parser_factory methodThe method provides a clean way to create parser functions with preset arguments.
966-975: Good implementation of from_mapping methodThe method provides a clean way to initialize resources from a mapping.
976-1008: Well-implemented utility methodsThe update, keys, values, items, unwrapped_items, unwrapped_nonstr_items, copy, and get methods provide a comprehensive API.
1009-1013: Good use of type alias for ValidScopeThe ValidScope type alias makes the code more maintainable and self-documenting.
1013-1041: Well-documented ResourceScopes classThe class provides clear documentation and proper validation of resource scopes.
1042-1052: Good implementation of _normalize_sizes methodThe method properly handles size resource normalization and raises appropriate errors for duplications.
1053-1056: Good implementation of defaults methodThe method provides sensible default scopes for common resources.
1057-1064: Good implementation of _test methodThe method provides a clean way to test if a resource has a specific scope.
1065-1100: Well-documented is_local, is_global, and is_excluded methodsThese methods have clear documentation explaining their purpose.
1101-1147: Good implementation of locals, globals, and excluded propertiesThese properties provide a clean way to get resources by scope.
1148-1152: Good implementation of update methodThe method properly handles updating from another ResourceScopes or mapping.
|
Fixes #3367 |
08bd255 to
de55285
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (12)
src/snakemake/workflow.py (2)
1767-1780: Improved thread resource handlingThe thread resource handling has been refactored with a try-except block for
ResourceValidationError, providing better error messages when thread values are invalid.However, there's a minor issue with the error handling:
- except ResourceValidationError: - raise RuleException( - "Threads value has to be an integer, float, or a callable.", - rule=rule, - ) + except ResourceValidationError as err: + raise RuleException( + "Threads value has to be an integer, float, or a callable.", + rule=rule, + ) from errAdding
from errpreserves the exception chain, which helps with debugging by showing the original cause.🧰 Tools
🪛 Ruff (0.8.2)
1776-1779: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
1781-1795: Improved resource validation and handlingAdded proper error handling for resource parsing with informative error messages. Now resources are handled through the Resources class API with
Resources.from_mapping()rather than direct dictionary operations.The error handling could be improved by preserving the exception chain:
- except KeyError as err: - raise RuleException( - "Resources values have to be integers, strings, or callables " - f"(functions): type({err.args[0]} : {err.args[1]}) == " - f"{type(err.args[1])}", - rule=rule, - ) + except KeyError as err: + raise RuleException( + "Resources values have to be integers, strings, or callables " + f"(functions): type({err.args[0]} : {err.args[1]}) == " + f"{type(err.args[1])}", + rule=rule, + ) from err🧰 Tools
🪛 Ruff (0.8.2)
1788-1793: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
src/snakemake/rules.py (1)
1068-1094: Enhanced resource evaluation and constraint applicationImproved the handling of all resources with proper evaluation and constraint checking:
- Skips
_coreswhich is handled separately- Only evaluates resources that have evaluation functions
- Properly handles None values
- Applies constraints against global resources with proper error messages
- Includes type checking to ensure resources are of compatible types
This makes resource handling more robust and consistent.
The exception handling at line 1086 could benefit from using
from errto preserve the exception chain:- raise WorkflowError( - f"Resource {resource} is of type {type(val.value).__name__} but " - f"global resource constraint defines {global_type} with value " - f"{global_val}. Resources with the same name need to have the same " - "types (int, float, or str are allowed).", - rule=self, - ) + raise WorkflowError( + f"Resource {resource} is of type {type(val.value).__name__} but " + f"global resource constraint defines {global_type} with value " + f"{global_val}. Resources with the same name need to have the same " + "types (int, float, or str are allowed).", + rule=self, + ) from err🧰 Tools
🪛 Ruff (0.8.2)
1086-1092: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
tests/tests.py (1)
1093-1093: Consider using!=operator and improving variable name clarity.The line uses a negated equality check (
not l == "\n") which could be more clearly written asl != "\n". Additionally, the variable namelis ambiguous and could be renamed to something more descriptive likeline.- lines = [l for l in f.readlines() if not l == "\n"] + lines = [line for line in f.readlines() if line != "\n"]🧰 Tools
🪛 Ruff (0.8.2)
1093-1093: Ambiguous variable name:
l(E741)
1093-1093: Use
l != "\n"instead ofnot l == "\n"Replace with
!=operator(SIM201)
src/snakemake/resources.py (8)
219-220: Usefrom errwhen re-raising exceptions for better traceabilityWhen re-raising an exception from a caught exception, use
raise ... from errto preserve the original exception's traceback, which helps with debugging.except ResourceError as err: - raise ResourceInsufficiencyError(additive_resources, err.args[0]) + raise ResourceInsufficiencyError(additive_resources, err.args[0]) from err🧰 Tools
🪛 Ruff (0.8.2)
220-220: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
508-514: Consider using a ternary operator for better readabilityYou can simplify the if-else block with a ternary operator for a more concise representation.
def __repr__(self): - if self.is_evaluable(): - value = "function(...)" - else: - value = self._value + value = "function(...)" if self.is_evaluable() else self._value return f'Resource("{self.name}", {value})'🧰 Tools
🪛 Ruff (0.8.2)
509-512: Use ternary operator
value = "function(...)" if self.is_evaluable() else self._valueinstead ofif-else-blockReplace
if-else-block withvalue = "function(...)" if self.is_evaluable() else self._value(SIM108)
578-580: Use isinstance() instead of type comparisonUsing
isinstance()is preferred over direct type comparison as it handles inheritance properly. The current implementation could fail ifother_valis a subclass of the type ofself_val.self_val = self.value - if not isinstance(self_val, type(other_val)): + if not isinstance(self_val, type(other_val)) or not isinstance(other_val, type(self_val)): raise ResourceConstraintError(self_val, other_val)
586-638: Consider reducing code duplication between to_mib and to_mb methodsThe
to_mibandto_mbmethods share very similar structure and logic. Consider refactoring to reduce duplication by creating a helper method that handles the common logic.You could create a private helper method like
_convert_unitsthat both methods use:def _convert_units(self, target_suffix, conversion_func, evaluable_wrapper): if isinstance(self._value, TBDString): value = self._value elif self.is_evaluable(): value = evaluable_wrapper(self.name, self._get_evaluator()) elif not isinstance(self._value, int): errmsg = ( f"Resource must be of type 'int' to convert to {target_suffix}. {self.name} == " f"{self._value} (type {type(self._value)})" ) raise TypeError(errmsg) else: value = conversion_func(self._value) return Resource(f"{self.name.removesuffix(target_suffix == 'mib' and '_mb' or '_mib')}_{target_suffix}", value) def to_mib(self): """Convert the resource to mebibytes...""" return self._convert_units('mib', mb_to_mib, evaluable_from_mb_to_mib) def to_mb(self): """Convert the resource to megabytes...""" return self._convert_units('mb', mib_to_mb, evaluable_from_mib_to_mb)
735-748: Usefrom errorfrom Nonewhen re-raising exceptionsWhen re-raising an exception from a caught exception, use
raise ... from errto preserve the original exception's traceback, orfrom Noneto suppress it intentionally.raise WorkflowError( "Failed to evaluate resources value " f"'{val}'.\n" " String arguments may need additional " "quoting. E.g.: --default-resources " "\"tmpdir='/home/user/tmp'\" or " "--set-resources \"somerule:someresource='--nice=100'\". " "This also holds for setting resources inside of a profile, where " "you might have to enclose them in single and double quotes, " "i.e. someresource: \"'--nice=100'\".", - e, + e, + ) from e🧰 Tools
🪛 Ruff (0.8.2)
737-748: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
835-839: Usefrom errwhen re-raising exceptionsWhen re-raising an exception from a caught exception, include
from errto preserve the traceback for better debugging.def decode_arg(cls, arg: str) -> Tuple[str, str]: try: return cast(Tuple[str, str], tuple(arg.split("=", maxsplit=1))) except ValueError: - raise ValueError("Resources have to be defined as name=value pairs.") + raise ValueError("Resources have to be defined as name=value pairs.") from None🧰 Tools
🪛 Ruff (0.8.2)
839-839: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
845-854: Usefrom errwhen re-raising exceptionsInclude
from Nonewhen re-raising an exception to clarify that this is a new exception with no traceback linkage to a previous one.try: default_resources = cls.DEFAULTS[mode or "full"] except KeyError: - raise ValueError(f"Unexpected mode for DefaultResources: {mode}") + raise ValueError(f"Unexpected mode for DefaultResources: {mode}") from None🧰 Tools
🪛 Ruff (0.8.2)
853-853: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
778-786: Consider moving defaults to a configuration fileHardcoded default configurations can be difficult to maintain and change. Consider moving these defaults to a configuration file that can be loaded at runtime.
This would make it easier to update default resources without modifying code, and potentially allow users to customize defaults without code changes.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (19)
src/snakemake/api.py(2 hunks)src/snakemake/cli.py(6 hunks)src/snakemake/common/__init__.py(1 hunks)src/snakemake/dag.py(2 hunks)src/snakemake/exceptions.py(2 hunks)src/snakemake/io.py(7 hunks)src/snakemake/jobs.py(5 hunks)src/snakemake/logging.py(2 hunks)src/snakemake/resources.py(14 hunks)src/snakemake/rules.py(6 hunks)src/snakemake/scheduler.py(5 hunks)src/snakemake/script/__init__.py(1 hunks)src/snakemake/settings/types.py(2 hunks)src/snakemake/workflow.py(7 hunks)tests/common.py(6 hunks)tests/test_all_temp/Snakefile(0 hunks)tests/test_default_resources/Snakefile(1 hunks)tests/test_group_jobs_resources/Snakefile(2 hunks)tests/tests.py(17 hunks)
💤 Files with no reviewable changes (1)
- tests/test_all_temp/Snakefile
✅ Files skipped from review due to trivial changes (1)
- src/snakemake/dag.py
🚧 Files skipped from review as they are similar to previous changes (7)
- tests/test_group_jobs_resources/Snakefile
- src/snakemake/logging.py
- src/snakemake/script/init.py
- src/snakemake/api.py
- src/snakemake/jobs.py
- src/snakemake/common/init.py
- tests/test_default_resources/Snakefile
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
src/snakemake/scheduler.pysrc/snakemake/settings/types.pysrc/snakemake/workflow.pysrc/snakemake/rules.pysrc/snakemake/exceptions.pysrc/snakemake/io.pysrc/snakemake/cli.pysrc/snakemake/resources.pytests/common.pytests/tests.py
🪛 Ruff (0.8.2)
src/snakemake/workflow.py
1776-1779: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
1788-1793: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/rules.py
1086-1092: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/resources.py
220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
509-512: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block
Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value
(SIM108)
737-748: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
839-839: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
853-853: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
tests/common.py
196-196: Undefined name List
(F821)
tests/tests.py
1093-1093: Ambiguous variable name: l
(E741)
1093-1093: Use l != "\n" instead of not l == "\n"
Replace with != operator
(SIM201)
🔇 Additional comments (59)
src/snakemake/scheduler.py (5)
9-9: Adding copy module to support shallow copying of resourcesThe
copymodule is now needed for creating a shallow copy of resources instead of creating a new dictionary, which aligns with the move to the new Resources class.
17-17: Improving type safety with conditional importsAdding TYPE_CHECKING for importing types only during static type checking is a good practice that prevents circular imports at runtime while maintaining proper type checking.
Also applies to: 31-33
60-60: Enhanced type annotation for workflow parameterAdding explicit type hints for the
workflowparameter improves code clarity and enables better IDE support and static type checking.
90-90: Updated resource access methodChanged from using
workflow.global_resources.items()toworkflow.global_resources.unwrapped_nonstr_items(), which is part of the new Resources class API. This method likely returns the primitive values of resources rather than the Resource objects themselves.
99-99: Simplified resource initialization with shallow copyUsing
copy.copy(self.global_resources)rather than manually creating a new dictionary indicates a shift to treating resources as a cohesive object rather than individual key-value pairs. This simplifies the code and ensures all properties of the Resources class are preserved.src/snakemake/workflow.py (6)
58-60: Added ResourceValidationError to exceptionsAdded import for the new
ResourceValidationErrorexception which is used for handling validation errors with resources.
118-119: Updated resource-related importsNow importing
ParsedResource,ResourceScopes, andResourcesfrom the newsnakemake.resourcesmodule, which is part of the centralized resource handling refactoring.
166-169: Simplified global_resources assignmentNow directly assigning
self.resource_settings.resourcestoglobal_resourcesinstead of creating a dictionary from it, which is cleaner and preserves the object-oriented nature of the Resources class.
568-569: Updated resource value accessNow accessing resource values through the
.valueproperty instead of directly, which is consistent with the new Resource object model where values are wrapped in objects with additional metadata.Also applies to: 572-573
629-629: Added return type annotation to get_ruleExplicitly annotating the return type of
get_ruleasRuleimproves type checking and documentation.
1761-1762: Optimized resource copyingUsing
.copy()method instead ofdeepcopy()for resources, which is more efficient when a shallow copy is sufficient. This is appropriate here since the Resources class likely handles its internal state properly during copying.src/snakemake/rules.py (9)
28-28: Updated import to include ResourceListNow importing
ResourceListfromsnakemake.io, which is used as the return type for the refactoredexpand_resourcesmethod.
48-53: Added imports for resource-related classesAdded imports for
Resource,ResourceConstraintError,ResourceValidationError, andResourcesfrom the newsnakemake.resourcesmodule, supporting the centralized resource handling approach.
74-76: Added conditional import for Workflow typeAdded conditional import of
Workflowfor type checking, preventing circular imports at runtime.
90-91: Added type annotation for workflow attributeAdded type annotation for the
workflowattribute, improving type safety and documentation.
102-103: Added type annotation for resources attributeAdded type annotation for the
resourcesattribute, specifying it can be either aResourcesobject orNone.
1031-1058: Implemented standalone evaluate function for resource handlingReplaced the nested
applyfunction with a standaloneevaluatefunction that provides improved handling of resource values:
- Properly handles skipping evaluation based on the
skip_evaluationparameter- Returns a Resource with a TBDString for skipped resources
- Uses proper error handling for ResourceValidationError with detailed error messages
- Handles input function application correctly
This is a significant improvement in the resource evaluation logic.
However, similar to the previous files, the exception handling could preserve the exception chain:
- except ResourceValidationError as err: - raise WorkflowError( - f"Resource {resource} is neither int, float (would be rounded to " - "nearest int), str, or None.", - rule=self, - ) from errThis is correctly using
from errto preserve the exception chain, which is good practice.
1059-1067: Updated thread resource evaluationNow properly evaluates thread resources and applies constraints based on the workflow's max_threads setting. The type checking ensures that thread values are integers, providing better error handling.
1095-1102: Added special handling for memory and disk resourcesImproved resource handling by adding automatic conversion between different memory and disk size units:
- For both
memanddiskresources, automatically sets the_mbvariant with the same value- Also sets the
_mibvariant using a conversion method, ensuring consistency across different unitsThis allows for human-readable resource specifications while maintaining backward compatibility.
1102-1102: Changed return type to ResourceListNow returning a
ResourceListobject instead of a plain dictionary, which is more consistent with the object-oriented approach to resource handling and provides additional methods for working with resources.tests/common.py (4)
154-154: Good enhancement toshouldfailparameter.The type annotation has been improved from just
booltobool | type[Exception], enabling more precise testing by allowing specific exception types to be validated.
195-196: Better type annotations for resource parameters.The type annotations have been updated to reflect the more structured approach to resource handling:
resourcesnow uses a clearerMapping[str, Any] | Nonetypedefault_resourcesnow accepts a dedicatedResourcesobject or a list of strings🧰 Tools
🪛 Ruff (0.8.2)
196-196: Undefined name
List(F821)
336-349: Good implementation of resource handling with the new Resources class.The implementation now properly uses the new
Resourcesclass methods:
Resources.from_mapping()to convert mapping to resources objectResources.parse()for parsing list-based resource definitions- Appropriate conditional handling for different input types
This aligns well with the broader refactoring of resource management.
442-446: Improved exception testing mechanism.The change from a simple truthy check to an explicit
not Falsecheck is more precise. Now whenshouldfailis an exception type rather than justTrue, the code properly uses pytest'sraisescontext manager to verify the specific exception type.src/snakemake/settings/types.py (4)
33-33: Updated import for the new Resources class.Import now references the new consolidated
Resourcesclass instead of the previous resource implementation.
344-344: Good implementation of Resources default factory.Using
field(default_factory=Resources)is a clean way to initialize theresourcesattribute with an emptyResourcesinstance.
348-349: Consistent type updates for resource-related attributes.The type annotations have been updated to use the new
Resourcesclass:
overwrite_resourcesnow mapped toResourcesobjects instead of nested dictionariesdefault_resourcesnow explicitly usesOptional[Resources]This makes the type system more consistent and easier to understand.
353-353: Good replacement of DefaultResources with Resources.default.The initialization of
default_resourcesnow uses the static methodResources.default("bare")instead of creating aDefaultResourcesinstance, maintaining the same functionality while using the new unified resource class.src/snakemake/cli.py (5)
12-12: Updated typing imports.The import statement now includes a more comprehensive set of typing annotations, adding
Dict,List, andUnionwhich are used in the updated code.
43-43: Updated import for the Resources class.Now using the new consolidated
Resourcesclass instead of the previous implementation.
156-182: Significantly simplified resource parsing.The
parse_set_resourcesfunction has been completely rewritten to use the newResources.parsemethod, which simplifies the code considerably:
- Better type annotation (
List[str] | None)- Clearer error handling with specific error messages
- More straightforward logic for parsing rule/resource assignments
- Delegating the actual parsing to
Resources.parseThis refactoring reduces complexity while maintaining functionality.
515-516: Updated resource argument defaults.The default value for the
--resargument now usesResources()directly, and the parsing function usesResources.parser_factory(), making it consistent with the new resource handling approach.
594-596: Enhanced default resources parsing.The
--default-resourcesargument now usesResources.parser_factorywith specific parameters for defaults and expression evaluation, providing better consistency and flexibility.src/snakemake/exceptions.py (5)
10-10: Added Sequence type import.Added
Sequencefrom typing module to support type annotations in the new exception classes.
602-603: Added base ResourceError class.Good addition of a base exception class for all resource-related errors, allowing for more specific error catching.
606-625: Well-designed ResourceInsufficiencyError class.This exception provides detailed information about resource insufficiency errors:
- Takes specific resource types (additive vs excess) as parameters
- Constructs a clear error message explaining the issue
- Properly handles singular/plural grammar in the message
- Stores the resource lists as attributes for potential programmatic access
The error message is particularly helpful as it explains the context (pipe groups) and how resources are calculated.
635-640: Added resource validation exception classes.Added two specialized exception types:
ResourceValidationErrorfor type validation issuesResourceConstraintErrorfor constraint violationsThese will help with more specific error handling and better error messages.
643-650: Well-implemented ResourceDuplicationError.This exception handles the case when equivalent resources are specified multiple times:
- Accepts and stores a list of duplicate resources
- Provides a clear error message explaining why this is problematic
- Inherits from ValueError which is appropriate for this type of error
This will help users understand and fix resource duplication issues more easily.
src/snakemake/io.py (7)
30-32: Appropriate addition of new imports needed for generic types.The addition of
GenericandIteratorfrom the typing module is necessary to support the improvements to the type annotations in the Namedlist class.
35-35: Good addition of Tuple import.Adding
Tupleto the imports is needed for the return type annotation of theitems()method later in the code.
1635-1639: Improved type variable declarations for stronger typing.The changes to
_TNamedListand the addition of_TNamedKeysimprove type safety and flexibility:
- Removing the bound to
Namedlistfrom_TNamedListallows for more generic usage- The new type variable
_TNamedKeysenables proper typing of dictionary-like operations- Clear docstrings explain their purpose for self-returning methods
These changes prepare for making Namedlist properly generic.
1642-1642: Enhanced Namedlist with proper generic typing.The Namedlist class now properly inherits from
list[_TNamedList]and usesGeneric[_TNamedKeys, _TNamedList], which significantly improves type safety. This will provide better IDE support and static type checking capabilities.
1651-1651: Improved type annotation for fromdict parameter.The
fromdictparameter now has a more precise type annotationOptional[Dict[_TNamedKeys, _TNamedList]]which clearly indicates that the dictionary keys and values should match the generic types of the Namedlist.
1755-1755: Added return type annotation for items() method.The return type annotation
Iterator[Tuple[_TNamedKeys, _TNamedList]]for theitems()method clarifies that it returns an iterator of key-value pairs that match the generic types of the Namedlist. This improves type safety and documentation.
1869-1870: Renamed Resources class to ResourceList with appropriate generic parameters.Renaming
ResourcestoResourceListindicates the actual purpose of this class as a list of resources, which avoids confusion with the newResourcesclass mentioned in the PR objectives. The generic type parameters[str, str | int]correctly specify that keys are strings and values can be either strings or integers.tests/tests.py (12)
16-16: Added import for ResourceDuplicationError.This import is needed for the new tests that verify the correct handling of duplicated resource specifications.
18-18: Updated resources imports to use the new Resources class.The import has been updated to include both
GroupResourcesand the new unifiedResourcesclass, replacing what was likely aDefaultResourcesimport previously, aligning with the PR objective of centralizing resource handling.
814-814: Simplified assertion for better readability.The
assert_resourcesfunction has been simplified to directly compare the resources dictionary with expected resources, making the code more straightforward and easier to maintain.
835-851: Updated resource handling to use the new Resources interface.This test has been updated to:
- Use a list of resource strings instead of DefaultResources
- Add new resource types (mem_mib, disk, disk_mb, disk_mib)
- Use a simpler assertion with direct dictionary comparison
These changes align with the PR objectives of standardizing resource handling and supporting various resource formats.
854-878: Added test for resources provided in mebibytes (MiB).This new test ensures that resources can be correctly specified in mebibytes (MiB) and are properly converted to their megabyte (MB) counterparts, as mentioned in the PR objectives. The test verifies that specifying
mem_mibresults in correct values for bothmem_mbandmem_mib.
881-905: Added test for human-readable resource specifications.This test verifies that global resources can be specified in human-readable format (e.g., "60 Gb") and are correctly converted to their numeric equivalents. This directly addresses one of the PR objectives of supporting human-readable resources in global resource limits.
908-927: Added test to prevent specifying the same resource in multiple formats.This parametrized test ensures that an error is raised when the same effective resource is specified in multiple formats (e.g., both
memandmem_mb), directly addressing the PR objective to raise an error in such cases.
930-940: Added test to prevent human-readable format for suffixed resources.This test verifies that an appropriate error is raised when attempting to specify a suffixed resource constraint (mem_mb) in human-readable format, which helps enforce consistent resource handling.
942-952: Added test for preventing human-readable resources in rule declarations.This test ensures that rule resource specifications cannot use human-readable formats for suffixed resources, maintaining consistency in how resources are defined throughout the codebase.
966-968: Updated multiple tests to use the new Resources.parse method.The tests have been updated to use the new
Resources.parsemethod instead of the previousDefaultResourcesclass, ensuring that all tests consistently use the new resource handling mechanism. The calls now include parameters likedefaults="full"andallow_expressions=Truewhich provide more flexibility in resource specification.Also applies to: 996-998, 1035-1037, 1061-1063, 1087-1089, 1113-1115, 1135-1136, 1156-1158, 1179-1181, 1206-1208, 1237-1239
1406-1424: Added test for human-readable default resources.This test verifies that default resources can be specified in human-readable format (e.g., "1 Gb"), ensuring this feature works consistently across the codebase.
1073-1096: Added test to verify resource prefix handling in scope overwrite.This test ensures that overwriting a resource scope works correctly regardless of whether the base resource name or a prefixed version is used (e.g.,
memvsmem_mb), which is important for consistent resource handling.🧰 Tools
🪛 Ruff (0.8.2)
1093-1093: Ambiguous variable name:
l(E741)
1093-1093: Use
l != "\n"instead ofnot l == "\n"Replace with
!=operator(SIM201)
src/snakemake/resources.py (2)
276-281: Good refactoring of mutable default argumentsReplacing mutable default arguments with
Nonedefaults and initializing them within the function is a good practice that avoids unexpected behavior.
459-1130: Excellent implementation of the new Resource management systemThe comprehensive refactoring of resource handling with the new
Resource,Resources, andResourceScopesclasses provides a robust and type-safe system. The implementation correctly handles:
- Human-readable resources
- Proper enforcement of memory and disk limits
- Conversion between MB and MiB units
- Error cases for duplicate resource specifications
- Default resource handling
The extensive type annotations also improve code clarity and maintainability.
🧰 Tools
🪛 Ruff (0.8.2)
509-512: Use ternary operator
value = "function(...)" if self.is_evaluable() else self._valueinstead ofif-else-blockReplace
if-else-block withvalue = "function(...)" if self.is_evaluable() else self._value(SIM108)
737-748: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
839-839: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
853-853: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (3)
src/snakemake/resources.py (3)
273-282: 🛠️ Refactor suggestionFix mutable default arguments
Using mutable default arguments (empty lists or dictionaries) can lead to unexpected behavior when the function is called multiple times.
def _merge_resource_dict( cls, resources: List[Dict[str, str | int]], - skip: Optional[List[str]] = None, - methods: Optional[Dict[str, Callable[[List[int]], int]] = None, + skip: Optional[List[str]] = None, + methods: Optional[Dict[str, Callable[[List[int]], int]] = None, default_method: Callable[[List[int]], int] = max, ): skip = skip or [] methods = methods or {}
589-595: 🛠️ Refactor suggestionUse isinstance() instead of type comparison
The constrain method uses direct type comparison, which should be replaced with isinstance().
def constrain(self, other: Resource | int | None): """Use ``other`` as the maximum value for ``Self``, but only if both are integers. Returns self unaltered if either resource is ``None`` or if both are the same non-integer type. Errors if ``other`` has a different type from ``Self``, or if either are evaluable resources. """ if isinstance(self._value, TBDString): return self other_val = other.value if isinstance(other, Resource) else other if self._value is None or other_val is None: return self self_val = self.value - if not isinstance(self_val, type(other_val)): + if not isinstance(self_val, type(other_val)) or not isinstance(other_val, type(self_val)): raise ResourceConstraintError(self_val, other_val) if isinstance(self_val, int): assert isinstance(other_val, int) return Resource(self.name, min(other_val, self_val)) return self
799-802: 🛠️ Refactor suggestionUse "is None" instead of "== None"
Using
is Noneis the proper way to check for None in Python.def __init__(self, mapping: Dict[str, Resource] | None = None): - if mapping == None: + if mapping is None: self._data: Dict[str, Resource] = {} return self._data = mapping
🧹 Nitpick comments (8)
src/snakemake/resources.py (8)
29-29: Remove unused importsThe
get_function_paramsandoverwrite_function_paramsare imported but not used in this file.-from snakemake.common import get_function_params, get_input_function_aux_params, mb_to_mib, mib_to_mb, overwrite_function_params +from snakemake.common import get_input_function_aux_params, mb_to_mib, mib_to_mb🧰 Tools
🪛 Ruff (0.8.2)
29-29:
snakemake.common.get_function_paramsimported but unusedRemove unused import
(F401)
29-29:
snakemake.common.overwrite_function_paramsimported but unusedRemove unused import
(F401)
477-477: Fix typo in docstringThe word "retreived" is misspelled and should be "retrieved".
- ensures a resource has been computed before its value is retreived. + ensures a resource has been computed before its value is retrieved.🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 477-477:
retreived ==> retrieved🪛 GitHub Actions: Codespell
[error] 477-477: codespell: 'retreived' should be 'retrieved'.
482-482: Fix typo in docstringThe word "intepreted" is misspelled and should be "interpreted".
- will be intepreted as a human readable resource and converted to an integer of + will be interpreted as a human readable resource and converted to an integer of🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 482-482:
intepreted ==> interpreted
564-564: Fix typo in docstringThe word "evaulated" is misspelled and should be "evaluated".
- standardized. Resources should thus only be evaulated via this method. + standardized. Resources should thus only be evaluated via this method.🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 564-564:
evaulated ==> evaluated
680-680: Fix typo in docstringThe word "evaulated" is misspelled and should be "evaluated".
- value: python expression to be evaulated + value: python expression to be evaluated🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 680-680:
evaulated ==> evaluated
901-903: Fix typo in docstringThe word "assigments" is misspelled and should be "assignments".
- """Parse a series of CLI-style string resource assigments. + """Parse a series of CLI-style string resource assignments.🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 901-901:
assigments ==> assignments
220-220: Improve exception chain handlingWhen raising an exception in an except block, you should use
raise ... from errto preserve the exception chain for better debugging.try: layers = cls._get_layers( int_resources, list(sorted_constraints.values()), sortby ) except ResourceError as err: - raise ResourceInsufficiencyError(additive_resources, err.args[0]) + raise ResourceInsufficiencyError(additive_resources, err.args[0]) from err🧰 Tools
🪛 Ruff (0.8.2)
220-220: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
529-533: Use ternary operator for conditional assignmentYou can simplify this conditional assignment using a Python ternary operator.
def __repr__(self): - if self.is_evaluable(): - value = "function(...)" - else: - value = self._value + value = "function(...)" if self.is_evaluable() else self._value return f'Resource("{self.name}", {value})'🧰 Tools
🪛 Ruff (0.8.2)
529-532: Use ternary operator
value = "function(...)" if self.is_evaluable() else self._valueinstead ofif-else-blockReplace
if-else-block withvalue = "function(...)" if self.is_evaluable() else self._value(SIM108)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
src/snakemake/common/__init__.py(3 hunks)src/snakemake/io.py(8 hunks)src/snakemake/resources.py(14 hunks)tests/common.py(6 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/common.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
src/snakemake/common/__init__.pysrc/snakemake/io.pysrc/snakemake/resources.py
🪛 Ruff (0.8.2)
src/snakemake/resources.py
29-29: snakemake.common.get_function_params imported but unused
Remove unused import
(F401)
29-29: snakemake.common.overwrite_function_params imported but unused
Remove unused import
(F401)
220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
529-532: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block
Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value
(SIM108)
748-759: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
850-850: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
864-864: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py
[failure] 477-477:
retreived ==> retrieved
[failure] 482-482:
intepreted ==> interpreted
[failure] 564-564:
evaulated ==> evaluated
[failure] 680-680:
evaulated ==> evaluated
[failure] 901-901:
assigments ==> assignments
🪛 GitHub Actions: Codespell
src/snakemake/resources.py
[error] 477-477: codespell: 'retreived' should be 'retrieved'.
⏰ Context from checks skipped due to timeout of 90000ms (40)
- GitHub Check: tests (10, windows-latest, py312)
- GitHub Check: tests (10, windows-latest, py311)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (9, windows-latest, py312)
- GitHub Check: tests (9, windows-latest, py311)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (8, windows-latest, py312)
- GitHub Check: tests (8, windows-latest, py311)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (7, windows-latest, py312)
- GitHub Check: tests (7, windows-latest, py311)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (6, windows-latest, py312)
- GitHub Check: tests (6, windows-latest, py311)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (5, windows-latest, py312)
- GitHub Check: tests (5, windows-latest, py311)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (4, windows-latest, py312)
- GitHub Check: tests (4, windows-latest, py311)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (3, windows-latest, py312)
- GitHub Check: tests (3, windows-latest, py311)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (2, windows-latest, py312)
- GitHub Check: tests (2, windows-latest, py311)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (1, windows-latest, py312)
- GitHub Check: tests (1, windows-latest, py311)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py311)
🔇 Additional comments (12)
src/snakemake/resources.py (4)
430-457: Well-implemented resource unit conversionThese helper functions properly handle the conversion between megabytes and mebibytes for evaluable resources, including appropriate error handling for invalid return types.
471-670: Well-designed Resource classThe
Resourceclass provides a comprehensive solution for standardizing resource representation, validation, and conversion. It supports different resource types (int, str, callable) and provides proper error handling.🧰 Tools
🪛 Ruff (0.8.2)
529-532: Use ternary operator
value = "function(...)" if self.is_evaluable() else self._valueinstead ofif-else-blockReplace
if-else-block withvalue = "function(...)" if self.is_evaluable() else self._value(SIM108)
🪛 GitHub Check: Check for spelling errors
[failure] 477-477:
retreived ==> retrieved
[failure] 482-482:
intepreted ==> interpreted
[failure] 564-564:
evaulated ==> evaluated🪛 GitHub Actions: Codespell
[error] 477-477: codespell: 'retreived' should be 'retrieved'.
773-996: Well-implemented Resources container classThe
Resourcesclass provides a standardized container for resources with proper handling of resource types, parsing of CLI-style resource assignments, and management of default resources. The implementation follows the Mapping protocol correctly.🧰 Tools
🪛 Ruff (0.8.2)
850-850: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
864-864: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
🪛 GitHub Check: Check for spelling errors
[failure] 901-901:
assigments ==> assignments
1007-1146: Well-structured ResourceScopes classThe
ResourceScopesclass appropriately manages resource scopes (local, global, excluded) with comprehensive methods for checking scopes and properties for retrieving resources by scope.src/snakemake/common/__init__.py (2)
64-70: Good implementation of megabyte-mebibyte conversion functionsThe implementation of
mb_to_mibandmib_to_mbfunctions provides a clear and consistent way to convert between these units, with proper rounding behavior (ceiling for mb to mib, flooring for mib to mb).
308-320: Improved type annotations for auxiliary parameter extractionThe
get_input_function_aux_paramsfunction now has proper type annotations, which improves code readability and type safety. The function correctly filters parameters based on what the provided function can accept.src/snakemake/io.py (6)
1-1: Good addition of future annotations importAdding the
__future__import for annotations allows for more flexible type hinting throughout the file.
1636-1640: Improved type variable declarationsThe type variables
_TNamedListand_TNamedKeysare now properly documented with clear comments explaining their purpose.
1643-1643: Enhanced type safety with generic NamedlistThe
Namedlistclass now properly uses generics to ensure type safety when working with keys and values.class Namedlist(list[_TNamedList], Generic[_TNamedKeys, _TNamedList]):
1652-1652: Improved parameter type annotationThe
fromdictparameter in theNamedlist.__init__method now has proper type annotation, improving clarity about expected input types.
1756-1756: Enhanced method return type annotationThe
items()method now has an explicit return type annotation, which improves code clarity and IDE support.
1870-1871: Appropriate class renamingRenaming the class from
ResourcestoResourceListis a good change as it more accurately describes its purpose and avoids confusion with the newResourcesclass inresources.py.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
src/snakemake/resources.py (2)
589-594: 🛠️ Refactor suggestionUse isinstance() instead of type comparison.
Direct type comparison should be replaced with
isinstance()for more robust type checking.- if not isinstance(self_val, type(other_val)): + if not isinstance(self_val, type(other_val)) or not isinstance(other_val, type(self_val)): raise ResourceConstraintError(self_val, other_val)
275-282: 🛠️ Refactor suggestionFix mutable default arguments.
Using mutable default arguments (empty lists or dictionaries) can lead to unexpected behavior when the function is called multiple times.
@classmethod def _merge_resource_dict( cls, resources: List[Dict[str, str | int]], - skip: Optional[List[str]] = None, - methods: Optional[Dict[str, Callable[[List[int]], int]]] = None, + skip: Optional[List[str]] = None, + methods: Optional[Dict[str, Callable[[List[int]], int]]] = None, default_method: Callable[[List[int]], int] = max, ): skip = skip or [] methods = methods or {}
🧹 Nitpick comments (11)
src/snakemake/resources.py (11)
29-29: Remove unused imports.The imports
get_function_paramsandoverwrite_function_paramsare not used in this file.-from snakemake.common import get_function_params, get_input_function_aux_params, mb_to_mib, mib_to_mb, overwrite_function_params +from snakemake.common import get_input_function_aux_params, mb_to_mib, mib_to_mb🧰 Tools
🪛 Ruff (0.8.2)
29-29:
snakemake.common.get_function_paramsimported but unusedRemove unused import
(F401)
29-29:
snakemake.common.overwrite_function_paramsimported but unusedRemove unused import
(F401)
471-477: Good class documentation with spelling error.The class documentation clearly explains the purpose and behavior of the
Resourceclass, but contains a spelling error in line 477.- ensures a resource has been computed before its value is retreived. + ensures a resource has been computed before its value is retrieved.🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 477-477:
retreived ==> retrieved🪛 GitHub Actions: Codespell
[error] 477-477: codespell: 'retreived' is a misspelling. Correct spelling is 'retrieved'.
480-484: Fix spelling error in documentation.There's a spelling error in the documentation explaining the class behavior.
- will be intepreted as a human readable resource and converted to an integer of + will be interpreted as a human readable resource and converted to an integer of🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 482-482:
intepreted ==> interpreted
529-534: Simplify code with ternary operator.This if-else block can be simplified using a ternary operator.
- def __repr__(self): - if self.is_evaluable(): - value = "function(...)" - else: - value = self._value - return f'Resource("{self.name}", {value})' + def __repr__(self): + value = "function(...)" if self.is_evaluable() else self._value + return f'Resource("{self.name}", {value})'🧰 Tools
🪛 Ruff (0.8.2)
529-532: Use ternary operator
value = "function(...)" if self.is_evaluable() else self._valueinstead ofif-else-blockReplace
if-else-block withvalue = "function(...)" if self.is_evaluable() else self._value(SIM108)
564-564: Fix spelling error in documentation.There's a spelling error in the documentation for the
evaluatemethod.- This method ensures the returned resource is correctly returned in a - new ``Resource`` class, validated, and standardized. Resources should thus - only be evaulated via this method. + This method ensures the returned resource is correctly returned in a + new ``Resource`` class, validated, and standardized. Resources should thus + only be evaluated via this method.🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 564-564:
evaulated ==> evaluated
680-680: Fix spelling error in documentation.There's a spelling error in the documentation for the
from_cli_expressionmethod.- value: - python expression to be evaulated + value: + python expression to be evaluated🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 680-680:
evaulated ==> evaluated
748-759: Add exception chaining with 'from' clause.When raising a new exception from a caught exception, use the
fromclause to preserve the traceback.- raise WorkflowError( + raise WorkflowError( "Failed to evaluate resources value " f"'{val}'.\n" " String arguments may need additional " "quoting. E.g.: --default-resources " "\"tmpdir='/home/user/tmp'\" or " "--set-resources \"somerule:someresource='--nice=100'\". " "This also holds for setting resources inside of a profile, where " "you might have to enclose them in single and double quotes, " "i.e. someresource: \"'--nice=100'\".", - e, + e, + ) from e🧰 Tools
🪛 Ruff (0.8.2)
748-759: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
900-902: Fix spelling error in documentation.There's a spelling error in the documentation for the
parsemethod.- """Parse a series of CLI-style string resource assigments. + """Parse a series of CLI-style string resource assignments.🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 901-901:
assigments ==> assignments
220-220: Add exception chaining with 'from' clause.When raising a new exception from a caught exception, use the
fromclause to preserve the traceback.- except ResourceError as err: - raise ResourceInsufficiencyError(additive_resources, err.args[0]) + except ResourceError as err: + raise ResourceInsufficiencyError(additive_resources, err.args[0]) from err🧰 Tools
🪛 Ruff (0.8.2)
220-220: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
850-850: Add exception chaining with 'from' clause.When raising a new exception from a caught exception, use the
fromclause to preserve the traceback.- except ValueError: - raise ValueError("Resources have to be defined as name=value pairs.") + except ValueError as e: + raise ValueError("Resources have to be defined as name=value pairs.") from e🧰 Tools
🪛 Ruff (0.8.2)
850-850: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
864-864: Add exception chaining with 'from' clause.When raising a new exception from a caught exception, use the
fromclause to preserve the traceback.- except KeyError: - raise ValueError(f"Unexpected mode for DefaultResources: {mode}") + except KeyError as e: + raise ValueError(f"Unexpected mode for DefaultResources: {mode}") from e🧰 Tools
🪛 Ruff (0.8.2)
864-864: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
src/snakemake/io.py(8 hunks)src/snakemake/resources.py(14 hunks)tests/common.py(6 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/common.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
src/snakemake/io.pysrc/snakemake/resources.py
🪛 Ruff (0.8.2)
src/snakemake/resources.py
29-29: snakemake.common.get_function_params imported but unused
Remove unused import
(F401)
29-29: snakemake.common.overwrite_function_params imported but unused
Remove unused import
(F401)
220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
529-532: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block
Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value
(SIM108)
748-759: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
850-850: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
864-864: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py
[failure] 477-477:
retreived ==> retrieved
[failure] 482-482:
intepreted ==> interpreted
[failure] 564-564:
evaulated ==> evaluated
[failure] 680-680:
evaulated ==> evaluated
[failure] 901-901:
assigments ==> assignments
🪛 GitHub Actions: Codespell
src/snakemake/resources.py
[error] 477-477: codespell: 'retreived' is a misspelling. Correct spelling is 'retrieved'.
⏰ Context from checks skipped due to timeout of 90000ms (31)
- GitHub Check: tests (10, windows-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (9, windows-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (8, windows-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (7, windows-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (6, windows-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (5, windows-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (4, windows-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (3, windows-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (2, windows-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (1, windows-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py311)
- GitHub Check: apidocs
🔇 Additional comments (11)
src/snakemake/io.py (7)
1-1: Good use of future annotations import.Adding
from __future__ import annotationsenables postponed evaluation of type annotations, which is a modern Python practice that helps with circular references in type hints.
31-37: Appropriate type imports added.The additional imports for Generic, Iterator, and Tuple types support the enhanced type safety introduced in this file.
1636-1640: Improved type variable definitions with clear documentation.The modification of
_TNamedListto remove its bound toNamedlistallows for more flexible usage, and the addition of_TNamedKeysenhances type safety for self-returning methods. Both have clear docstrings explaining their purpose.
1643-1643: Enhanced type safety for Namedlist class.Updating the
Namedlistclass to inherit fromlist[_TNamedList]andGeneric[_TNamedKeys, _TNamedList]improves type checking and enables better IDE support.
1652-1652: Improved parameter typing.Specifying the type of
fromdictasOptional[Dict[_TNamedKeys, _TNamedList]]clarifies the expected structure of the input parameter.
1756-1756: Enhanced method return type annotation.Adding the return type annotation
Iterator[Tuple[_TNamedKeys, _TNamedList]]to theitemsmethod improves type safety and code clarity.
1870-1871: Appropriate class renaming.Renaming
ResourcestoResourceListand specifying its inheritance asNamedlist[str, str | int]helps avoid confusion with the newResourcesclass in resources.py, while clearly indicating its purpose and structure.src/snakemake/resources.py (4)
1-26: Good use of type hints with comprehensive imports.The import structure is well-organized, using future annotations and a thorough set of typing imports that support the implementation of type-safe resource handling.
430-457: Well-implemented conversion functions with proper error handling.The helper functions
evaluable_from_mb_to_mibandevaluable_from_mib_to_mbprovide clear and type-safe conversions between different memory unit formats, with appropriate error handling for invalid types.
773-787: Well-designed Resource container with clear documentation.The
Resourcesclass provides a standardized container for resources with excellent handling of suffixed and unsuffixed versions of the same size resource. The docstring clearly explains its purpose and initialization methods.
1010-1035: Good implementation of resource scopes with validation.The
ResourceScopesclass effectively defines and enforces valid scopes for resources, with appropriate validation and normalization of size resources.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (14)
src/snakemake/io.py (1)
1636-1638: Consider converting the standalone string literal to a comment or docstring.
If the string on line 1637 is intended as documentation (rather than just an unused literal), consider using a code comment or adding it to a docstring block for clarity.+# This type variable is for self-returning methods on Namedlist-deriving classes.tests/tests.py (1)
835-836: Optional debug line can be removed or documented.
Line 836 is commented out. If it is no longer needed, you may remove it to keep the codebase clean, or add a short explanation.- default_resources=(["mem_mb=0"]), - # cleanup=False + default_resources=(["mem_mb=0"]),src/snakemake/resources.py (12)
29-29: Remove unused imports.The imports
get_function_paramsandoverwrite_function_paramsappear to be unused in this file.-from snakemake.common import get_function_params, get_input_function_aux_params, mb_to_mib, mib_to_mb, overwrite_function_params +from snakemake.common import get_input_function_aux_params, mb_to_mib, mib_to_mb🧰 Tools
🪛 Ruff (0.8.2)
29-29:
snakemake.common.get_function_paramsimported but unusedRemove unused import
(F401)
29-29:
snakemake.common.overwrite_function_paramsimported but unusedRemove unused import
(F401)
220-220: Use explicit exception chaining.When raising exceptions inside an except block, you should use
raise ... from errto maintain the exception chain for better debugging.- raise ResourceInsufficiencyError(additive_resources, err.args[0]) + raise ResourceInsufficiencyError(additive_resources, err.args[0]) from err🧰 Tools
🪛 Ruff (0.8.2)
220-220: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
479-479: Fix typo in docstring.There's a spelling error in the documentation.
- ensures a resource has been computed before its value is retreived. + ensures a resource has been computed before its value is retrieved.🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 479-479:
retreived ==> retrieved🪛 GitHub Actions: Codespell
[error] 479-479: codespell: 'retreived' is a misspelling. Suggested correction: 'retrieved'.
484-484: Fix typo in docstring.There's a spelling error in the documentation.
- will be intepreted as a human readable resource and converted to an integer of + will be interpreted as a human readable resource and converted to an integer of🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 484-484:
intepreted ==> interpreted
531-534: Use ternary operator for conciseness.Consider using a ternary operator to improve code readability and conciseness.
- if self.is_evaluable(): - value = "function(...)" - else: - value = self._value + value = "function(...)" if self.is_evaluable() else self._value🧰 Tools
🪛 Ruff (0.8.2)
531-534: Use ternary operator
value = "function(...)" if self.is_evaluable() else self._valueinstead ofif-else-blockReplace
if-else-block withvalue = "function(...)" if self.is_evaluable() else self._value(SIM108)
566-566: Fix typo in docstring.There's a spelling error in the documentation.
- This method ensures the returned resource is correctly returned in a - new ``Resource`` class, validated, and standardized. Resources should thus - only be evaulated via this method. + This method ensures the returned resource is correctly returned in a + new ``Resource`` class, validated, and standardized. Resources should thus + only be evaluated via this method.🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 566-566:
evaulated ==> evaluated
590-596: Improve type comparison logic.The current type comparison isn't symmetric. Consider using a more clear approach to check if types match.
- if not isinstance(self_val, type(other_val)): + if type(self_val) != type(other_val): raise ResourceConstraintError(self_val, other_val) if isinstance(self_val, int): assert isinstance(other_val, int) return Resource(self.name, min(other_val, self_val)) return self
682-682: Fix typo in docstring.There's a spelling error in the documentation.
- python expression to be evaulated + python expression to be evaluated🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 682-682:
evaulated ==> evaluated
750-761: Use explicit exception chaining.When raising exceptions inside an except block, you should use
raise ... from eto maintain the exception chain for better debugging.raise WorkflowError( "Failed to evaluate resources value " f"'{val}'.\n" " String arguments may need additional " "quoting. E.g.: --default-resources " "\"tmpdir='/home/user/tmp'\" or " "--set-resources \"somerule:someresource='--nice=100'\". " "This also holds for setting resources inside of a profile, where " "you might have to enclose them in single and double quotes, " "i.e. someresource: \"'--nice=100'\".", - e, + e) from e🧰 Tools
🪛 Ruff (0.8.2)
750-761: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
852-852: Use explicit exception chaining.When raising exceptions inside an except block, you should use
raise ... from Noneto explicitly suppress the original exception while maintaining clarity.- raise ValueError("Resources have to be defined as name=value pairs.") + raise ValueError("Resources have to be defined as name=value pairs.") from None🧰 Tools
🪛 Ruff (0.8.2)
852-852: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
866-866: Use explicit exception chaining.When raising exceptions inside an except block, you should use
raise ... from Noneto explicitly suppress the original exception while maintaining clarity.- raise ValueError(f"Unexpected mode for DefaultResources: {mode}") + raise ValueError(f"Unexpected mode for DefaultResources: {mode}") from None🧰 Tools
🪛 Ruff (0.8.2)
866-866: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
903-903: Fix typo in docstring.There's a spelling error in the documentation.
- """Parse a series of CLI-style string resource assigments. + """Parse a series of CLI-style string resource assignments.🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 903-903:
assigments ==> assignments
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
src/snakemake/io.py(8 hunks)src/snakemake/resources.py(14 hunks)tests/tests.py(18 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
src/snakemake/io.pytests/tests.pysrc/snakemake/resources.py
🪛 Ruff (0.8.2)
tests/tests.py
1095-1095: Ambiguous variable name: l
(E741)
1095-1095: Use l != "\n" instead of not l == "\n"
Replace with != operator
(SIM201)
src/snakemake/resources.py
29-29: snakemake.common.get_function_params imported but unused
Remove unused import
(F401)
29-29: snakemake.common.overwrite_function_params imported but unused
Remove unused import
(F401)
220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
531-534: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block
Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value
(SIM108)
750-761: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
852-852: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
866-866: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py
[failure] 479-479:
retreived ==> retrieved
[failure] 484-484:
intepreted ==> interpreted
[failure] 566-566:
evaulated ==> evaluated
[failure] 682-682:
evaulated ==> evaluated
[failure] 903-903:
assigments ==> assignments
🪛 GitHub Actions: Codespell
src/snakemake/resources.py
[error] 479-479: codespell: 'retreived' is a misspelling. Suggested correction: 'retrieved'.
⏰ Context from checks skipped due to timeout of 90000ms (31)
- GitHub Check: tests (10, windows-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (9, windows-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (8, windows-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (7, windows-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (6, windows-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (5, windows-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (4, windows-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (3, windows-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (2, windows-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (1, windows-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py311)
- GitHub Check: apidocs
🔇 Additional comments (20)
src/snakemake/io.py (5)
1-1: Good use of postponed evaluation of type hints.
This import is standard when forward references or advanced typing features are needed and can help avoid circular dependency issues.
31-36: No issues with newly added type-related imports.
These cleanly align with Python's typing best practices (e.g.,Generic,Iterator,Tuple) and are used consistently in the updated code below.
1643-1643: Use of generics inNamedlist.
DefiningNamedlist(list, Generic[_TNamedKeys, _TNamedList])appropriately leverages Python’s generic system to enforce type-checked usage of_TNamedKeys&_TNamedList.
1652-1652: Clearer parameter type annotation in the constructor.
Specifyingfromdict: Optional[Dict[_TNamedKeys, _TNamedList]] = Noneis concise and precise, ensuring better type safety when converting dicts intoNamedlist.
1757-1757: Proper return type annotation for theitemsmethod.
Returning anIterator[Tuple[_TNamedKeys, _TNamedList]]is consistent with your generics usage and helps IDEs and type checkers infer usage more accurately.tests/tests.py (11)
15-19: Validated imports for new exception and resource classes.
These imports (e.g.ResourceDuplicationError,GroupResources,Resources) ensure code references are well-defined and consistent with the new resource-management functionality.
814-814: Simplified assertion logic inassert_resources.
Directly comparing dictionaries (assert resources == expected_resources) is both more readable and robust than partial dictionary checks.
842-851: Verifying resource dictionary with extra fields.
These newly added lines (mem, mem_mib, disk, disk_mib, etc.) appear consistent with the new test scenario validating resource translation. Everything looks correct for the numeric checks.
854-879:test_resources_can_be_provided_in_mibthoroughly exercises Mebibyte-based resource usage.
The test sets{"mem_mib": 57221}and confirms successful translation to standard MB values. This scenario is well-covered, ensuring correctness in resource conversion.
881-905:test_global_resources_can_be_human_readablevalidates intuitive user inputs.
Verifying “60 Gb” for memory helps ensure the parser correctly handles human-readable formats. The test coverage is strong, and the approach is straightforward.
908-927: Correctly preventing mixed resource prefixes (e.g.memvsmem_mb).
These lines confirm that specifying both unprefixed and suffixed resource variants together raises aResourceDuplicationError, reinforcing resource definition integrity.
930-941: Ensuring human-readable resources are invalid when a suffix is required.
This covers the scenario wheremem_mbmust be numeric only, preventing strings like “60 GB.” Good job raisingWorkflowErrorfor invalid usage.
943-955: Rejecting human-readable formats in declarations with suffixed resource keys.
This test enforces consistency for resources such asmem_mb, ensuring that only strict integer or numeric inputs are allowed.
967-969: Precise usage ofResources.parsewith expression flags.
Allowing expressions (e.g.mem_mb=0) indefault_resourcesis consistent with the rest of the refactoring. This usage is correct and well-tested.
1409-1414:test_default_resources_humanreadableensures fractional Gigabyte values are rounded properly.
Great job testing real-world memory designations like'1 Gb'and'0.001 TB'. This confirms proper scaling.
1419-1425:test_default_resources_mebibytesensures partial Mebibyte values are correctly handled.
The test checks for non-integer inputs (e.g.,953.67) and verifies rounding or conversion logic. Nicely done.src/snakemake/resources.py (4)
276-282: Good handling of mutable default arguments.The code correctly handles mutable default arguments by using
Noneand initializing them inside the function body.
775-1007: Great implementation of the Resources class.The Resources class is well-designed with a clear interface and strong type safety. It effectively replaces the old DefaultResources class and provides a standardized container for working with resources.
Key strengths:
- Good handling of resource normalization
- Clear implementation of the Mapping interface
- Helpful class methods for different initialization scenarios
- Proper type annotations throughout
🧰 Tools
🪛 Ruff (0.8.2)
852-852: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
866-866: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
🪛 GitHub Check: Check for spelling errors
[failure] 903-903:
assigments ==> assignments
473-773: Well-structured Resource class with comprehensive functionality.The Resource class is well-designed with clear responsibilities:
- Standardizing resource representation
- Handling human-readable resource parsing
- Supporting resource evaluation and constraints
- Providing conversion between different resource units (MB/MiB)
This implementation will significantly improve resource handling consistency across the codebase.
🧰 Tools
🪛 Ruff (0.8.2)
531-534: Use ternary operator
value = "function(...)" if self.is_evaluable() else self._valueinstead ofif-else-blockReplace
if-else-block withvalue = "function(...)" if self.is_evaluable() else self._value(SIM108)
750-761: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
🪛 GitHub Check: Check for spelling errors
[failure] 479-479:
retreived ==> retrieved
[failure] 484-484:
intepreted ==> interpreted
[failure] 566-566:
evaulated ==> evaluated
[failure] 682-682:
evaulated ==> evaluated🪛 GitHub Actions: Codespell
[error] 479-479: codespell: 'retreived' is a misspelling. Suggested correction: 'retrieved'.
1012-1151: Good enhancement of the ResourceScopes class.The ResourceScopes class has been effectively enhanced to inherit from Dict, making it more Pythonic and easier to use. The implementation of methods for checking and normalizing resource scopes is clean and well-documented.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
src/snakemake/resources.py (1)
599-600: 🛠️ Refactor suggestionImprove type checking logic for resource constraint
The current implementation has a potential logical issue in the type comparison. It's comparing types with direct equality which may not work correctly for inheritance hierarchies.
self_val = self.value - if not isinstance(self_val, type(other_val)): + if not isinstance(self_val, type(other_val)) or not isinstance(other_val, type(self_val)): raise ResourceConstraintError(self_val, other_val)
🧹 Nitpick comments (12)
src/snakemake/workflow.py (3)
1755-1756: More efficient resource copying.Using the
copy()method instead ofdeepcopy()is more efficient and appropriate here asResourcesobjects don't contain nested mutable objects that would require deep copying.
1761-1774: Improved error handling for thread resources.The try-except block with specific
ResourceValidationErrorhandling provides better error reporting. However, the raised exception could be improved.Apply the following change to improve the exception chain:
try: if overwrite_threads is not None: rule.resources["_cores"] = overwrite_threads elif ruleinfo.threads is not None: rule.resources["_cores"] = ruleinfo.threads else: rule.resources["_cores"] = 1 except ResourceValidationError: - raise RuleException( + raise RuleException( "Threads value has to be an integer, float, or a callable.", rule=rule, - ) + ) from NoneThis makes it clear that you're explicitly replacing the original exception with a more user-friendly one.
🧰 Tools
🪛 Ruff (0.8.2)
1770-1773: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
1779-1788: Enhanced resource validation with Resources.from_mapping.The use of
Resources.from_mappingprovides structured validation, but the exception handling should be improved.Apply the following change to improve the exception chain:
try: resources = Resources.from_mapping(resources) except KeyError as err: raise RuleException( "Resources values have to be integers, strings, or callables " f"(functions): type({err.args[0]} : {err.args[1]}) == " f"{type(err.args[1])}", rule=rule, - ) + ) from errThis preserves the original exception in the chain, making debugging easier.
🧰 Tools
🪛 Ruff (0.8.2)
1782-1787: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
src/snakemake/cli.py (1)
211-229: Consider raising from the original exception for clarity.When wrapping an exception at line 227, you can preserve the original traceback by adding
from e. For example:- raise ValueError(f"{errmsg} Cause: {e}") + raise ValueError(f"{errmsg} Cause: {e}") from eThis helps distinguish errors in exception handling from the underlying exception.
🧰 Tools
🪛 Ruff (0.8.2)
220-220: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
227-227: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
tests/tests.py (1)
1097-1097: Use a clearer variable name and avoidnot l == "\n".Reading lines with:
lines = [line for line in f if line != "\n"]would improve readability and satisfy the lint suggestion (
!=instead ofnot ... ==), while also avoiding the single-letter variable namel.🧰 Tools
🪛 Ruff (0.8.2)
1097-1097: Ambiguous variable name:
l(E741)
1097-1097: Use
l != "\n"instead ofnot l == "\n"Replace with
!=operator(SIM201)
src/snakemake/resources.py (7)
4-4: Remove unused importsThere are several imports that are declared but never used in the code:
dataclasses.dataclass(line 4)snakemake.common.get_function_params(line 30)snakemake.common.overwrite_function_params(line 34)-from dataclasses import dataclass from humanfriendly import InvalidTimespan, InvalidSize, parse_size, parse_timespanfrom snakemake.common import ( - get_function_params, get_input_function_aux_params, mb_to_mib, mib_to_mb, - overwrite_function_params, )Also applies to: 30-30, 34-34
🧰 Tools
🪛 Ruff (0.8.2)
4-4:
dataclasses.dataclassimported but unusedRemove unused import:
dataclasses.dataclass(F401)
220-220: Useraise ... from errfor proper exception chainingWhen raising an exception inside an except block, it's a best practice to use
raise ... from errto maintain the exception chain for better debugging.try: layers = cls._get_layers( int_resources, list(sorted_constraints.values()), sortby ) except ResourceError as err: - raise ResourceInsufficiencyError(additive_resources, err.args[0]) + raise ResourceInsufficiencyError(additive_resources, err.args[0]) from err🧰 Tools
🪛 Ruff (0.8.2)
220-220: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
479-479: Fix spelling errors in docstringsThere are several spelling errors in the docstrings:
"""Standardized representation of a resource name and value. In addition to ``int`` and ``str``, the standard resource types, ``Resource`` can store ``None``, signifying an unset resource, and a callable, signifying a resource - to be calculated in the context of a rule. Via the ``.value`` property, ``Resource`` - ensures a resource has been computed before its value is retreived. + to be calculated in the context of a rule. Via the ``.value`` property, ``Resource`` + ensures a resource has been computed before its value is retrieved. The class additionally handles input validation and standardization. ``float``s get rounded into ``int``. If the resource name corresponds to a size resource ("disk" and "mem") or a timespan resource ("runtime"), strings values - will be intepreted as a human readable resource and converted to an integer of + will be interpreted as a human readable resource and converted to an integer of appropriate denomination (Mb for size, hr for timespan)."""Evaluate the resource with given args. This method ensures the returned resource is correctly returned in a new ``Resource`` class, validated, and standardized. Resources should thus - only be evaulated via this method. + only be evaluated via this method. """"""Create a new evaluable resource based on a python expression. Threads can optionally be included in the expression environment Arguments ========= name: name of the resource value: - python expression to be evaulated + python expression to be evaluated with_threads_arg: boolean If True, include ``threads`` as an argument in the returned function """- """Parse a series of CLI-style string resource assigments. + """Parse a series of CLI-style string resource assignments. Arguments ========= exprs: list of str Each expr is in the form RESOURCE=VALUE, where RESOURCE is a valid python identifier, and VALUE is a number, a quoted string, or a valid python expression.Also applies to: 484-484, 572-572, 690-690, 911-911
🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 479-479:
retreived ==> retrieved
420-422: Add more context to resource error messageThe error message when a resource constraint is exceeded could be more informative to help users understand what went wrong.
error_text = [ f"\t{res}: {amount}/{constraint}" for res, amount, constraint in too_high ] - raise ResourceError("\n".join(error_text)) + raise ResourceError(f"Resource requirement exceeds constraint:\n" + "\n".join(error_text))
537-541: Simplify condition with ternary operatorThis can be simplified using a ternary operator for better readability.
def __repr__(self): - if self.is_evaluable(): - value = "function(...)" - else: - value = self._value + value = "function(...)" if self.is_evaluable() else self._value return f'Resource("{self.name}", {value})'🧰 Tools
🪛 Ruff (0.8.2)
537-540: Use ternary operator
value = "function(...)" if self.is_evaluable() else self._valueinstead ofif-else-blockReplace
if-else-block withvalue = "function(...)" if self.is_evaluable() else self._value(SIM108)
758-769: Useraise ... from efor proper exception chainingWhen raising a new exception inside an except block, using
raise ... from emaintains the exception chain for better debugging.raise WorkflowError( "Failed to evaluate resources value " f"'{val}'.\n" " String arguments may need additional " "quoting. E.g.: --default-resources " "\"tmpdir='/home/user/tmp'\" or " "--set-resources \"somerule:someresource='--nice=100'\". " "This also holds for setting resources inside of a profile, where " "you might have to enclose them in single and double quotes, " "i.e. someresource: \"'--nice=100'\".", - e, + e, + ) from e🧰 Tools
🪛 Ruff (0.8.2)
758-769: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
725-770: Ensure consistent error handling for resource evaluationThe
cli_evaluatormethod catches exceptions during resource evaluation but uses different approaches for different error types. It would be better to have a consistent approach to exception handling.try: value = eval( val, { "input": input, "attempt": attempt, "system_tmpdir": tempfile.gettempdir(), **({"threads": threads} if threads is not None else {}), }, ) # Triggers for string arguments like n1-standard-4 except NameError: return val except Exception as e: try: # check if resource is parsable as human friendly (given the correct # name and formatted value). If it is, we return the parsed value to # save a step later. if name in HumanFriendlyResources: return Resource._parse_human_friendly(name, val) - except (InvalidSize, InvalidTimespan): + except (InvalidSize, InvalidTimespan) as parse_err: pass if is_file_not_found_error(e, input): return TBDString() raise WorkflowError( "Failed to evaluate resources value " f"'{val}'.\n" " String arguments may need additional " "quoting. E.g.: --default-resources " "\"tmpdir='/home/user/tmp'\" or " "--set-resources \"somerule:someresource='--nice=100'\". " "This also holds for setting resources inside of a profile, where " "you might have to enclose them in single and double quotes, " "i.e. someresource: \"'--nice=100'\".", e, - ) + ) from e🧰 Tools
🪛 Ruff (0.8.2)
758-769: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
tests/test_set_resources_complex/expected-results/test.outis excluded by!**/*.out
📒 Files selected for processing (6)
src/snakemake/cli.py(7 hunks)src/snakemake/resources.py(13 hunks)src/snakemake/settings/types.py(2 hunks)src/snakemake/spawn_jobs.py(2 hunks)src/snakemake/workflow.py(7 hunks)tests/tests.py(19 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
src/snakemake/workflow.pysrc/snakemake/spawn_jobs.pysrc/snakemake/settings/types.pysrc/snakemake/resources.pytests/tests.pysrc/snakemake/cli.py
🪛 Ruff (0.8.2)
src/snakemake/workflow.py
1770-1773: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
1782-1787: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/resources.py
4-4: dataclasses.dataclass imported but unused
Remove unused import: dataclasses.dataclass
(F401)
30-30: snakemake.common.get_function_params imported but unused
Remove unused import
(F401)
34-34: snakemake.common.overwrite_function_params imported but unused
Remove unused import
(F401)
220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
537-540: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block
Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value
(SIM108)
758-769: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
860-860: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
874-874: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
tests/tests.py
1097-1097: Ambiguous variable name: l
(E741)
1097-1097: Use l != "\n" instead of not l == "\n"
Replace with != operator
(SIM201)
src/snakemake/cli.py
220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
227-227: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py
[failure] 479-479:
retreived ==> retrieved
[failure] 484-484:
intepreted ==> interpreted
[failure] 572-572:
evaulated ==> evaluated
[failure] 690-690:
evaulated ==> evaluated
[failure] 911-911:
assigments ==> assignments
⏰ Context from checks skipped due to timeout of 90000ms (31)
- GitHub Check: tests (10, windows-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (9, windows-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (8, windows-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (7, windows-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (6, windows-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (5, windows-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (4, windows-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (3, windows-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (2, windows-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (1, windows-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: apidocs
- GitHub Check: tests (1, ubuntu-latest, py311)
🔇 Additional comments (34)
src/snakemake/spawn_jobs.py (2)
95-96: Resource value access simplified.The code now directly accesses the
rawattribute of the resource value instead of using a helper function. This change is aligned with the newResourcesclass that standardizes resource handling.
105-106: Using consistent resource value access pattern.The same pattern is now applied to threads values, maintaining consistency in how resource values are accessed.
src/snakemake/workflow.py (8)
58-61: Added ResourceValidationError to import statement.The import of
ResourceValidationErrorsupports the new resource handling system that provides better error messaging for resource validation issues.
118-118: Updated resource import.Importing
Resourcesfromsnakemake.resourcesreflects the centralization of resource handling, which improves code organization.
166-166: Simplified global resources initialization.Direct assignment of
self.resource_settings.resourcesremoves redundant code and makes the implementation cleaner.
568-568: Updated _cores property to access value attribute.Accessing the
.valueattribute of resources is consistent with the newResourcesclass approach.
572-572: Updated nodes property to access value attribute.Similar to the _cores property, this change maintains consistency with the new resource handling approach.
629-629: Added return type annotation to get_rule method.Adding the return type annotation improves code clarity and assists with type checking.
1775-1778: Validation for unnamed resources.The code correctly validates that resources must be named, providing a clear error message.
1790-1791: Properly applying overwritten resources.The code correctly updates rule resources with any overwritten values provided via command-line arguments.
src/snakemake/cli.py (6)
12-12: No issues found with the new typing import.
42-44: ImportingResourceandResourceslooks good.This aligns with the project's refactoring from
DefaultResourcesto the new resource classes. No concerns here.
156-181: Refactoredparse_set_resourcesfunction is well-structured.
- The logic for splitting and grouping resource assignments into a dictionary is clear.
- Error handling with
raise ValueError(errmsg) from errpreserves the original exception context, which is best practice.No further issues spotted; everything appears correct.
516-517: Initialization of resource arguments is a good change.Using
Resources()as the default instead of a bare dictionary clarifies the nature of these constraints.
596-597: Approach for parsing default resources with expressions looks fine.The
Resources.parser_factorycall is consistent with the new resource design.
969-971: Configured default resources with parsing logic.Specifying default expressions (like
"mem_mb=0") is appropriate if that aligns with your workflow’s requirements.src/snakemake/settings/types.py (2)
33-33: ImportedResourceandResourcessuccessfully.This replacement is consistent with the ongoing resource-management refactoring.
345-353: EnhancedResourceSettingsclass initialization.
- Using
field(default_factory=Resources)and storing more structured resource data is a clear improvement over raw mappings.- The fallback to
Resources.default("bare")at line 353 is a straightforward way to ensure consistent defaults.No red flags identified.
tests/tests.py (9)
15-18: Importing new exceptions and resource classes for testing.Bringing in
WorkflowError,ResourceDuplicationError, andGroupResources,Resourcesis aligned with the refactored codebase.
816-817:assert_resourceshelper is concise and clear.Simply checking for equality is sufficient for these tests.
829-854: Refined test logic fortest_group_jobs_resources.
- Setting
default_resources=(["mem_mb=0"])ensures explicit control over resource defaults in the test.- The final
assert_resources(...)calls validate mem/disk conversions thoroughly.All changes look good.
856-881:test_resources_can_be_provided_in_mibis well-structured.
- Verifies that specifying memory in MiB is processed correctly (
mem_mib=57221→mem_mb=60000).- Good coverage of the MiB→MB conversion logic.
No concerns here.
883-908:test_global_resources_can_be_human_readablecoverage is excellent.This confirms that human-readable resource strings (e.g.,
"60 Gb") are converted properly.
910-930:test_resources_cannot_provide_prefixed_and_unprefixed_together: well-defined negative test.Ensuring that mixing e.g.
memwithmem_mbraisesResourceDuplicationErrorprevents ambiguous resource definitions.
932-943:test_suffixed_resource_constraints_cannot_be_human_readable: solid negative test.Correctly expecting a
WorkflowErrorfor “60 GB” with_mbconstraints avoids confusion of scale units.
945-957:test_suffixed_resource_declarations_cannot_be_human_readableis consistent with the preceding logic.Validates that you can’t define a
_mbresource with “Gb” strings. Great catch to avoid mismatch.
1076-1101:test_resource_prefix_does_not_affect_scope_overwrite: thorough scenario.
- Validates correct overrides for resource scopes like “global_res” and “mem” vs. “mem_mb”.
- Confirms that matching resource prefixes does not inadvertently skip the scope rules.
All looks fine.
🧰 Tools
🪛 Ruff (0.8.2)
1097-1097: Ambiguous variable name:
l(E741)
1097-1097: Use
l != "\n"instead ofnot l == "\n"Replace with
!=operator(SIM201)
src/snakemake/resources.py (7)
273-282: LGTM: Proper handling of mutable default argumentsThe code correctly avoids using mutable default arguments by setting defaults to None and initializing them in the function body.
430-443: LGTM: Well-implemented unit conversion functionsThe implementation of unit conversion functions for evaluable resources (MB to MiB and vice versa) is well done. The functions properly handle type checking and conversions.
Also applies to: 445-457
607-632: LGTM: Resource unit conversion methodsThe
to_mibandto_mbmethods are well designed with proper error handling and support for converting both evaluated and unevaluated resources.Also applies to: 633-658
783-1015: LGTM: Well-designed Resources classThe
Resourcesclass is well-designed as a standardized container for resource objects. It properly handles unit conversion, resource normalization, and provides convenient methods for resource manipulation.The implementation of the Mapping interface methods and additional utility methods is thorough and follows best practices.
🧰 Tools
🪛 Ruff (0.8.2)
860-860: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
874-874: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
🪛 GitHub Check: Check for spelling errors
[failure] 911-911:
assigments ==> assignments
1020-1154: LGTM: Improved ResourceScopes classThe
ResourceScopesclass is well-implemented with clear semantics for resource scope handling. The methods for checking resource scopes (is_local,is_global,is_excluded) and the properties for retrieving scoped resources are well-designed.
54-253: LGTM: Comprehensive implementation of resource handling for group jobsThe
GroupResources.basic_layeredmethod is a well-documented and robust implementation for calculating resource requirements for group jobs. The implementation handles different resource types, constraints, and optimization strategies.The code is well-structured with clear separation of concerns between resource parsing, constraint checking, and layer optimization.
🧰 Tools
🪛 Ruff (0.8.2)
220-220: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
824-825:Details
❓ Verification inconclusive
Verify error handling for resource duplication
The code raises a
ResourceDuplicationErrorwhen multiple variants of the same sized resource are specified. Verify that this error is properly caught and handled in the calling code.
🏁 Script executed:
#!/bin/bash # Check if ResourceDuplicationError is handled in the caller rg -A 5 "ResourceDuplicationError" --include="*.py" --exclude="resources.py"Length of output: 516
Below is a revised shell script command that correctly restricts the search to Python files (excluding the file under review) and looks for explicit exception handling of
ResourceDuplicationError. This will help verify whether callers are catching the error in a try/except block.Please execute the following command and share the output so we can confirm the intended error handling is in place:
#!/bin/bash # Check for explicit try/except blocks catching ResourceDuplicationError in the codebase (excluding resources.py) rg -A 5 "except.*ResourceDuplicationError" -g "*.py" -g "!resources.py"
Once you’ve run this script, we can proceed further with the verification.
**
873ada9 to
c13583a
Compare
There was a problem hiding this comment.
Actionable comments posted: 6
🔭 Outside diff range comments (4)
src/snakemake/common/__init__.py (1)
106-112:⚠️ Potential issueConsider preserving the exception chain for better debugging.
While the change to
NestedCoroutineErroris correct, consider usingraise ... from eto preserve the exception chain as suggested by static analysis:- raise NestedCoroutineError( + raise NestedCoroutineError( "Error running coroutine in event loop. Snakemake currently does not " "support being executed from an already running event loop. " "If you run Snakemake e.g. from a Jupyter notebook, make sure to spawn a " "separate process for Snakemake.", - e, - ) + ) from e🧰 Tools
🪛 Ruff (0.11.9)
106-112: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
src/snakemake/workflow.py (2)
783-795: 🛠️ Refactor suggestionFix exception chaining for better error diagnostics.
The exception handling pattern is correct, but should use proper exception chaining for better debugging.
Apply this diff to improve exception chaining:
- except ResourceValidationError: + except ResourceValidationError as e: raise RuleException( "Threads value has to be an integer, float, or a callable.", rule=rule, - ) + ) from e
797-810: 🛠️ Refactor suggestionFix exception chaining and approve the resource mapping approach.
The use of
Resources.from_mapping()is the correct approach for converting rule resources. However, the exception handling needs proper chaining.Apply this diff to improve exception chaining:
except KeyError as err: raise RuleException( "Resources values have to be integers, strings, or callables " f"(functions): type({err.args[0]} : {err.args[1]}) == " f"{type(err.args[1])}", rule=rule, - ) + ) from errtests/tests.py (1)
2516-2516:⚠️ Potential issueUpdate to use the new Resources.parse pattern for consistency.
This line still uses the old
DefaultResourcesclass instead of the newResources.parsepattern used throughout the rest of the file.Apply this diff to maintain consistency with the resource system refactoring:
- default_resources=DefaultResources(["mem_mb=120"]), + default_resources=Resources.parse( + ["mem_mb=120"], defaults="full", allow_expressions=True + ),🧰 Tools
🪛 Ruff (0.11.9)
2516-2516: Undefined name
DefaultResources(F821)
🧹 Nitpick comments (8)
src/snakemake/rules.py (1)
1089-1089: Fix spelling error."inital" should be "initial" in the comment.
- # dag construction routine are run as independent asynchronous loops. + # dag construction routine are run as independent asynchronous loops.Wait, let me check the actual line again. The error is "inital" -> "initial":
- # If input.size_mb is run in an input method, the loops will be nested + # If input.size_mb is run in an input method, the loops will be nestedActually, looking at line 1089-1091:
- # Need to catch this because both input.size_mb and the inital + # Need to catch this because both input.size_mb and the initial🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 1089-1089:
inital ==> initialsrc/snakemake/cli.py (1)
226-236: Good enhancement but improve exception handling.The wrapper function support is well-implemented and maintains backward compatibility. However, consider improving exception chaining as suggested by static analysis.
Apply this diff to improve exception handling:
if wrapper is None: - raise ValueError(errmsg) + raise ValueError(errmsg) from Noneif wrapper is not None: try: assignments[key] = wrapper(value) except Exception as e: - raise ValueError(f"{errmsg} Cause: {e}") + raise ValueError(f"{errmsg} Cause: {e}") from e🧰 Tools
🪛 Ruff (0.11.9)
227-227: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
234-234: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
src/snakemake/jobs.py (1)
37-37: LGTM on import updates, but clean up unused import.The import changes align well with the resource management refactoring, introducing the new
ResourceList,GroupResources,Resources, andResourceInsufficiencyErrorclasses.However, there's an unused import that should be removed:
-from snakemake.io import ( - _IOFile, - IOFile, - ResourceList, - is_callable, - Wildcards, - is_flagged, - get_flag_value, - wait_for_files, -) +from snakemake.io import ( + _IOFile, + IOFile, + ResourceList, + Wildcards, + is_flagged, + get_flag_value, + wait_for_files, +)Also applies to: 45-45, 51-51
tests/tests.py (2)
1146-1146: Fix code quality issues: ambiguous variable name and comparison style.Two improvements needed:
- The variable name
lis ambiguous and can be confused with1orI- Use
!=instead ofnot ==for better readability-lines = [l for l in f.readlines() if not l == "\n"] +lines = [line for line in f.readlines() if line != "\n"]🧰 Tools
🪛 Ruff (0.11.9)
1146-1146: Ambiguous variable name:
l(E741)
1146-1146: Use
l != "\n"instead ofnot l == "\n"Replace with
!=operator(SIM201)
1146-1147: Improve variable naming and comparison style.The static analysis tools correctly identified some style issues:
- The variable name
lis ambiguous and could be confused with1orI- The comparison
not l == "\n"is less clear than using!=Apply this diff to improve readability:
- lines = [l for l in f.readlines() if not l == "\n"] + lines = [line for line in f.readlines() if line != "\n"]🧰 Tools
🪛 Ruff (0.11.9)
1146-1146: Ambiguous variable name:
l(E741)
1146-1146: Use
l != "\n"instead ofnot l == "\n"Replace with
!=operator(SIM201)
src/snakemake/resources.py (3)
472-494: Fix spelling errors in docstring.Multiple spelling errors need correction in the class docstring.
"""Standardized representation of a resource name and value. In addition to ``int`` and ``str``, the standard resource types, ``Resource`` can store ``None``, signifying an unset resource, and a callable, signifying a resource - to be calculated in the context of a rule. Via the ``.value`` property, ``Resource`` - ensures a resource has been computed before its value is retreived. + to be calculated in the context of a rule. Via the ``.value`` property, ``Resource`` + ensures a resource has been computed before its value is retrieved. The class additionally handles input validation and standardization. ``float``s get rounded into ``int``. If the resource name corresponds to a size resource ("disk" and "mem") or a timespan resource ("runtime"), strings values - will be intepreted as a human readable resource and converted to an integer of + will be interpreted as a human readable resource and converted to an integer of appropriate denomination (Mb for size, hr for timespan). Arguments ========= name: str Name of the resource (note that "disk", "mem", and "runtime" has special behaviour, as described above). value: int, str, Callable, None The concrete or unevaluated value """🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 478-478:
retreived ==> retrieved
[failure] 483-483:
intepreted ==> interpreted🪛 GitHub Actions: Codespell
[error] 478-478: codespell: 'retreived' is a misspelling of 'retrieved'
681-698: Fix spelling error in docstring.@classmethod def from_cli_expression( cls, name: str, value: str, *, with_threads_arg: bool = True ): """Create a new evaluable resource based on a python expression. Threads can optionally be included in the expression environment Arguments ========= name: name of the resource value: - python expression to be evaulated + python expression to be evaluated with_threads_arg: boolean If True, include ``threads`` as an argument in the returned function """🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 694-694:
evaulated ==> evaluated
1-1166: Outstanding refactoring of the resource management system!This comprehensive rewrite significantly improves Snakemake's resource handling:
- Type safety: Strong typing throughout with proper validation
- Human-friendly: Support for parsing "60 Gb", "2 hours" etc.
- Unit conversion: Seamless handling of MB/MiB conversions
- Error handling: Clear, specific exceptions with helpful messages
- Clean architecture: Well-separated concerns between Resource, Resources, and ResourceScopes
The implementation successfully addresses all the bugs mentioned in the PR objectives.
Would you like me to help with any documentation updates or additional test cases for edge cases?
🧰 Tools
🪛 Ruff (0.11.9)
4-4:
dataclasses.dataclassimported but unusedRemove unused import:
dataclasses.dataclass(F401)
219-219: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
536-539: Use ternary operator
value = "function(...)" if self.is_evaluable() else self._valueinstead ofif-else-blockReplace
if-else-block withvalue = "function(...)" if self.is_evaluable() else self._value(SIM108)
765-776: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
867-867: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
881-881: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
🪛 GitHub Check: Check for spelling errors
[failure] 478-478:
retreived ==> retrieved
[failure] 483-483:
intepreted ==> interpreted
[failure] 694-694:
evaulated ==> evaluated
[failure] 918-918:
assigments ==> assignments🪛 GitHub Actions: Codespell
[error] 478-478: codespell: 'retreived' is a misspelling of 'retrieved'
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (21)
src/snakemake/api.py(2 hunks)src/snakemake/cli.py(7 hunks)src/snakemake/common/__init__.py(5 hunks)src/snakemake/dag.py(2 hunks)src/snakemake/exceptions.py(3 hunks)src/snakemake/io/__init__.py(8 hunks)src/snakemake/jobs.py(5 hunks)src/snakemake/logging.py(2 hunks)src/snakemake/resources.py(14 hunks)src/snakemake/rules.py(6 hunks)src/snakemake/scheduler.py(5 hunks)src/snakemake/script/__init__.py(1 hunks)src/snakemake/settings/types.py(2 hunks)src/snakemake/spawn_jobs.py(2 hunks)src/snakemake/workflow.py(7 hunks)tests/common.py(6 hunks)tests/test_all_temp/Snakefile(0 hunks)tests/test_default_resources/Snakefile(1 hunks)tests/test_github_issue3271/Snakefile_should_fail(1 hunks)tests/test_group_jobs_resources/Snakefile(2 hunks)tests/tests.py(20 hunks)
💤 Files with no reviewable changes (1)
- tests/test_all_temp/Snakefile
🚧 Files skipped from review as they are similar to previous changes (11)
- src/snakemake/spawn_jobs.py
- src/snakemake/api.py
- src/snakemake/script/init.py
- tests/test_default_resources/Snakefile
- src/snakemake/logging.py
- tests/test_group_jobs_resources/Snakefile
- src/snakemake/scheduler.py
- src/snakemake/dag.py
- src/snakemake/exceptions.py
- src/snakemake/settings/types.py
- tests/common.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest type annotations for functions that are defined inside of functions or methods. Do not suggest type annotation of the `s...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
src/snakemake/io/__init__.pytests/tests.pysrc/snakemake/resources.pysrc/snakemake/cli.pysrc/snakemake/common/__init__.pysrc/snakemake/jobs.pysrc/snakemake/rules.pysrc/snakemake/workflow.py
🧬 Code Graph Analysis (7)
src/snakemake/io/__init__.py (2)
src/snakemake/gui.py (1)
index(84-95)src/snakemake/resources.py (1)
items(999-1000)
tests/tests.py (3)
src/snakemake/exceptions.py (1)
ResourceDuplicationError(640-647)src/snakemake/resources.py (5)
GroupResources(53-426)Resources(790-1021)from_mapping(979-985)parse(910-976)defaults(1068-1069)tests/common.py (1)
run(152-511)
src/snakemake/resources.py (6)
src/snakemake/common/__init__.py (3)
get_input_function_aux_params(308-320)mb_to_mib(64-65)mib_to_mb(68-69)src/snakemake/exceptions.py (6)
ResourceConstraintError(636-637)ResourceError(599-600)ResourceInsufficiencyError(603-622)ResourceScopesException(625-629)ResourceValidationError(632-633)is_file_not_found_error(675-682)src/snakemake/common/tbdstring.py (1)
TBDString(3-96)src/snakemake/io/__init__.py (11)
AnnotatedString(944-964)Wildcards(1942-1943)items(1836-1838)get(1881-1886)inner(1498-1518)is_callable(87-87)is_callable(325-326)is_callable(955-956)is_callable(1152-1157)keys(1869-1870)update(1811-1817)src/snakemake/jobs.py (22)
jobs(1346-1347)jobs(1350-1351)Job(177-1287)resources(466-483)resources(1491-1505)name(1259-1260)name(1642-1646)group(446-449)group(452-453)wildcards(314-315)wildcards(318-319)input(322-323)input(326-327)input(1512-1517)attempt(456-457)attempt(460-463)attempt(1713-1714)attempt(1717-1722)threads(408-409)threads(1706-1707)is_local(1014-1022)is_local(1662-1663)src/snakemake/workflow.py (8)
resources(2151-2156)name(2235-2240)group(2165-2170)func(1630-1636)input(1983-1988)attempt(383-387)threads(2130-2135)is_local(676-680)
src/snakemake/cli.py (2)
src/snakemake/resources.py (9)
Resource(472-787)Resources(790-1021)from_cli_expression(682-727)args(855-860)parse(910-976)value(550-560)default(870-888)parser_factory(891-907)defaults(1068-1069)src/snakemake/common/__init__.py (1)
parse_key_value_arg(72-79)
src/snakemake/common/__init__.py (1)
src/snakemake/exceptions.py (1)
NestedCoroutineError(550-551)
src/snakemake/rules.py (4)
src/snakemake/io/__init__.py (3)
ResourceList(1950-1951)items(1836-1838)get(1881-1886)src/snakemake/jobs.py (15)
resources(466-483)resources(1491-1505)threads(408-409)threads(1706-1707)name(1259-1260)name(1642-1646)wildcards(314-315)wildcards(318-319)input(322-323)input(326-327)input(1512-1517)attempt(456-457)attempt(460-463)attempt(1713-1714)attempt(1717-1722)src/snakemake/resources.py (9)
Resource(472-787)Resources(790-1021)evaluate(566-583)constrain(585-608)value(550-560)items(999-1000)is_evaluable(562-564)get(1017-1021)to_mib(610-635)src/snakemake/exceptions.py (4)
ResourceConstraintError(636-637)ResourceValidationError(632-633)NestedCoroutineError(550-551)InputFunctionException(250-262)
src/snakemake/workflow.py (4)
src/snakemake/exceptions.py (1)
ResourceValidationError(632-633)src/snakemake/jobs.py (6)
resources(466-483)resources(1491-1505)name(1259-1260)name(1642-1646)threads(408-409)threads(1706-1707)src/snakemake/resources.py (8)
Resources(790-1021)value(550-560)copy(1013-1015)get(1017-1021)args(855-860)from_mapping(979-985)update(987-991)update(1162-1165)src/snakemake/rules.py (3)
name(137-138)name(141-142)Rule(83-1327)
🪛 Ruff (0.11.9)
tests/tests.py
1146-1146: Ambiguous variable name: l
(E741)
1146-1146: Use l != "\n" instead of not l == "\n"
Replace with != operator
(SIM201)
src/snakemake/resources.py
4-4: dataclasses.dataclass imported but unused
Remove unused import: dataclasses.dataclass
(F401)
219-219: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
536-539: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block
Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value
(SIM108)
765-776: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
867-867: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
881-881: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/cli.py
227-227: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
234-234: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/common/__init__.py
106-112: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/jobs.py
38-38: snakemake.io.is_callable imported but unused
Remove unused import: snakemake.io.is_callable
(F401)
src/snakemake/rules.py
1125-1131: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/workflow.py
1792-1795: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
1804-1809: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py
[failure] 478-478:
retreived ==> retrieved
[failure] 483-483:
intepreted ==> interpreted
[failure] 694-694:
evaulated ==> evaluated
[failure] 918-918:
assigments ==> assignments
src/snakemake/rules.py
[failure] 1089-1089:
inital ==> initial
🪛 GitHub Actions: Codespell
src/snakemake/resources.py
[error] 478-478: codespell: 'retreived' is a misspelling of 'retrieved'
⏰ Context from checks skipped due to timeout of 90000ms (30)
- GitHub Check: tests (10, windows-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (9, macos-latest, py312)
- GitHub Check: tests (9, windows-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (8, macos-latest, py312)
- GitHub Check: tests (8, windows-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (7, macos-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (6, macos-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (5, macos-latest, py312)
- GitHub Check: tests (5, windows-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (4, windows-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (3, windows-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (2, windows-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py311)
🔇 Additional comments (53)
tests/test_github_issue3271/Snakefile_should_fail (1)
7-7: LGTM! The resource specification correctly tests the new resource evaluation system.The change from
attempt * 1000toinput.size_mbaligns with the PR's objective of standardizing resource handling and tests the enhanced resource evaluation capabilities.src/snakemake/common/__init__.py (4)
15-15: Imports correctly updated for type annotations and new exception handling.The addition of
AnyandMappingsupports the new type annotations, and the switch toNestedCoroutineErroraligns with the new exception hierarchy.Also applies to: 24-24
64-64: Good addition of type annotation for clarity.The explicit
inttype annotation improves code clarity and type safety.
68-70: Correct implementation of MiB to MB conversion.The function correctly implements the inverse of
mb_to_mibusing the appropriate conversion factor and floor operation to ensure integer results.
308-310: Well-designed type annotations improve function clarity.The use of
Mappinginstead ofDictcorrectly indicates that the function doesn't modify the parameters, and the return type properly reflects the filtered mapping.src/snakemake/io/__init__.py (8)
1-1: Good addition of future annotations for modern Python compatibility.The
from __future__ import annotationsenables postponed evaluation of annotations, improving import performance and allowing forward references.
32-33: Necessary typing imports for enhanced type safety.The addition of
Generic,Iterator, andTuplesupports the generic typing improvements in theNamedlistclass.Also applies to: 37-37
716-723: Excellent enhancement ofNamedlistwith generic typing.The introduction of type variables
_TNamedListand_TNamedKeysalong with makingNamedlistgeneric significantly improves type safety and IDE support. The docstrings clearly explain the purpose of each type variable.
732-732: Type annotation correctly uses generic parameters.The
fromdictparameter type now properly enforces type consistency with the genericNamedlistparameters.
836-836: Return type properly reflects the generic key-value pairs.The
items()method return type now correctly indicates it yields tuples of the generic key and value types.
1458-1458: Good practice usingcollections.abc.Iterablefor abstract types.Using the fully qualified name from
collections.abcis the recommended approach for abstract base classes.
1618-1618: Clear type annotation improves code readability.The explicit
List[str]type annotation fornamesmakes the code's intent clearer.
950-950:Details
❓ Verification inconclusive
Important rename from
ResourcestoResourceListto avoid naming conflicts.This rename is crucial for the resource management refactoring, as it distinguishes this class from the new
Resourcesclass inresources.py.
🏁 Script executed:
#!/bin/bash # Verify that all references to the old Resources class have been updated echo "Checking for remaining references to io.Resources that might need updating..." rg -A 3 'from snakemake\.io import.*Resources' --type py rg -A 3 'io\.Resources' --type pyLength of output: 259
Ensure complete rename from
ResourcestoResourceListI ran the provided ripgrep check and didn’t find any remaining imports or usages of the old
io.Resourcesclass. However, please manually verify that:
- All tests, examples, and documentation have been updated to use
ResourceList.- There are no stray references (e.g., in comments, docstrings, or auxiliary scripts) to the old
Resourcesname.src/snakemake/workflow.py (7)
61-61: LGTM! Import changes align with resource system refactor.The new imports for
ResourceValidationErrorandResourcesproperly support the modernized resource handling architecture.Also applies to: 119-119
167-169: LGTM! Direct assignment reflects the new typed resource architecture.The direct assignment of
self.resource_settings.resourceseliminates the need for manual copying and leverages the newResourcescontainer type.
587-587: LGTM! Proper use of the new Resource API.Accessing the
.valueattribute correctly extracts the underlying value from theResourceobjects while maintaining type safety and consistency with the new resource architecture.Also applies to: 591-591
648-648: LGTM! Good addition of type annotation.The return type annotation
-> Ruleimproves type safety and developer experience.
777-779: LGTM! Cleaner resource copying with the new API.Using
Resources.copy()is more elegant and safer than manual dictionary copying.
812-813: LGTM! Clean resource updates with the new container.The direct update from
overwrite_resourcesleverages theResourcescontainer's update capabilities effectively.
928-928: LGTM! Simplified resource updates with improved container handling.The direct update with
global_resourcesindicates that the newResourcescontainer properly handles edge cases internally, making the code cleaner and more robust.src/snakemake/rules.py (6)
28-28: Import changes look good.The new imports properly support the resource system refactor by bringing in the modernized
Resource,Resources,ResourceListclasses and related exceptions. The type checking import forWorkflowenables proper type annotations.Also applies to: 49-54, 57-57, 76-77
92-92: Type annotations improve code clarity and type safety.The type annotations for
workflowandresourcesattributes are accurate and help with IDE support and static analysis.Also applies to: 104-104
1065-1096: Well-designed resource evaluation function.The new
evaluatefunction provides comprehensive exception handling for resource evaluation, including proper handling ofResourceValidationError,NestedCoroutineError, and file-not-found scenarios. The logic for skipping evaluation and returningTBDStringfor async loop conflicts is sound.🧰 Tools
🪛 GitHub Check: Check for spelling errors
[failure] 1089-1089:
inital ==> initial
1098-1105: Proper handling of the cores resource.The explicit handling of the
_coresresource first, with constraint bymax_threadsand type validation, ensures thread count is properly managed before evaluating other resources that may depend on it.
1107-1140: Comprehensive resource processing with good error handling.The main resource evaluation loop correctly:
- Skips non-evaluable resources with
Nonevalues- Evaluates callable resources with thread context
- Applies global resource constraints with proper type checking
- Handles special
memanddiskresources to set their_mband_mibvariantsThe logic preserves the original functionality while leveraging the new resource system.
🧰 Tools
🪛 Ruff (0.11.9)
1125-1131: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
1141-1141: Return type change aligns with modernized resource system.Returning
ResourceListinstead ofResourcesis consistent with the broader refactor to separate resource containers from resource definitions, as evidenced by the usage injobs.py.src/snakemake/cli.py (6)
43-44: LGTM! Import changes align with resource standardization.The replacement of
DefaultResourcesandParsedResourcewith the newResourceandResourcesclasses is consistent with the PR objectives to centralize resource handling.
108-118: Well-executed refactoring to use new Resource system.The function now properly integrates with the new
Resourceclass while maintaining the same external interface. The wrapper approach cleanly converts thread specifications to Resource objects.
163-189: Excellent refactoring that improves parsing logic and error handling.The new implementation properly groups resource assignments by rule and leverages the standardized
Resources.parsemethod. The error handling is improved with proper exception chaining, making debugging easier for users.
523-524: Proper integration with new Resources system.The argument parser configuration correctly uses the new
Resourcesclass for both default values and parsing, maintaining consistency with the standardized resource management.
602-604: Excellent configuration for default resources.The parser configuration appropriately enables full default presets and Python expression support, which aligns perfectly with the documented default resource examples like
mem_mb=min(max(2*input.size_mb, 1000), 8000).
108-189: Comprehensive and well-executed resource system migration.This refactoring successfully migrates CLI argument parsing to use the new unified
ResourceandResourcesclasses while maintaining backward compatibility and improving error handling. The changes are consistent throughout and align perfectly with the PR objectives to standardize resource handling in Snakemake.Also applies to: 523-524, 602-604
🧰 Tools
🪛 Ruff (0.11.9)
149-149: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
src/snakemake/jobs.py (4)
96-113: Excellent refactoring of scheduler resource filtering logic.The changes improve resource handling in several ways:
Better evaluation logic: Replacing the callable check with
TBDStringcheck is more explicit about which resources cannot be evaluated yet.Cleaner scope filtering: Using
workflow.resource_scopes.is_local(k)is more maintainable than checking callable status.Type safety: Returning
ResourceList(fromdict=res_dict)instead of a plain dict provides better type consistency.These changes align well with the standardized resource handling goals of this PR.
466-466: Good type annotation improvement.Adding the explicit return type annotation
-> ResourceListimproves type safety and documentation. This is consistent with the refactoring to standardize onResourceListfor resource representations.
1322-1322: Good type annotation for constructor parameter.Adding the type annotation
global_resources: Resourcesimproves code documentation and type safety. This makes it clear what type of resource object is expected for the global resource constraints.
1501-1505: Excellent error handling improvements.The changes enhance error handling in several ways:
More specific exception handling: Catching
ResourceInsufficiencyErrorin addition toWorkflowErrorprovides better granularity for resource-related errors.Proper exception chaining: Using
from errpreserves the original exception context for better debugging.Informative error messages: Including the group ID in the error message helps with troubleshooting.
Type consistency: Returning
ResourceList(fromdict=self._resources)maintains consistent return types.These improvements align well with the goal of standardizing resource handling across the codebase.
tests/tests.py (14)
14-18: LGTM! Import changes align with the new resource management system.The addition of
WorkflowError,ResourceDuplicationError, andResourcesimports is appropriate for testing the new resource handling functionality.
670-671: Good addition: Test for thread overwriting via CLI.This test validates that threads can be dynamically set using CLI expressions, which is an important feature of the new resource system.
864-865: Good simplification of the assert_resources function.Direct dictionary comparison is cleaner and more straightforward than the previous implementation.
878-1293: Excellent comprehensive test coverage for the new resource management system!The test modifications and additions thoroughly validate:
- Resource normalization (mem_mb → mem, disk_mb → disk)
- Human-readable resource parsing
- Mebibyte unit support with proper conversion
- Prevention of duplicate resource specifications using
ResourceDuplicationError- Resource scope management (local/global)
The tests are well-structured and cover important edge cases.
🧰 Tools
🪛 Ruff (0.11.9)
1146-1146: Ambiguous variable name:
l(E741)
1146-1146: Use
l != "\n"instead ofnot l == "\n"Replace with
!=operator(SIM201)
14-18: LGTM! Import changes align with the resource refactoring.The new imports for
WorkflowError,ResourceDuplicationError,GroupResources, andResourcesproperly support the refactored resource handling system described in the PR objectives.
670-671: LGTM! Test function looks correct.The test function call is properly structured and appears to test thread overwriting functionality correctly.
864-866: LGTM! Clean helper function for resource assertions.The
assert_resourcesfunction provides a clear and simple way to verify resource dictionaries in tests.
878-903: LGTM! Test properly validates resource normalization.The updated test correctly demonstrates the new resource normalization feature where resources like
mem_mbare expanded to include both the base form (mem) and suffixed variants (mem_mb,mem_mib). The migration fromDefaultResourcestoResources.parsewithdefaults="full"andallow_expressions=Trueis appropriate.
905-930: LGTM! Test validates mebibyte resource specification.This new test properly validates that resources can be specified in mebibyte format (
mem_mib) and are correctly converted to their megabyte equivalents. The assertion confirms that all resource variants are properly computed.
932-957: LGTM! Test validates human-readable resource format.This test properly validates that global resources can be specified in human-readable format ("60 Gb") and are correctly converted to numeric values (60000 MB). This demonstrates an important usability improvement in the new resource system.
959-978: LGTM! Test validates resource duplication error handling.This parametrized test properly validates that the new resource system detects and prevents conflicting resource specifications (e.g., providing both
memandmem_mbsimultaneously). The use ofResourceDuplicationErroras the expected failure type is appropriate.
981-1005: LGTM! Tests validate suffixed resource format constraints.These tests properly validate that suffixed resources (like
mem_mb) cannot use human-readable formats, while unsuffixed base resources (likemem) can. This enforces the design constraint that human-readable formats are only allowed for base resource names.
1018-1021: LGTM! Consistent migration to Resources.parse pattern.The migration from
DefaultResourcestoResources.parse(["mem_mb=0"], defaults="full", allow_expressions=True)is consistently applied across multiple test functions. This aligns with the resource system refactoring described in the PR objectives.Also applies to: 1048-1050, 1087-1089, 1113-1115, 1140-1142, 1166-1168, 1187-1189, 1209-1211, 1232-1234, 1259-1261, 1290-1292
1467-1484: LGTM! Tests validate new default resource format capabilities.These new tests properly validate that default resources can be specified in both human-readable format ("1 Gb") and mebibyte format ("953.67 MiB"), extending the new resource capabilities to default resource specifications.
src/snakemake/resources.py (3)
1027-1166: Excellent implementation of ResourceScopes with proper validation!The class properly:
- Validates resource scopes (local/global/excluded)
- Normalizes size resources to prevent duplicates
- Provides convenient helper methods for scope checking
- Handles suffixed resources correctly
602-608:⚠️ Potential issueUse isinstance() for type checking instead of direct type comparison.
Direct type comparison is fragile and doesn't work with subclasses. Use
isinstance()for proper type checking.self_val = self.value -if not isinstance(self_val, type(other_val)): +if type(self_val) != type(other_val): raise ResourceConstraintError(self_val, other_val) if isinstance(self_val, int): assert isinstance(other_val, int) return Resource(self.name, min(other_val, self_val)) return selfLikely an incorrect or invalid review comment.
272-281:⚠️ Potential issueFix mutable default arguments to prevent unexpected behavior.
Using mutable objects as default arguments can lead to unexpected behavior when the function is called multiple times.
def _merge_resource_dict( cls, resources: List[Dict[str, str | int]], - skip: Optional[List[str]] = None, - methods: Optional[Dict[str, Callable[[List[int]], int]]] = None, + skip: List[str] | None = None, + methods: Dict[str, Callable[[List[int]], int]] | None = None, default_method: Callable[[List[int]], int] = max, ): skip = skip or [] methods = methods or {}Likely an incorrect or invalid review comment.
c13583a to
287e661
Compare
There was a problem hiding this comment.
Actionable comments posted: 5
♻️ Duplicate comments (6)
src/snakemake/rules.py (1)
1129-1135: Exception handling follows codebase conventions.The exception handling here correctly avoids exception chaining, which is consistent with the codebase's backward compatibility requirements. The error message provides sufficient context for debugging without needing the original exception chain.
src/snakemake/resources.py (5)
4-4: Remove unused import.The
dataclassimport is not used anywhere in this file.-from dataclasses import dataclass
218-219: Exception chaining issue flagged by static analysis.Based on the retrieved learnings, the Snakemake codebase avoids exception chaining for backward compatibility reasons. However, static analysis is flagging this pattern.
765-776: Exception chaining flagged by static analysis.Based on the retrieved learnings, the Snakemake codebase avoids exception chaining for backward compatibility reasons. The current pattern is consistent with the codebase standards.
867-867: Exception chaining flagged by static analysis.Based on the retrieved learnings, the Snakemake codebase avoids exception chaining for backward compatibility reasons.
881-881: Exception chaining flagged by static analysis.Based on the retrieved learnings, the Snakemake codebase avoids exception chaining for backward compatibility reasons.
🧹 Nitpick comments (3)
src/snakemake/rules.py (1)
1093-1093: Fix spelling error in comment.- # Need to catch this because both input.size_mb and the inital + # Need to catch this because both input.size_mb and the initialsrc/snakemake/jobs.py (1)
38-38: Remove unused import.The
is_callableimport is not used in this file and should be removed to clean up the imports.- is_callable,src/snakemake/cli.py (1)
218-237: Good generalization but address exception chaining.The function has been well-generalized to use a
wrapperfunction instead of afallbackparameter, providing more flexibility for value processing.However, please address the exception chaining issue flagged by static analysis:
try: assignments[key] = wrapper(value) - except Exception as e: - raise ValueError(f"{errmsg} Cause: {e}") + except Exception as e: + raise ValueError(f"{errmsg} Cause: {e}") from eThis preserves the original exception context for better debugging.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (20)
src/snakemake/api.py(2 hunks)src/snakemake/cli.py(7 hunks)src/snakemake/common/__init__.py(5 hunks)src/snakemake/dag.py(2 hunks)src/snakemake/exceptions.py(3 hunks)src/snakemake/io/__init__.py(8 hunks)src/snakemake/jobs.py(5 hunks)src/snakemake/logging.py(2 hunks)src/snakemake/resources.py(14 hunks)src/snakemake/rules.py(6 hunks)src/snakemake/scheduler.py(5 hunks)src/snakemake/script/__init__.py(1 hunks)src/snakemake/settings/types.py(2 hunks)src/snakemake/spawn_jobs.py(2 hunks)src/snakemake/workflow.py(7 hunks)tests/common.py(6 hunks)tests/test_default_resources/Snakefile(1 hunks)tests/test_github_issue3271/Snakefile_should_fail(1 hunks)tests/test_group_jobs_resources/Snakefile(2 hunks)tests/tests.py(19 hunks)
✅ Files skipped from review due to trivial changes (1)
- src/snakemake/api.py
🚧 Files skipped from review as they are similar to previous changes (12)
- tests/test_default_resources/Snakefile
- tests/test_group_jobs_resources/Snakefile
- src/snakemake/spawn_jobs.py
- src/snakemake/logging.py
- tests/test_github_issue3271/Snakefile_should_fail
- src/snakemake/script/init.py
- src/snakemake/scheduler.py
- src/snakemake/dag.py
- src/snakemake/settings/types.py
- tests/common.py
- src/snakemake/exceptions.py
- src/snakemake/io/init.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit Configuration File
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
tests/tests.pysrc/snakemake/resources.pysrc/snakemake/cli.pysrc/snakemake/common/__init__.pysrc/snakemake/jobs.pysrc/snakemake/rules.pysrc/snakemake/workflow.py
🧠 Learnings (8)
📓 Common learnings
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
tests/tests.py (4)
Learnt from: johanneskoester
PR: #3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in tests/test_wrapper/Snakefile, are for testing purposes and do not require updates to the project documentation.
Learnt from: johanneskoester
PR: #3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.
Learnt from: johanneskoester
PR: #3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the tests/test_cores_cluster/qsub script is a dummy script for testing, and input validation and error handling are not required in such scripts.
Learnt from: mbhall88
PR: #3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.
src/snakemake/resources.py (15)
Learnt from: johanneskoester
PR: #3026
File: snakemake/assets/init.py:0-0
Timestamp: 2024-08-27T20:15:58.699Z
Learning: Do not suggest type annotations for cls arguments in class methods when reviewing code for the Snakemake project.
Learnt from: johanneskoester
PR: #3026
File: snakemake/assets/init.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Do not suggest type annotations for cls arguments in class methods when reviewing code for the Snakemake project.
Learnt from: johanneskoester
PR: #3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In snakemake/dag.py, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.
Learnt from: johanneskoester
PR: #3014
File: snakemake/dag.py:1265-1266
Timestamp: 2024-10-11T07:09:54.544Z
Learning: For the Snakemake project, avoid suggesting trailing commas as they use Black as an autoformatter.
Learnt from: johanneskoester
PR: #3014
File: snakemake/dag.py:1265-1266
Timestamp: 2024-08-13T16:09:09.726Z
Learning: For the Snakemake project, avoid suggesting trailing commas as they use Black as an autoformatter.
Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-08-13T16:22:09.641Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.
Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.
Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the snakemake codebase, exception chaining (using raise ... from ...) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in snakemake codebase.
Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-10-08T17:41:54.542Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.
Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.
Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the snakemake codebase, especially when the cases handled don't make sense.
Learnt from: johanneskoester
PR: #3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.
Learnt from: johanneskoester
PR: #3107
File: snakemake/logging.py:510-515
Timestamp: 2025-01-14T14:04:30.554Z
Learning: In Snakemake's logging handlers, KeyboardInterrupt and SystemExit exceptions should be ignored (pass) rather than re-raised, as these are control flow signals that should not disrupt the logging process. The exception variable in the final except block should be kept for potential debugging purposes.
src/snakemake/cli.py (1)
Learnt from: johanneskoester
PR: #3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the snakemake/cli.py file, ExecutorPluginRegistry() is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.
src/snakemake/common/__init__.py (14)
Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.
Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-08-13T16:22:09.641Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.
Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-08-13T16:07:33.369Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.
Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.
Learnt from: johanneskoester
PR: #2985
File: tests/tests.py:2051-2051
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.
Learnt from: johanneskoester
PR: #2985
File: tests/tests.py:2051-2051
Timestamp: 2024-08-13T09:25:24.046Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.
Learnt from: johanneskoester
PR: #3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in tests/test_wrapper/Snakefile, are for testing purposes and do not require updates to the project documentation.
Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Learnt from: johanneskoester
PR: #2927
File: snakemake/ioutils/branch.py:44-51
Timestamp: 2024-12-14T13:13:09.273Z
Learning: Snakemake needs to maintain compatibility with Python 3.7; avoid using syntax or features introduced after Python 3.7.
Learnt from: johanneskoester
PR: #2927
File: snakemake/ioutils/evaluate.py:13-17
Timestamp: 2024-12-14T13:05:08.032Z
Learning: The Snakemake project supports Python 3.7, so code contributions must remain compatible with Python 3.7 and avoid features introduced in later Python versions.
Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the snakemake codebase, exception chaining (using raise ... from ...) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in snakemake codebase.
Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-10-08T17:41:54.542Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.
Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.
src/snakemake/jobs.py (2)
Learnt from: lczech
PR: #3113
File: snakemake/scheduler.py:912-914
Timestamp: 2024-10-04T16:12:18.927Z
Learning: In snakemake/scheduler.py, avoid suggesting the use of asyncio.gather in the jobs_rewards method due to overhead concerns and the need for immediate results.
Learnt from: johanneskoester
PR: #3600
File: src/snakemake/jobs.py:960-964
Timestamp: 2025-05-23T09:40:24.474Z
Learning: In the cleanup method of the Job class in src/snakemake/jobs.py, files in the to_remove list should be formatted with fmt_iofile without specifying as_output=True or as_input=True parameters, as these files should be displayed as generic files rather than specifically as output files.
src/snakemake/rules.py (8)
Learnt from: johanneskoester
PR: #2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In snakemake/ioutils/evaluate.py, the use of eval() with globals() is acceptable because it's called by the user under consideration of the implications.
Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the snakemake codebase, exception chaining (using raise ... from ...) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in snakemake codebase.
Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-10-08T17:41:54.542Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.
Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.
Learnt from: johanneskoester
PR: #3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.
Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the snakemake codebase, especially when the cases handled don't make sense.
src/snakemake/workflow.py (6)
Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the snakemake codebase, exception chaining (using raise ... from ...) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in snakemake codebase.
Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the snakemake codebase, especially when the cases handled don't make sense.
Learnt from: johanneskoester
PR: #3132
File: snakemake/rules.py:1105-1105
Timestamp: 2024-10-11T12:49:08.705Z
Learning: In snakemake/rules.py, within the Rule class, self.basedir is a custom class (not a pathlib.Path object) that has a join() method. Therefore, using self.basedir.join(conda_env) is appropriate in this context.
Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Learnt from: kdm9
PR: #3562
File: src/snakemake/checkpoints.py:90-90
Timestamp: 2025-05-06T01:37:23.382Z
Learning: In Snakemake checkpoints implementation, tracking only the first missing output for each checkpoint is sufficient, because if one output is missing, all outputs for that checkpoint are considered incomplete. This was the behavior before PR #3562 and maintained in the pluralized checkpoints.get() implementation.
🧬 Code Graph Analysis (6)
tests/tests.py (3)
src/snakemake/exceptions.py (1)
ResourceDuplicationError(640-647)src/snakemake/resources.py (4)
Resources(790-1021)from_mapping(979-985)parse(910-976)defaults(1068-1069)tests/common.py (1)
run(152-511)
src/snakemake/resources.py (6)
src/snakemake/common/__init__.py (3)
get_input_function_aux_params(308-320)mb_to_mib(64-65)mib_to_mb(68-69)src/snakemake/exceptions.py (7)
ResourceConstraintError(636-637)ResourceDuplicationError(640-647)ResourceError(599-600)ResourceInsufficiencyError(603-622)ResourceScopesException(625-629)ResourceValidationError(632-633)is_file_not_found_error(675-682)src/snakemake/common/tbdstring.py (1)
TBDString(3-96)src/snakemake/io/__init__.py (9)
AnnotatedString(933-953)Wildcards(1941-1942)items(1835-1837)get(1880-1885)is_callable(314-315)is_callable(944-945)is_callable(1151-1156)keys(1868-1869)update(1810-1816)src/snakemake/jobs.py (20)
jobs(1346-1347)jobs(1350-1351)Job(177-1287)resources(466-483)resources(1491-1505)name(1259-1260)name(1642-1646)group(446-449)group(452-453)wildcards(314-315)wildcards(318-319)input(322-323)input(326-327)input(1512-1517)attempt(456-457)attempt(460-463)attempt(1713-1714)attempt(1717-1722)is_local(1014-1022)is_local(1662-1663)src/snakemake/workflow.py (7)
resources(2167-2172)name(2251-2256)group(2181-2186)func(1646-1652)input(1999-2004)attempt(390-394)is_local(687-691)
src/snakemake/cli.py (2)
src/snakemake/resources.py (10)
Resource(472-787)Resources(790-1021)from_cli_expression(682-727)args(855-860)parse(910-976)items(999-1000)value(550-560)default(870-888)parser_factory(891-907)defaults(1068-1069)src/snakemake/common/__init__.py (1)
parse_key_value_arg(72-79)
src/snakemake/common/__init__.py (1)
src/snakemake/exceptions.py (1)
NestedCoroutineError(550-551)
src/snakemake/rules.py (5)
src/snakemake/io/__init__.py (2)
ResourceList(1949-1950)items(1835-1837)src/snakemake/jobs.py (13)
resources(466-483)resources(1491-1505)name(1259-1260)name(1642-1646)wildcards(314-315)wildcards(318-319)input(322-323)input(326-327)input(1512-1517)attempt(456-457)attempt(460-463)attempt(1713-1714)attempt(1717-1722)src/snakemake/resources.py (8)
Resource(472-787)Resources(790-1021)evaluate(566-583)constrain(585-608)value(550-560)items(999-1000)is_evaluable(562-564)to_mib(610-635)src/snakemake/exceptions.py (4)
ResourceConstraintError(636-637)ResourceValidationError(632-633)NestedCoroutineError(550-551)InputFunctionException(250-262)src/snakemake/common/tbdstring.py (1)
TBDString(3-96)
src/snakemake/workflow.py (4)
src/snakemake/exceptions.py (2)
ResourceValidationError(632-633)RuleException(209-247)src/snakemake/jobs.py (6)
resources(466-483)resources(1491-1505)name(1259-1260)name(1642-1646)threads(408-409)threads(1706-1707)src/snakemake/resources.py (8)
Resources(790-1021)value(550-560)copy(1013-1015)get(1017-1021)args(855-860)from_mapping(979-985)update(987-991)update(1162-1165)src/snakemake/rules.py (3)
name(138-139)name(142-143)Rule(84-1331)
🪛 Ruff (0.12.2)
tests/tests.py
1162-1162: Ambiguous variable name: l
(E741)
1162-1162: Use l != "\n" instead of not l == "\n"
Replace with != operator
(SIM201)
src/snakemake/resources.py
4-4: dataclasses.dataclass imported but unused
Remove unused import: dataclasses.dataclass
(F401)
219-219: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
536-539: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block
Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value
(SIM108)
765-776: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
867-867: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
881-881: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/cli.py
227-227: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
234-234: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/common/__init__.py
106-112: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/jobs.py
38-38: snakemake.io.is_callable imported but unused
Remove unused import: snakemake.io.is_callable
(F401)
src/snakemake/rules.py
1129-1135: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/workflow.py
1808-1811: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
1820-1825: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py
[failure] 918-918:
assigments ==> assignments
[failure] 694-694:
evaulated ==> evaluated
[failure] 483-483:
intepreted ==> interpreted
[failure] 478-478:
retreived ==> retrieved
src/snakemake/rules.py
[failure] 1093-1093:
inital ==> initial
🪛 GitHub Actions: Codespell
src/snakemake/resources.py
[error] 478-478: codespell: 'retreived' is a misspelling; did you mean 'retrieved'?
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (56)
- GitHub Check: tests (10, ubuntu-latest, py313)
- GitHub Check: tests (8, ubuntu-latest, py313)
- GitHub Check: tests (10, windows-latest, py312)
- GitHub Check: tests (8, windows-latest, py313)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py313)
- GitHub Check: tests (10, windows-latest, py313)
- GitHub Check: tests (9, windows-latest, py313)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (9, windows-latest, py312)
- GitHub Check: tests (8, windows-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (7, windows-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (4, windows-latest, py313)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (7, windows-latest, py313)
- GitHub Check: tests (6, windows-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (5, windows-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py313)
- GitHub Check: tests (4, windows-latest, py312)
- GitHub Check: tests (5, windows-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py313)
- GitHub Check: tests (6, windows-latest, py312)
- GitHub Check: tests (2, windows-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py313)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py313)
- GitHub Check: tests (4, ubuntu-latest, py313)
- GitHub Check: tests (2, windows-latest, py313)
- GitHub Check: tests (3, windows-latest, py313)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (2, macos-latest, py312)
- GitHub Check: tests (1, macos-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py313)
- GitHub Check: tests (2, macos-latest, py313)
- GitHub Check: tests (3, windows-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (3, macos-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (1, windows-latest, py313)
- GitHub Check: tests (1, ubuntu-latest, py311)
- GitHub Check: tests (1, macos-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: tests (1, windows-latest, py312)
- GitHub Check: apidocs
🔇 Additional comments (38)
src/snakemake/common/__init__.py (6)
15-15: LGTM!The additional typing imports (
AnyandMapping) are appropriately added to support the type annotations introduced in this PR.
24-24: LGTM!The change to import
NestedCoroutineErrorinstead ofWorkflowErroris appropriate for the more specific exception handling in theasync_runfunction and aligns with the new exception hierarchy.
64-64: LGTM!The type annotation for the
mbparameter is appropriate and improves code clarity by explicitly indicating the expected integer type.
68-70: LGTM!The new
mib_to_mbfunction appropriately complements the existingmb_to_mibfunction, using the correct mathematical inverse (floor division) and maintaining consistent type annotations.
106-112: LGTM!The change to raise
NestedCoroutineErrorinstead ofWorkflowErrorprovides better exception specificity for async execution contexts while maintaining the same error message and behavior.
308-310: LGTM!The type annotations accurately describe the function interface: a callable accepting any arguments, a mapping of string keys to any values for parameters, and returning a similar mapping. This improves code clarity and IDE support.
src/snakemake/workflow.py (9)
61-61: LGTM! Import changes align with new resource system.The additions of
ResourceValidationErrorandResourcesimports are consistent with the centralized resource handling approach described in the PR objectives.Also applies to: 119-119
167-167: LGTM! Correct integration with new Resources class.The direct assignment reflects that
resource_settings.resourcesis now aResourcesobject rather than a dictionary, which aligns with the new centralized resource handling system.
598-598: LGTM! Correct adaptation to Resource object structure.The
.valueattribute access is consistent with the newResourceclass design, where resource values are encapsulated withinResourceobjects rather than stored as raw values.Also applies to: 602-602
659-659: LGTM! Good addition of type annotation.The return type annotation improves code clarity and type safety without changing functionality.
793-795: LGTM! Proper initialization with new Resources class.The changes correctly use
Resources.copy()for copying default resources andResources()constructor for empty resource initialization, aligning with the new resource management system.
799-811: LGTM! Proper resource validation and error handling.The code correctly catches
ResourceValidationErrorand converts it toRuleExceptionwith an appropriate error message. The exception handling follows snakemake's convention of not using exception chaining for backward compatibility.
813-826: LGTM! Correct resource parsing with improved error handling.The use of
Resources.from_mapping()is the proper way to convert resource mappings in the new system. The error handling correctly catchesKeyErrorand provides informative error messages, following snakemake's exception handling patterns.
828-829: LGTM! Direct update with Resources object.The direct update is correct since
overwrite_resourcesnow containsResourcesobjects rather than dictionaries, consistent with the new resource management approach.
944-944: LGTM! Correct handover resource handling.The direct update with
global_resourcesis appropriate since it's now aResourcesobject that handles resource management internally, eliminating the need for manual filtering ofNonevalues.src/snakemake/rules.py (3)
28-28: LGTM! New imports support the resource refactoring.The added imports for the new resource management classes (
Resource,Resources,ResourceList) and related exceptions are appropriate for the modernized resource handling system.Also applies to: 50-55, 58-58, 77-78
93-93: LGTM! Type annotations improve code clarity.The type annotations for
workflowandresourcesattributes enhance type safety and align with the new resource management system.Also applies to: 105-105
1065-1145: Excellent refactoring of resource evaluation logic.The rewritten
expand_resourcesmethod significantly improves resource handling with:
- Clear separation of concerns via the nested
evaluatefunction- Proper exception handling for
ResourceValidationError,NestedCoroutineError, and input function exceptions- Explicit constraint enforcement with meaningful error messages
- Automatic conversion of memory/disk resources to
_mband_mibvariants- Type-safe return of
ResourceListinstead of plain dictionaryThe implementation correctly handles the core resource evaluation flow: evaluate
_coresfirst with thread constraints, then process other resources with global constraint validation.src/snakemake/jobs.py (4)
96-113: LGTM!The scheduler resource filtering logic has been correctly updated to work with the new resource system. The iteration over resource items and direct TBDString checking on values is appropriate, and the use of ResourceList constructor aligns with the refactored architecture.
466-466: LGTM!The explicit return type annotation for
ResourceListimproves type safety and aligns with the resource system refactor where ResourceList is used for job resource contexts.
1322-1322: LGTM!The type annotation for
global_resourcesparameter correctly specifies theResourcestype, improving type safety and aligning with the refactored resource system.
1501-1505: LGTM!The enhanced error handling correctly catches the new
ResourceInsufficiencyErrorexception type and provides proper exception chaining with group context. The conversion toResourceListaligns with the refactored resource architecture.src/snakemake/cli.py (5)
12-12: LGTM! Import changes align with resource system refactoring.The addition of
Unionto typing imports and the replacement of legacy resource classes (DefaultResources,ParsedResource,eval_resource_expression) with the newResourceandResourcesclasses is consistent with the PR's objective to standardize resource handling.Also applies to: 43-44
109-118: Excellent integration with the new resource system.The function now properly uses
Resource.from_cli_expressionto parse thread specifications, which aligns with the unified resource parsing approach. The wrapper function cleanly encapsulates the resource creation logic and thewith_threads_arg=Falseparameter appropriately indicates this is for thread-specific parsing.
163-189: Excellent simplification and integration with Resources class.The function has been significantly simplified by delegating the complex parsing logic to
Resources.parse(). The new approach:
- Groups resource assignments by rule name using
defaultdict- Uses the standardized
Resources.parse(exprs, allow_expressions=True)method- Maintains proper error handling with context preservation via
from errThis change eliminates duplicate parsing logic and ensures consistent resource validation across the codebase.
525-526: Proper integration with Resources class.The changes correctly update the
--resourcesargument to:
- Use
Resources()as the default instead of an empty dict- Use
Resources.parser_factory()for standardized parsingThis ensures consistent resource handling throughout the CLI interface.
604-606: Correct configuration of default resources parser.The parser configuration properly uses:
maybe_base64()wrapper for encoding supportdefaults="full"to enable the full default resource setallow_expressions=Trueto support Python expressions in default resourcesThis aligns with the documented behavior of allowing expressions like
2*input.size_mbin default resource specifications.tests/tests.py (8)
15-18: LGTM - Import changes support the new resource system.The addition of
ResourceDuplicationErrorand the updatedResourcesimport align with the resource management refactor described in the PR objectives.
674-675: LGTM - Good test coverage for thread overwriting.This test appropriately verifies the
--set-threadsfunctionality with expression evaluation.
880-918: LGTM - Resource normalization testing is comprehensive.The changes properly test the new resource system where:
- Resources are normalized from suffixed to base forms (e.g.,
mem_mb→mem)- All variants are automatically calculated and available (mem, mem_mb, mem_mib)
- Default resources use the simplified list format
- Mathematical conversions are correct (60000 MB = 57221 MiB)
921-1021: Excellent test coverage for new resource system features.These tests comprehensively verify the key capabilities introduced in the resource refactor:
- Mebibyte support: Tests that
mem_mibspecifications work correctly- Human-readable formats: Validates that global resources can use formats like "60 Gb"
- Duplicate detection: Properly tests
ResourceDuplicationErrorfor conflicting resource keys- Validation constraints: Ensures suffixed resources cannot use human-readable formats
The parameterized testing approach for duplicate detection is particularly well-designed.
1034-1036: LGTM - Clean migration to new Resources API.The systematic replacement of complex
DefaultResourcesconstruction withResources.parse()provides a cleaner, more maintainable approach:
- String-based resource definitions are more readable
- Explicit
defaults="full"andallow_expressions=Trueparameters clarify intent- Consistent pattern across all affected test functions
Also applies to: 1064-1066, 1103-1105, 1129-1131, 1156-1158, 1182-1184, 1203-1205, 1225-1227, 1248-1250, 1275-1277, 1306-1308
1211-1252: LGTM - Good integration testing for cluster resource submission.These tests properly verify:
- Resource normalization during cluster submission (mem_mb → mem)
- Correct handling of resource scopes (excluded resources not submitted)
- Integration with executor plugin interfaces
The use of mocking to test the submission interface is appropriate and follows good testing practices.
1141-1165: LGTM - Important edge case testing for resource scope handling.This test validates that resource scope overrides work correctly when users specify base resource names (e.g.,
mem) to override scopes for prefixed variants (e.g.,mem_mb). This addresses a potentially confusing user scenario and ensures consistent behavior across the normalization system.
1484-1499: LGTM - Good coverage for human-readable default resources.These tests properly verify that default resources can be specified using human-readable formats like "1 Gb" and "0.001 TB", as well as mebibyte specifications. This complements the earlier tests for global resource limits.
src/snakemake/resources.py (3)
603-604: LGTM - Improved type checking.Good use of isinstance() for type checking instead of direct type comparison. This is a more robust approach.
790-822: Excellent resource management design.The Resources class provides a clean abstraction with proper validation, normalization of size resources, and clear separation of concerns. The handling of suffixed resources (MB/MiB) is particularly well-implemented.
1027-1165: Well-designed resource scopes management.The ResourceScopes class provides a clean, consistent interface for managing resource scope assignments. The normalization of size resources matches the pattern used in the Resources class, and the property-based interface for querying scopes is intuitive and well-documented.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (7)
src/snakemake/resources.py (7)
4-4: Remove unused import.The
dataclassimport is not used anywhere in this file.
479-479: Fix spelling error.The word "retreived" should be "retrieved".
484-484: Fix spelling error.The word "intepreted" should be "interpreted".
695-695: Fix spelling error.The word "evaulated" should be "evaluated".
771-771: Fix spelling error.The word "intepreted" should be "interpreted".
776-776: Fix spelling error.The word "interpetation" should be "interpretation".
925-925: Fix spelling error.The word "assigments" should be "assignments".
🧹 Nitpick comments (1)
src/snakemake/resources.py (1)
604-604: Simplify type comparison logic.The current type comparison logic can be simplified and made more readable.
- if not isinstance(self_val, type(other_val)): + if type(self_val) != type(other_val):This is more direct and follows the pattern used elsewhere in the codebase for type comparisons.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
src/snakemake/resources.py(14 hunks)src/snakemake/scheduler.py(4 hunks)tests/test_ioutils/Snakefile(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- src/snakemake/scheduler.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit Configuration File
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
src/snakemake/resources.py
🧠 Learnings (19)
📓 Common learnings
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
📚 Learning: changes made within test cases, such as in `tests/test_wrapper/snakefile`, are for testing purposes ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.
Applied to files:
tests/test_ioutils/Snakefile
📚 Learning: new test dependencies for snakemake should be introduced in separate prs rather than being added as ...
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
Applied to files:
tests/test_ioutils/Snakefile
📚 Learning: in `snakemake/assets/__init__.py`, the `version` placeholders in asset urls refer to the versions of...
Learnt from: johanneskoester
PR: snakemake/snakemake#3203
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-11-21T10:15:16.961Z
Learning: In `snakemake/assets/__init__.py`, the `version` placeholders in asset URLs refer to the versions of external assets, not the Snakemake version. Therefore, these placeholders should not be flagged for version inconsistencies related to Snakemake.
Applied to files:
tests/test_ioutils/Snakefile
📚 Learning: the file "test.jsonl" in tests/test_nonstr_params is automatically created by snakemake, rather than...
Learnt from: johanneskoester
PR: snakemake/snakemake#2925
File: tests/test_nonstr_params/Snakefile:14-15
Timestamp: 2024-12-21T15:10:31.992Z
Learning: The file "test.jsonl" in tests/test_nonstr_params is automatically created by Snakemake, rather than manually generated in the Snakefile.
Applied to files:
tests/test_ioutils/Snakefile
📚 Learning: in the snakemake repository, avoid suggesting return type annotations for test functions....
Learnt from: johanneskoester
PR: snakemake/snakemake#2985
File: tests/tests.py:2051-2051
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.
Applied to files:
tests/test_ioutils/Snakefile
📚 Learning: in `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoi...
Learnt from: johanneskoester
PR: snakemake/snakemake#3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.
Applied to files:
src/snakemake/resources.py
📚 Learning: do not suggest type annotations for `cls` arguments in class methods when reviewing code for the sna...
Learnt from: johanneskoester
PR: snakemake/snakemake#3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.
Applied to files:
src/snakemake/resources.py
📚 Learning: for the snakemake project, avoid suggesting trailing commas as they use black as an autoformatter....
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/dag.py:1265-1266
Timestamp: 2024-10-11T07:09:54.544Z
Learning: For the Snakemake project, avoid suggesting trailing commas as they use Black as an autoformatter.
Applied to files:
src/snakemake/resources.py
📚 Learning: in `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syn...
Learnt from: johanneskoester
PR: snakemake/snakemake#3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.
Applied to files:
src/snakemake/resources.py
📚 Learning: avoid suggesting type annotations for functions that are inside methods in the snakemake codebase....
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.
Applied to files:
src/snakemake/resources.py
📚 Learning: avoid adding input validation or error handling that unnecessarily complicates the code in the `snak...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.
Applied to files:
src/snakemake/resources.py
📚 Learning: snakemake needs to maintain compatibility with python 3.7; avoid using syntax or features introduced...
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/branch.py:44-51
Timestamp: 2024-12-14T13:13:09.273Z
Learning: Snakemake needs to maintain compatibility with Python 3.7; avoid using syntax or features introduced after Python 3.7.
Applied to files:
src/snakemake/resources.py
📚 Learning: in the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backw...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.
Applied to files:
src/snakemake/resources.py
📚 Learning: in `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intention...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Applied to files:
src/snakemake/resources.py
📚 Learning: the codebase must remain compatible with older python versions. avoid suggesting syntax not supporte...
Learnt from: johanneskoester
PR: snakemake/snakemake#3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.
Applied to files:
src/snakemake/resources.py
📚 Learning: the user does not want to use exception chaining with “raise ... from e” for backward compatibility ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.
Applied to files:
src/snakemake/resources.py
📚 Learning: in snakemake's logging handlers, keyboardinterrupt and systemexit exceptions should be ignored (pass...
Learnt from: johanneskoester
PR: snakemake/snakemake#3107
File: snakemake/logging.py:510-515
Timestamp: 2025-01-14T14:04:30.554Z
Learning: In Snakemake's logging handlers, KeyboardInterrupt and SystemExit exceptions should be ignored (pass) rather than re-raised, as these are control flow signals that should not disrupt the logging process. The exception variable in the final except block should be kept for potential debugging purposes.
Applied to files:
src/snakemake/resources.py
📚 Learning: in `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's ...
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.
Applied to files:
src/snakemake/resources.py
🧬 Code Graph Analysis (1)
src/snakemake/resources.py (6)
src/snakemake/common/__init__.py (3)
get_input_function_aux_params(308-320)mb_to_mib(64-65)mib_to_mb(68-69)src/snakemake/exceptions.py (8)
NestedCoroutineError(550-551)ResourceConstraintError(636-637)ResourceDuplicationError(640-647)ResourceError(599-600)ResourceInsufficiencyError(603-622)ResourceScopesException(625-629)ResourceValidationError(632-633)is_file_not_found_error(675-682)src/snakemake/common/tbdstring.py (1)
TBDString(3-96)src/snakemake/io/__init__.py (9)
AnnotatedString(933-953)Wildcards(1941-1942)items(1835-1837)get(1880-1885)is_callable(314-315)is_callable(944-945)is_callable(1151-1156)keys(1868-1869)update(1810-1816)src/snakemake/jobs.py (21)
jobs(1346-1347)jobs(1350-1351)resources(466-483)resources(1491-1505)name(1259-1260)name(1642-1646)group(446-449)group(452-453)wildcards(314-315)wildcards(318-319)input(322-323)input(326-327)input(1512-1517)attempt(456-457)attempt(460-463)attempt(1713-1714)attempt(1717-1722)threads(408-409)threads(1706-1707)is_local(1014-1022)is_local(1662-1663)src/snakemake/workflow.py (8)
resources(2167-2172)name(2251-2256)group(2181-2186)func(1646-1652)input(1999-2004)attempt(390-394)threads(2146-2151)is_local(687-691)
🪛 Ruff (0.12.2)
src/snakemake/resources.py
4-4: dataclasses.dataclass imported but unused
Remove unused import: dataclasses.dataclass
(F401)
220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
537-540: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block
Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value
(SIM108)
768-783: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
874-874: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
888-888: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py
[failure] 925-925:
assigments ==> assignments
[failure] 776-776:
interpetation ==> interpretation
[failure] 771-771:
intepreted ==> interpreted
[failure] 695-695:
evaulated ==> evaluated
[failure] 484-484:
intepreted ==> interpreted
[failure] 479-479:
retreived ==> retrieved
🪛 GitHub Actions: Codespell
src/snakemake/resources.py
[error] 479-479: codespell: 'retreived' is a misspelling; did you mean 'retrieved'?
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (56)
- GitHub Check: tests (8, windows-latest, py313)
- GitHub Check: tests (10, windows-latest, py313)
- GitHub Check: tests (10, ubuntu-latest, py313)
- GitHub Check: tests (10, windows-latest, py312)
- GitHub Check: tests (9, windows-latest, py313)
- GitHub Check: tests (7, windows-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (9, ubuntu-latest, py313)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py313)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (9, windows-latest, py312)
- GitHub Check: tests (8, windows-latest, py312)
- GitHub Check: tests (6, windows-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (6, windows-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (7, windows-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py313)
- GitHub Check: tests (5, windows-latest, py313)
- GitHub Check: tests (6, ubuntu-latest, py313)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (5, windows-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (4, windows-latest, py313)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (5, ubuntu-latest, py313)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (2, ubuntu-latest, py313)
- GitHub Check: tests (3, ubuntu-latest, py313)
- GitHub Check: tests (2, windows-latest, py313)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (3, windows-latest, py313)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py313)
- GitHub Check: tests (3, windows-latest, py312)
- GitHub Check: tests (2, macos-latest, py313)
- GitHub Check: tests (3, macos-latest, py313)
- GitHub Check: tests (4, windows-latest, py312)
- GitHub Check: tests (2, windows-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (1, macos-latest, py312)
- GitHub Check: tests (2, macos-latest, py312)
- GitHub Check: tests (1, macos-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (1, windows-latest, py313)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: tests (1, windows-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py313)
- GitHub Check: tests (1, ubuntu-latest, py311)
- GitHub Check: apidocs
🔇 Additional comments (8)
src/snakemake/resources.py (7)
219-220: Exception chaining not used per project guidelines.Based on the retrieved learnings, this project avoids exception chaining for backward compatibility reasons, so the current implementation is correct.
276-281: LGTM: Mutable default arguments fixed.The change from mutable default arguments to
Nonewith conditional assignment inside the function is correct and addresses the static analysis warning.
823-823: LGTM: Correct None comparison.Using
is Noneis the proper way to check for None values in Python.
874-874: Exception handling follows project guidelines.The exception is re-raised without chaining, which is correct per the project's backward compatibility requirements.
888-888: Exception handling follows project guidelines.The exception is re-raised without chaining, which is correct per the project's backward compatibility requirements.
1034-1173: LGTM: Well-designed ResourceScopes class.The
ResourceScopesclass provides a clean interface for managing resource scopes with proper validation, normalization of suffixed resources, and comprehensive scope querying methods. The inheritance fromDict[str, ValidScope]is appropriate and the validation logic is robust.
1-1173: Excellent refactoring of resource management system.This comprehensive refactor successfully modernizes Snakemake's resource handling with several key improvements:
- Type Safety: Introduction of
ResourceandResourcesclasses with proper type annotations- Validation: Robust validation of resource values and prevention of duplicate equivalent resources
- Unit Conversion: Seamless handling of MB/MiB conversions and human-friendly resource parsing
- Error Handling: Clear error messages and appropriate exception types
- Extensibility: Clean separation of concerns and modular design
The new system addresses the stated objectives of standardizing resource handling while maintaining backward compatibility.
tests/test_ioutils/Snakefile (1)
89-89: LGTM: Test updated for new resource handling.The assertion correctly reflects the new resource system where
resources.memnow returns the parsed numeric value (1000 MB) instead of the original string representation ("1GB"). This change is consistent with the resource refactoring insrc/snakemake/resources.py.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (3)
src/snakemake/rules.py (1)
1130-1136: Exception chaining intentionally omitted per project standards.The static analysis suggests adding exception chaining, but based on retrieved learnings, the Snakemake codebase intentionally avoids exception chaining for backward compatibility reasons.
tests/tests.py (2)
1141-1143: Fix variable naming in list comprehension.The static analysis correctly identified an ambiguous variable name that should be fixed for better readability.
Apply this diff to fix the variable naming:
- lines = [line for line in f.readlines() if line != "\n"] + lines = [line for line in f.readlines() if line != "\n"]
1148-1173: New test validates scope inheritance for resource variants.Good test ensuring that scope overrides using base resource names properly apply to all variants. However, there's a code quality issue to address.
Fix the variable naming and comparison style at line 1169:
- lines = [l for l in f.readlines() if not l == "\n"] + lines = [line for line in f.readlines() if line != "\n"]
🧹 Nitpick comments (1)
src/snakemake/rules.py (1)
1067-1098: LGTM: Well-structured resource evaluation with comprehensive error handling.The
evaluatefunction properly handles resource evaluation with appropriate exception handling for different error types. The logic correctly delegates toapply_input_functionand handles edge cases like nested coroutines.Minor spelling fix needed:
- # Need to catch this because both input.size_mb and the inital + # Need to catch this because both input.size_mb and the initial
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
src/snakemake/rules.py(7 hunks)tests/tests.py(19 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit Configuration File
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
src/snakemake/rules.pytests/tests.py
🧠 Learnings (13)
📓 Common learnings
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
📚 Learning: in `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's ...
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.
Applied to files:
src/snakemake/rules.py
📚 Learning: in the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backw...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.
Applied to files:
src/snakemake/rules.py
📚 Learning: in `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intention...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Applied to files:
src/snakemake/rules.py
📚 Learning: the codebase must remain compatible with older python versions. avoid suggesting syntax not supporte...
Learnt from: johanneskoester
PR: snakemake/snakemake#3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.
Applied to files:
src/snakemake/rules.py
📚 Learning: the user does not want to use exception chaining with “raise ... from e” for backward compatibility ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.
Applied to files:
src/snakemake/rules.py
📚 Learning: avoid adding input validation or error handling that unnecessarily complicates the code in the `snak...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.
Applied to files:
src/snakemake/rules.py
📚 Learning: in the snakemake project, type annotations are not necessary for internal helper functions....
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.
Applied to files:
src/snakemake/rules.py
📚 Learning: changes made within test cases, such as in `tests/test_wrapper/snakefile`, are for testing purposes ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.
Applied to files:
tests/tests.py
📚 Learning: in the snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing,...
Learnt from: johanneskoester
PR: snakemake/snakemake#3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.
Applied to files:
tests/tests.py
📚 Learning: in `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syn...
Learnt from: johanneskoester
PR: snakemake/snakemake#3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.
Applied to files:
tests/tests.py
📚 Learning: in the snakemake test suite, tests for configuration keys like 'group-components' may already be cov...
Learnt from: johanneskoester
PR: snakemake/snakemake#3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.
Applied to files:
tests/tests.py
📚 Learning: in test scripts within the snakemake project, concise code is preferred over verbose error handling ...
Learnt from: mbhall88
PR: snakemake/snakemake#3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.
Applied to files:
tests/tests.py
🧬 Code Graph Analysis (1)
src/snakemake/rules.py (4)
src/snakemake/io/__init__.py (3)
ResourceList(1949-1950)get(1880-1885)items(1835-1837)src/snakemake/jobs.py (13)
resources(466-483)resources(1491-1505)name(1259-1260)name(1642-1646)wildcards(314-315)wildcards(318-319)input(322-323)input(326-327)input(1512-1517)attempt(456-457)attempt(460-463)attempt(1713-1714)attempt(1717-1722)src/snakemake/resources.py (9)
Resource(473-794)Resources(797-1028)evaluate(567-584)constrain(586-609)get(1024-1028)value(551-561)items(1006-1007)is_evaluable(563-565)to_mib(611-636)src/snakemake/exceptions.py (4)
ResourceConstraintError(636-637)ResourceValidationError(632-633)NestedCoroutineError(550-551)InputFunctionException(250-262)
🪛 Ruff (0.12.2)
src/snakemake/rules.py
1130-1136: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
tests/tests.py
1169-1169: Ambiguous variable name: l
(E741)
1169-1169: Use l != "\n" instead of not l == "\n"
Replace with != operator
(SIM201)
🪛 GitHub Check: Check for spelling errors
src/snakemake/rules.py
[failure] 1091-1091:
inital ==> initial
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (56)
- GitHub Check: apidocs
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py313)
- GitHub Check: tests (10, windows-latest, py312)
- GitHub Check: tests (10, windows-latest, py313)
- GitHub Check: tests (7, windows-latest, py313)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py313)
- GitHub Check: tests (9, windows-latest, py313)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (9, windows-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (8, windows-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (8, windows-latest, py313)
- GitHub Check: tests (8, ubuntu-latest, py313)
- GitHub Check: tests (6, windows-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py313)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (7, windows-latest, py312)
- GitHub Check: tests (6, windows-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py313)
- GitHub Check: tests (5, windows-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (5, windows-latest, py313)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (4, macos-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (2, macos-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py313)
- GitHub Check: tests (3, windows-latest, py313)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (4, macos-latest, py313)
- GitHub Check: tests (4, windows-latest, py313)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (2, windows-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (3, windows-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py313)
- GitHub Check: tests (3, ubuntu-latest, py313)
- GitHub Check: tests (3, macos-latest, py312)
- GitHub Check: tests (4, windows-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (2, macos-latest, py313)
- GitHub Check: tests (2, windows-latest, py313)
- GitHub Check: tests (1, windows-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py313)
- GitHub Check: tests (1, ubuntu-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (1, ubuntu-latest, py311)
- GitHub Check: tests (1, windows-latest, py313)
- GitHub Check: tests (1, ubuntu-latest, py312)
🔇 Additional comments (28)
src/snakemake/rules.py (2)
28-28: LGTM: Clean integration with new resource system.The import additions and type annotations properly integrate with the new resource management system. The use of
TYPE_CHECKINGblock correctly avoids circular imports, and the type annotations improve code clarity without violating the coding guidelines.Also applies to: 50-55, 58-58, 77-79, 93-93, 105-105
1100-1146: LGTM: Excellent resource processing implementation.The main resource processing logic is well-implemented with:
- Proper handling of
_coresresource first with thread constraints- Type consistency enforcement between rule and global resources
- Correct evaluation flow with skip_evaluation support
- Special handling for memory and disk resources to set
_mband_mibvariants- Appropriate return of
ResourceListinstead ofResourcesThe integration with the new resource system is seamless and maintains backward compatibility.
tests/tests.py (26)
15-18: Import changes align with resource management refactoring.The addition of
ResourceDuplicationErrorand the import of the newResourcesclass properly support the PR's objectives for centralized resource handling and duplicate resource validation.
674-676: New test validates expression support in thread settings.Good addition of test coverage for setting threads using mathematical expressions, which aligns with the enhanced resource handling capabilities.
880-881: Assertion tightened to enforce exact resource equality.The change to exact equality checking aligns with the stricter resource normalization and validation introduced by the new resource management system.
902-919: Test updated to verify resource normalization and variant handling.The changes correctly validate that the new resource system properly normalizes memory and disk resources to their base forms while maintaining all variants (mem/mem_mb/mem_mib and disk/disk_mb/disk_mib).
921-946: New test validates mebibyte resource specification support.Excellent test coverage for the PR objective allowing mem_mib specifications to correctly map to their _mb variants. The conversion from 57221 MiB to 60000 MB is accurate.
947-954: Test validates resource handling with limited cores.Good addition to test resource management behavior when global core supply is constrained.
956-980: Test validates human-readable resource specification support.Excellent coverage of the PR objective supporting human-readable resources. The conversion from "60 Gb" to numeric values is correctly tested.
982-1002: Comprehensive test for preventing duplicate resource specifications.Excellent parameterized test coverage ensuring that the system correctly raises
ResourceDuplicationErrorwhen the same effective resource is specified in multiple forms, directly addressing one of the PR's key objectives.
1004-1015: Test validates format constraints on suffixed resources.Good test ensuring that suffixed resource variants (mem_mb, mem_mib) cannot use human-readable formats, preventing potential unit conversion ambiguities.
1017-1028: Test ensures consistent format constraints across resource contexts.Good coverage ensuring that the human-readable format restriction for suffixed resources applies consistently to both global and rule-level resource declarations.
1041-1058: Test updated to use new Resources.parse API.Properly migrated from the legacy DefaultResources to the new Resources.parse method, maintaining test functionality while using the updated API.
1071-1088: Consistent migration to Resources.parse API.Test properly updated to use the new resource parsing API while maintaining coverage for resource-constrained scenarios.
1095-1118: Improved test documentation and consistent API usage.Good addition of explanatory comments clarifying the test's behavior. The migration to Resources.parse is consistent with other tests.
1189-1191: Consistent API migration.Test properly updated to use Resources.parse.
1210-1212: API update for cluster scope submission test.Properly migrated to Resources.parse.
1218-1240: Excellent test for resource normalization in cluster submission.This test validates a critical aspect of the PR - ensuring that resource variants (mem_mb, disk_mb) are properly normalized to their base forms (mem, disk) when submitted to cluster executors.
1242-1260: Test validates resource exclusion from cluster submission.Good coverage of the feature allowing certain resources to be excluded from cluster submission by setting their scope to "excluded".
1282-1301: Updated for new Resources API with pipe groups.Test properly migrated to use Resources.parse while maintaining coverage for pipe-based job groups.
1313-1329: New test for insufficient cores with pipe groups.Good addition of test coverage for the edge case where pipe groups cannot be scheduled due to insufficient cores. All Resources API migrations are consistent.
1497-1500: Migrated to list-based default resources.Test updated to use the new list-based format for default resources instead of the removed DefaultResources class.
1503-1511: Test validates human-readable format for default resources.Good coverage ensuring that human-readable formats work not just for global resources but also for default resource specifications.
1513-1520: Test validates mebibyte units in default resources.Good test coverage for mebibyte support in default resources, including fractional values to verify proper rounding behavior.
1693-1693: Consistent default resources format update.Updated to use list-based default resources format.
2051-2051: Added cleanup preservation for debugging.Good addition of
cleanup=Falseto help with debugging incomplete parameter scenarios.
2082-2084: Updated default resources format.Consistent migration to list-based default resources.
2575-2575: Consistent default resources migration.Updated to use list-based default resources format.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
src/snakemake/cli.py (1)
232-238: Improve exception handling pattern.The exception handling should use proper exception chaining for better error traceability.
Apply this diff to improve exception chaining:
- raise ValueError(f"{errmsg} Cause: {e}") + raise ValueError(f"{errmsg} Cause: {e}") from etests/tests.py (1)
1172-1172: Fix comparison style for better readability.Use the
!=operator instead ofnot ==for clearer comparison logic.- lines = [line for line in f.readlines() if not line == "\n"] + lines = [line for line in f.readlines() if line != "\n"]
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
src/snakemake/cli.py(7 hunks)src/snakemake/exceptions.py(3 hunks)src/snakemake/io/__init__.py(8 hunks)src/snakemake/resources.py(14 hunks)src/snakemake/scheduler.py(4 hunks)tests/tests.py(19 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- src/snakemake/scheduler.py
- src/snakemake/exceptions.py
- src/snakemake/io/init.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit Configuration File
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
tests/tests.pysrc/snakemake/resources.pysrc/snakemake/cli.py
🧠 Learnings (19)
📓 Common learnings
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
📚 Learning: changes made within test cases, such as in `tests/test_wrapper/snakefile`, are for testing purposes ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.
Applied to files:
tests/tests.py
📚 Learning: in the snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing,...
Learnt from: johanneskoester
PR: snakemake/snakemake#3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.
Applied to files:
tests/tests.py
📚 Learning: in `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syn...
Learnt from: johanneskoester
PR: snakemake/snakemake#3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.
Applied to files:
tests/tests.pysrc/snakemake/resources.py
📚 Learning: in the snakemake test suite, tests for configuration keys like 'group-components' may already be cov...
Learnt from: johanneskoester
PR: snakemake/snakemake#3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.
Applied to files:
tests/tests.py
📚 Learning: in test scripts within the snakemake project, concise code is preferred over verbose error handling ...
Learnt from: mbhall88
PR: snakemake/snakemake#3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.
Applied to files:
tests/tests.py
📚 Learning: in `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoi...
Learnt from: johanneskoester
PR: snakemake/snakemake#3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.
Applied to files:
src/snakemake/resources.py
📚 Learning: do not suggest type annotations for `cls` arguments in class methods when reviewing code for the sna...
Learnt from: johanneskoester
PR: snakemake/snakemake#3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.
Applied to files:
src/snakemake/resources.py
📚 Learning: for the snakemake project, avoid suggesting trailing commas as they use black as an autoformatter....
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/dag.py:1265-1266
Timestamp: 2024-10-11T07:09:54.544Z
Learning: For the Snakemake project, avoid suggesting trailing commas as they use Black as an autoformatter.
Applied to files:
src/snakemake/resources.py
📚 Learning: avoid suggesting type annotations for functions that are inside methods in the snakemake codebase....
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.
Applied to files:
src/snakemake/resources.py
📚 Learning: snakemake needs to maintain compatibility with python 3.7; avoid using syntax or features introduced...
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/branch.py:44-51
Timestamp: 2024-12-14T13:13:09.273Z
Learning: Snakemake needs to maintain compatibility with Python 3.7; avoid using syntax or features introduced after Python 3.7.
Applied to files:
src/snakemake/resources.py
📚 Learning: avoid adding input validation or error handling that unnecessarily complicates the code in the `snak...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.
Applied to files:
src/snakemake/resources.py
📚 Learning: in the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backw...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.
Applied to files:
src/snakemake/resources.py
📚 Learning: in `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intention...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Applied to files:
src/snakemake/resources.py
📚 Learning: the codebase must remain compatible with older python versions. avoid suggesting syntax not supporte...
Learnt from: johanneskoester
PR: snakemake/snakemake#3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.
Applied to files:
src/snakemake/resources.py
📚 Learning: the user does not want to use exception chaining with “raise ... from e” for backward compatibility ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.
Applied to files:
src/snakemake/resources.py
📚 Learning: in snakemake's logging handlers, keyboardinterrupt and systemexit exceptions should be ignored (pass...
Learnt from: johanneskoester
PR: snakemake/snakemake#3107
File: snakemake/logging.py:510-515
Timestamp: 2025-01-14T14:04:30.554Z
Learning: In Snakemake's logging handlers, KeyboardInterrupt and SystemExit exceptions should be ignored (pass) rather than re-raised, as these are control flow signals that should not disrupt the logging process. The exception variable in the final except block should be kept for potential debugging purposes.
Applied to files:
src/snakemake/resources.py
📚 Learning: in `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's ...
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.
Applied to files:
src/snakemake/resources.py
📚 Learning: in the `snakemake/cli.py` file, `executorpluginregistry()` is a singleton class, so multiple instant...
Learnt from: johanneskoester
PR: snakemake/snakemake#3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the `snakemake/cli.py` file, `ExecutorPluginRegistry()` is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.
Applied to files:
src/snakemake/cli.py
🪛 Ruff (0.12.2)
tests/tests.py
1172-1172: Use line != "\n" instead of not line == "\n"
Replace with != operator
(SIM201)
1262-1262: Do not assert False (python -O removes these calls), raise AssertionError()
Replace assert False
(B011)
src/snakemake/resources.py
4-4: dataclasses.dataclass imported but unused
Remove unused import: dataclasses.dataclass
(F401)
539-542: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block
Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value
(SIM108)
770-784: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
875-875: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
889-889: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/cli.py
229-229: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
236-236: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py
[failure] 777-777:
interpetation ==> interpretation
🪛 GitHub Actions: Codespell
src/snakemake/resources.py
[error] 777-777: codespell: 'interpetation' should be 'interpretation'
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (56)
- GitHub Check: tests (10, ubuntu-latest, py313)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (10, windows-latest, py313)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (9, windows-latest, py312)
- GitHub Check: tests (10, windows-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (9, windows-latest, py313)
- GitHub Check: tests (9, ubuntu-latest, py313)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (8, ubuntu-latest, py313)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (8, windows-latest, py313)
- GitHub Check: tests (7, macos-latest, py312)
- GitHub Check: tests (8, windows-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (7, windows-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py313)
- GitHub Check: tests (7, windows-latest, py313)
- GitHub Check: tests (6, windows-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (5, windows-latest, py312)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (5, ubuntu-latest, py313)
- GitHub Check: tests (5, windows-latest, py313)
- GitHub Check: tests (6, windows-latest, py313)
- GitHub Check: tests (4, windows-latest, py313)
- GitHub Check: tests (3, ubuntu-latest, py313)
- GitHub Check: tests (4, ubuntu-latest, py313)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (4, windows-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (3, windows-latest, py313)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (2, macos-latest, py313)
- GitHub Check: tests (2, macos-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (2, windows-latest, py313)
- GitHub Check: tests (3, windows-latest, py312)
- GitHub Check: tests (2, windows-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (1, macos-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (1, windows-latest, py313)
- GitHub Check: tests (1, ubuntu-latest, py313)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: tests (1, macos-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py311)
- GitHub Check: tests (1, windows-latest, py312)
- GitHub Check: apidocs
🔇 Additional comments (19)
src/snakemake/cli.py (4)
12-12: LGTM!The import updates correctly replace legacy resource classes with the new unified
ResourceandResourcesclasses, aligning with the centralized resource handling objectives.Also applies to: 43-44
109-120: LGTM!The integration with
Resource.from_cli_expressionproperly unifies thread parsing with the new resource system, providing consistent expression evaluation and type safety.
165-191: LGTM!The function rewrite properly leverages
Resources.parsefor unified resource handling. The logic correctly groups assignments by rule and maintains good exception handling practices with proper exception chaining.
527-528: LGTM!The argument parser integration with
Resources()defaults andResources.parser_factory()properly unifies CLI resource parsing with the new resource system, supporting both basic resource constraints and expression-enabled default resources.Also applies to: 606-608
tests/tests.py (9)
15-18: LGTM - Import updates align with resource system refactoring.The new imports for
ResourceDuplicationErrorand updated resource classes (GroupResources,Resources) are necessary to support the enhanced resource management system introduced in this PR.
882-883: LGTM - Enhanced precision in resource testing.Changing from partial resource matching to exact equality testing reflects the improved consistency and deterministic behavior of the new
Resourcesclass. This makes the tests more precise and reliable.
924-948: LGTM - Comprehensive test for mebibyte resource support.This test properly validates that resources can be specified in mebibytes (
mem_mib) and verifies the correct conversion to megabytes, supporting one of the key features mentioned in the PR objectives.
959-983: LGTM - Human-readable global resources test.This test validates the new capability to specify global resource limits using human-readable formats (e.g., "60 Gb"), which addresses a key improvement mentioned in the PR objectives.
986-1005: LGTM - Resource duplication validation test.This parameterized test effectively validates that the system properly prevents specifying the same effective resource multiple times (e.g.,
memandmem_mbtogether), which was one of the bugs this PR aimed to fix.
1008-1018: LGTM - Validation for suffixed resource constraints.This test ensures that suffixed resources like
mem_mbcannot accept human-readable strings, maintaining the intended behavior where only unsuffixed resources support human-readable formats.
1021-1031: LGTM - Resource declaration validation test.This test validates that suffixed resource declarations in rule overrides cannot use human-readable formats, ensuring consistent behavior across the resource system.
1044-1046: LGTM - Proper migration to new resource system.The systematic replacement of the legacy
DefaultResourcesclass withResources.parse()calls properly migrates to the new resource management system. The parametersdefaults="full"andallow_expressions=Truealign with the enhanced capabilities of the new system.
675-677: LGTM - Test for resource expression evaluation.This new test validates that resource expressions can be properly evaluated when setting threads via command line, supporting the enhanced expression capabilities in the new resource system.
src/snakemake/resources.py (6)
1-52: LGTM! Imports are well-organized and type annotations are appropriate.The imports properly support the new resource management system with appropriate type annotations and forward references.
54-430: Excellent type safety improvements and error handling.The GroupResources class has been significantly improved with:
- Comprehensive type annotations for all parameters and return values
- Better error handling with specific ResourceError and ResourceInsufficiencyError exceptions
- Cleaner separation of concerns in resource processing
The refactoring maintains backward compatibility while improving code clarity.
432-466: Well-implemented unit conversion helpers.The helper functions properly handle conversion between MB and MiB for evaluable resources with appropriate type validation and error messages.
475-796: Excellent Resource class implementation with comprehensive functionality.The Resource class provides:
- Proper validation and type conversion
- Support for evaluable resources with deferred evaluation
- Unit conversion between MB and MiB
- Constraint enforcement
- Human-friendly resource parsing
- CLI expression evaluation
The design is robust and handles edge cases well, including TBDString handling and proper error messages.
798-1030: Well-designed Resources container class.The Resources class provides excellent functionality:
- Proper Mapping interface implementation
- Automatic normalization of size resources (mem, disk) and their suffixed variants
- Comprehensive CLI-style parsing with expression support
- Good default resource management
- Proper validation and error handling
The class effectively replaces the old DefaultResources with better type safety and more features.
1032-1174: Solid ResourceScopes implementation.The ResourceScopes class provides:
- Proper validation of scope values (local, global, excluded)
- Automatic normalization of size resource suffixes
- Convenient property methods and test methods for different scopes
- Good integration with the overall resource system
The class design is clean and follows the established patterns from the Resources class.
fcd7c48 to
e74152d
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (3)
tests/tests.py (1)
1172-1174: Remove debug print statement and fix comparison style.Debug print should be removed and the comparison style should be fixed (though the latter is already flagged).
- lines = [line for line in f.readlines() if not line == "\n"] - print(lines) + lines = [line for line in f.readlines() if line != "\n"]src/snakemake/resources.py (2)
4-4: Remove unused import.
478-478: Fix spelling errors in docstring.Also applies to: 483-483
🧹 Nitpick comments (2)
tests/tests.py (2)
1145-1147: Remove debug print statement.Debug print statement should be removed from the test.
- lines = [line for line in f.readlines() if line != "\n"] - print(lines) + lines = [line for line in f.readlines() if line != "\n"]
1221-1275: Address commented-out test implementations.These tests for cluster resource submission appear incomplete. They should either be:
- Completed if the functionality is ready
- Marked with
@pytest.mark.skipwith a reason if temporarily disabled- Removed if no longer needed
The tests seem important for validating resource normalization and exclusion in cluster submissions.
Would you like help implementing these tests or should we create an issue to track their completion?
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (21)
src/snakemake/api.py(2 hunks)src/snakemake/cli.py(8 hunks)src/snakemake/common/__init__.py(5 hunks)src/snakemake/dag.py(2 hunks)src/snakemake/exceptions.py(3 hunks)src/snakemake/io/__init__.py(8 hunks)src/snakemake/jobs.py(5 hunks)src/snakemake/logging.py(2 hunks)src/snakemake/resources.py(14 hunks)src/snakemake/rules.py(7 hunks)src/snakemake/scheduler.py(4 hunks)src/snakemake/script/__init__.py(1 hunks)src/snakemake/settings/types.py(2 hunks)src/snakemake/spawn_jobs.py(2 hunks)src/snakemake/workflow.py(7 hunks)tests/common.py(6 hunks)tests/test_default_resources/Snakefile(1 hunks)tests/test_github_issue3271/Snakefile_should_fail(1 hunks)tests/test_group_jobs_resources/Snakefile(2 hunks)tests/test_ioutils/Snakefile(1 hunks)tests/tests.py(19 hunks)
🚧 Files skipped from review as they are similar to previous changes (14)
- tests/test_ioutils/Snakefile
- tests/test_github_issue3271/Snakefile_should_fail
- src/snakemake/logging.py
- tests/test_group_jobs_resources/Snakefile
- tests/test_default_resources/Snakefile
- src/snakemake/scheduler.py
- src/snakemake/api.py
- src/snakemake/io/init.py
- src/snakemake/spawn_jobs.py
- tests/common.py
- src/snakemake/script/init.py
- src/snakemake/settings/types.py
- src/snakemake/dag.py
- src/snakemake/exceptions.py
👮 Files not reviewed due to content moderation or server errors (5)
- src/snakemake/common/init.py
- src/snakemake/workflow.py
- src/snakemake/rules.py
- src/snakemake/jobs.py
- src/snakemake/cli.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit Configuration File
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
tests/tests.pysrc/snakemake/resources.pysrc/snakemake/cli.pysrc/snakemake/common/__init__.pysrc/snakemake/jobs.pysrc/snakemake/rules.pysrc/snakemake/workflow.py
🧠 Learnings (28)
📓 Common learnings
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
📚 Learning: 2024-10-06T14:09:54.370Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.
Applied to files:
tests/tests.pysrc/snakemake/common/__init__.py
📚 Learning: 2024-11-12T20:22:54.184Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.
Applied to files:
tests/tests.py
📚 Learning: 2024-10-14T09:42:11.571Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.
Applied to files:
tests/tests.pysrc/snakemake/resources.py
📚 Learning: 2024-10-13T14:29:38.902Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.
Applied to files:
tests/tests.py
📚 Learning: 2024-08-13T09:25:24.046Z
Learnt from: johanneskoester
PR: snakemake/snakemake#2985
File: tests/tests.py:2051-2051
Timestamp: 2024-08-13T09:25:24.046Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.
Applied to files:
tests/tests.pysrc/snakemake/common/__init__.py
📚 Learning: 2025-01-17T12:00:09.368Z
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: pytest-benchmark is not currently part of the Snakemake project's test dependencies. Suggestions involving pytest-benchmark should be deferred until it's added to the project.
Applied to files:
tests/tests.py
📚 Learning: 2025-01-17T12:00:09.368Z
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
Applied to files:
tests/tests.py
📚 Learning: 2024-11-12T12:08:20.342Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.
Applied to files:
tests/tests.pysrc/snakemake/resources.py
📚 Learning: 2024-10-04T16:12:18.927Z
Learnt from: lczech
PR: snakemake/snakemake#3113
File: snakemake/scheduler.py:912-914
Timestamp: 2024-10-04T16:12:18.927Z
Learning: In `snakemake/scheduler.py`, avoid suggesting the use of `asyncio.gather` in the `jobs_rewards` method due to overhead concerns and the need for immediate results.
Applied to files:
tests/tests.pysrc/snakemake/jobs.py
📚 Learning: 2024-11-07T00:32:44.137Z
Learnt from: mbhall88
PR: snakemake/snakemake#3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.
Applied to files:
tests/tests.pysrc/snakemake/resources.py
📚 Learning: 2024-10-11T07:09:54.544Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/dag.py:1265-1266
Timestamp: 2024-10-11T07:09:54.544Z
Learning: For the Snakemake project, avoid suggesting trailing commas as they use Black as an autoformatter.
Applied to files:
src/snakemake/resources.py
📚 Learning: 2024-08-27T20:15:58.699Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-08-27T20:15:58.699Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.
Applied to files:
src/snakemake/resources.py
📚 Learning: 2024-08-13T16:22:09.641Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-08-13T16:22:09.641Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.
Applied to files:
src/snakemake/resources.pysrc/snakemake/common/__init__.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.
Applied to files:
src/snakemake/resources.pysrc/snakemake/rules.pysrc/snakemake/workflow.py
📚 Learning: 2024-12-14T13:13:09.273Z
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/branch.py:44-51
Timestamp: 2024-12-14T13:13:09.273Z
Learning: Snakemake needs to maintain compatibility with Python 3.7; avoid using syntax or features introduced after Python 3.7.
Applied to files:
src/snakemake/resources.pysrc/snakemake/common/__init__.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.
Applied to files:
src/snakemake/resources.pysrc/snakemake/common/__init__.pysrc/snakemake/rules.pysrc/snakemake/workflow.py
📚 Learning: 2024-10-06T14:09:26.494Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Applied to files:
src/snakemake/resources.pysrc/snakemake/common/__init__.pysrc/snakemake/rules.pysrc/snakemake/workflow.py
📚 Learning: 2024-09-25T10:55:53.963Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.
Applied to files:
src/snakemake/resources.pysrc/snakemake/common/__init__.pysrc/snakemake/rules.py
📚 Learning: 2024-12-21T15:04:25.569Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.
Applied to files:
src/snakemake/resources.pysrc/snakemake/rules.py
📚 Learning: 2025-01-14T14:04:30.554Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3107
File: snakemake/logging.py:510-515
Timestamp: 2025-01-14T14:04:30.554Z
Learning: In Snakemake's logging handlers, KeyboardInterrupt and SystemExit exceptions should be ignored (pass) rather than re-raised, as these are control flow signals that should not disrupt the logging process. The exception variable in the final except block should be kept for potential debugging purposes.
Applied to files:
src/snakemake/resources.py
📚 Learning: 2024-12-14T13:11:32.988Z
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.
Applied to files:
src/snakemake/resources.pysrc/snakemake/rules.py
📚 Learning: 2024-10-13T14:10:37.796Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the `snakemake/cli.py` file, `ExecutorPluginRegistry()` is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.
Applied to files:
src/snakemake/cli.py
📚 Learning: 2024-08-13T16:07:33.369Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-08-13T16:07:33.369Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.
Applied to files:
src/snakemake/common/__init__.py
📚 Learning: 2024-12-14T13:05:08.032Z
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/evaluate.py:13-17
Timestamp: 2024-12-14T13:05:08.032Z
Learning: The Snakemake project supports Python 3.7, so code contributions must remain compatible with Python 3.7 and avoid features introduced in later Python versions.
Applied to files:
src/snakemake/common/__init__.py
📚 Learning: 2025-05-23T09:40:24.474Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3600
File: src/snakemake/jobs.py:960-964
Timestamp: 2025-05-23T09:40:24.474Z
Learning: In the `cleanup` method of the `Job` class in `src/snakemake/jobs.py`, files in the `to_remove` list should be formatted with `fmt_iofile` without specifying `as_output=True` or `as_input=True` parameters, as these files should be displayed as generic files rather than specifically as output files.
Applied to files:
src/snakemake/jobs.py
📚 Learning: 2024-10-11T12:49:08.705Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3132
File: snakemake/rules.py:1105-1105
Timestamp: 2024-10-11T12:49:08.705Z
Learning: In `snakemake/rules.py`, within the `Rule` class, `self.basedir` is a custom class (not a `pathlib.Path` object) that has a `join()` method. Therefore, using `self.basedir.join(conda_env)` is appropriate in this context.
Applied to files:
src/snakemake/workflow.py
📚 Learning: 2025-05-06T01:37:23.382Z
Learnt from: kdm9
PR: snakemake/snakemake#3562
File: src/snakemake/checkpoints.py:90-90
Timestamp: 2025-05-06T01:37:23.382Z
Learning: In Snakemake checkpoints implementation, tracking only the first missing output for each checkpoint is sufficient, because if one output is missing, all outputs for that checkpoint are considered incomplete. This was the behavior before PR #3562 and maintained in the pluralized `checkpoints.get()` implementation.
Applied to files:
src/snakemake/workflow.py
🧬 Code Graph Analysis (3)
src/snakemake/resources.py (5)
src/snakemake/common/__init__.py (3)
get_input_function_aux_params(308-320)mb_to_mib(64-65)mib_to_mb(68-69)src/snakemake/exceptions.py (8)
NestedCoroutineError(551-552)ResourceConstraintError(637-638)ResourceDuplicationError(641-648)ResourceError(600-601)ResourceInsufficiencyError(604-623)ResourceScopesException(626-630)ResourceValidationError(633-634)is_file_not_found_error(676-683)src/snakemake/common/tbdstring.py (1)
TBDString(3-96)src/snakemake/io/__init__.py (9)
AnnotatedString(934-954)Wildcards(1942-1943)items(1836-1838)get(1881-1886)is_callable(315-316)is_callable(945-946)is_callable(1152-1157)keys(1869-1870)update(1811-1817)src/snakemake/jobs.py (20)
jobs(1346-1347)jobs(1350-1351)Job(177-1287)resources(466-483)resources(1491-1505)name(1259-1260)name(1642-1646)group(446-449)group(452-453)wildcards(314-315)wildcards(318-319)input(322-323)input(326-327)input(1512-1517)attempt(456-457)attempt(460-463)attempt(1713-1714)attempt(1717-1722)is_local(1014-1022)is_local(1662-1663)
src/snakemake/common/__init__.py (1)
src/snakemake/exceptions.py (1)
NestedCoroutineError(551-552)
src/snakemake/workflow.py (4)
src/snakemake/exceptions.py (1)
ResourceValidationError(633-634)src/snakemake/jobs.py (6)
resources(466-483)resources(1491-1505)name(1259-1260)name(1642-1646)threads(408-409)threads(1706-1707)src/snakemake/resources.py (8)
Resources(798-1029)value(553-563)copy(1021-1023)get(1025-1029)args(863-868)from_mapping(987-993)update(995-999)update(1170-1173)src/snakemake/rules.py (3)
name(138-139)name(142-143)Rule(84-1332)
🪛 Ruff (0.12.2)
tests/tests.py
1172-1172: Use line != "\n" instead of not line == "\n"
Replace with != operator
(SIM201)
src/snakemake/resources.py
4-4: dataclasses.dataclass imported but unused
Remove unused import: dataclasses.dataclass
(F401)
539-542: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block
Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value
(SIM108)
770-784: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
875-875: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
889-889: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/cli.py
229-229: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
236-236: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/common/__init__.py
106-112: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/jobs.py
38-38: snakemake.io.is_callable imported but unused
Remove unused import: snakemake.io.is_callable
(F401)
src/snakemake/rules.py
1130-1136: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/workflow.py
1808-1811: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
1820-1825: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🪛 GitHub Check: Check for spelling errors
src/snakemake/cli.py
[failure] 317-317:
intepretation ==> interpretation
🪛 GitHub Actions: Codespell
src/snakemake/cli.py
[error] 317-317: codespell: Spelling mistake found - 'intepretation' should be 'interpretation'.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (49)
- GitHub Check: tests (2, windows-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (10, ubuntu-latest, py313)
- GitHub Check: tests (10, windows-latest, py313)
- GitHub Check: tests (10, windows-latest, py312)
- GitHub Check: tests (8, windows-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (9, windows-latest, py313)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (8, ubuntu-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py313)
- GitHub Check: tests (9, windows-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (6, ubuntu-latest, py313)
- GitHub Check: tests (8, windows-latest, py312)
- GitHub Check: tests (7, windows-latest, py312)
- GitHub Check: tests (3, windows-latest, py313)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (7, windows-latest, py313)
- GitHub Check: tests (5, windows-latest, py313)
- GitHub Check: tests (6, windows-latest, py313)
- GitHub Check: tests (4, windows-latest, py313)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (6, windows-latest, py312)
- GitHub Check: tests (4, macos-latest, py313)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (5, windows-latest, py312)
- GitHub Check: tests (4, windows-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: tests (3, windows-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py313)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py313)
- GitHub Check: tests (5, ubuntu-latest, py313)
- GitHub Check: tests (4, macos-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (1, windows-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (2, macos-latest, py313)
- GitHub Check: tests (2, windows-latest, py312)
- GitHub Check: tests (1, macos-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py313)
- GitHub Check: tests (1, macos-latest, py312)
🔇 Additional comments (21)
tests/tests.py (12)
15-19: LGTM! Import updates align with the new resource system.The imports correctly reflect the migration from
DefaultResourcesto the newResourcesclass and add necessary exception types for enhanced resource validation.
675-677: Good test coverage for resource expression evaluation.This test validates the new capability to set threads via expressions, which is a key feature of the enhanced resource management system.
882-883: Improved assertion strictness for resource validation.The change to exact equality checking provides more rigorous validation, ensuring all resources match exactly rather than just checking a subset.
904-921: Comprehensive validation of resource unit conversions.The updated test properly validates the new resource system's ability to handle different unit representations (MB vs MiB) and ensures correct conversions between them.
923-948: Excellent test for mebibyte resource specification.This test validates the new capability to specify resources using mebibyte units, with correct unit conversion logic.
950-956: Valid smoke test for global resource constraints.Simple but effective test to ensure global resource constraints properly limit core allocation.
958-983: Great test for human-readable resource specifications.This validates the important new feature of supporting human-readable strings like "60 Gb" for global resource limits.
985-1005: Comprehensive validation of resource duplication prevention.Excellent parameterized test ensuring the system correctly rejects simultaneous specification of equivalent resources with different suffixes.
1007-1031: Good validation of suffixed resource constraints.These tests correctly ensure that resources with explicit unit suffixes (e.g., mem_mb) cannot use human-readable strings, preventing ambiguity in unit interpretation.
1334-1345: Good test for pipe group resource validation.This test properly validates that insufficient cores for pipe groups are detected, using the new
Resources.parsemethod correctly.
1519-1536: Excellent coverage of default resource specification formats.These tests validate important new capabilities:
- Human-readable default resources ('1 Gb', '0.001 TB')
- Mebibyte unit specifications with fractional values
Both are key features of the enhanced resource system.
1709-1709: Consistent migration to new resource system.All test updates correctly migrate from
DefaultResourcesto either plain lists orResources.parse()with appropriate parameters, maintaining test functionality while using the new resource system.Also applies to: 2596-2596
src/snakemake/resources.py (9)
219-222: Add exception chaining for better error context.When re-raising exceptions, preserve the original exception context for better debugging.
except ResourceError as err: raise ResourceInsufficiencyError( additive_resources, err.args[0] - ) from err + ) from errWait, the code already has
from err. Let me re-check... Actually, the code correctly includes exception chaining. This is good practice.
275-310: Well-implemented resource merging with proper default handling.Good fix for mutable default arguments by initializing them within the function. The type annotations and logic are solid.
432-459: Clean implementation of unit conversion decorators.These decorators properly handle unit conversion for evaluable resources with appropriate type checking and error messages.
588-612: Well-implemented resource constraint logic.The
constrainmethod properly handles type checking and edge cases, ensuring resources are appropriately limited by constraints.
613-677: Excellent unit conversion implementation.The conversion methods properly handle all resource types including evaluable resources and TBDStrings, with clear error messages for invalid operations.
787-796: Clean human-friendly parsing implementation.Properly handles parsing of human-readable size and time strings with appropriate unit conversions.
798-841: Robust resource container with proper normalization.The
Resourcesclass correctly enforces that suffixed and unsuffixed versions of size resources cannot coexist, preventing ambiguity in resource specifications.
986-1030: Well-designed utility methods for resource management.The utility methods provide clean interfaces for resource manipulation with proper type handling and value unwrapping capabilities.
1032-1174: Comprehensive resource scope management.The
ResourceScopesclass properly validates scope values, normalizes size resource suffixes, and provides clean interfaces for querying resources by scope. The special handling of suffixed resources in theexcludedproperty is particularly well thought out.
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/common.py (1)
305-308: Capture the subprocess failure before re-raising.With the new
shouldfailbranch we now re-raiseexception, but in theshellcmdpath we never assign the caughtsubprocess.CalledProcessErrortoexception. When a CLI invocation is expected to fail andshouldfailis set to an exception type, this code will attempt toraise exceptionwhereexception is None, triggering aTypeErrorinstead of validating the original failure. Please assign the caught exception so that the laterpytest.raisesblock can re-raise it.except subprocess.CalledProcessError as e: success = False + exception = e print(e.stdout.decode(), file=sys.stderr)
♻️ Duplicate comments (2)
src/snakemake/dag.py (1)
18-18: Fix duplicate typing import.
Dictis still listed twice, which keeps the lint/docs job failing (Ruff F811). Please drop the trailing, Dict.-from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Union, Dict +from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Unionsrc/snakemake/scheduling/job_scheduler.py (1)
103-111: Global resource expansion still drops human-readable limits.We’re still calling
expand_items(..., expand_sized=False)and discarding every string value, so a global--resources mem="1 GB"no longer yieldsmem_mb/mem_miband the scheduler ends up with no numeric constraint at all. This recreates the bug flagged earlier: human-friendly limits are silently ignored for DAG/job scheduling. Please expand sized resources and keep only the numeric (orNone) values so the scheduler enforces the correct totals.- self.global_resources = { - name: (sys.maxsize if res is None else res) - for name, res in workflow.global_resources.expand_items( - constraints={}, - evaluate=None, - expand_sized=False, - ) - if not isinstance(res, str) - } + self.global_resources = { + name: (sys.maxsize if res is None else res) + for name, res in workflow.global_resources.expand_items( + constraints={}, + evaluate=None, + expand_sized=True, + ) + if res is None or isinstance(res, int) + }
🧹 Nitpick comments (7)
tests/tests.py (2)
1168-1171: Drop leftover debug print.The print can make CI logs noisy. Safe to remove.
- lines = [line for line in f.readlines() if line != "\n"] - print(lines) + lines = [line for line in f.readlines() if line != "\n"]
1175-1197: Simplify comparison style.Prefer
line != "\n"overnot line == "\n"for clarity and consistency with nearby code.- lines = [line for line in f.readlines() if not line == "\n"] + lines = [line for line in f.readlines() if line != "\n"]src/snakemake/workflow.py (1)
1891-1906: Avoid exception chaining in rule resource mapping.Remove “from err” to follow Snakemake’s style and keep errors clean.
- except ResourceDuplicationError as err: - raise RuleException(err, rule=rule) from err + except ResourceDuplicationError as err: + raise RuleException(err, rule=rule) @@ - except ResourceConversionError as err: + except ResourceConversionError as err: msg = "Standard resource specified with invalid type, got error:\n" - raise RuleException(msg + str(err), rule=rule) from err + raise RuleException(msg + str(err), rule=rule) @@ - except ResourceValidationError as err: - raise RuleException(err, rule=rule) from err + except ResourceValidationError as err: + raise RuleException(err, rule=rule)Based on learnings
src/snakemake/resources.py (4)
34-34: Remove redundant import.
humanfriendlyis already imported on line 6 with specific functions. This bare import on line 34 is redundant.Apply this diff:
-import humanfriendly
282-285: Consider using f-string for consistency.The error message uses
.format()instead of an f-string, which is inconsistent with the rest of the file.Apply this diff:
if len(unique) > 1: raise WorkflowError( - "Resource {name} is a string but not all jobs in group require the " - "same value. Observed values: {values}.".format( - name=name, values=unique - ) + f"Resource {name} is a string but not all jobs in group require the " + f"same value. Observed values: {unique}." )
399-399: Unusual type annotation syntax.The type hint
zip[Tuple[int, ...]]is non-standard Python. Thezipbuilt-in doesn't accept type parameters in this way. This should beIterator[Tuple[int, ...]]or left without a type hint.Apply this diff:
- resource_groups: zip[Tuple[int, ...]] = zip(*resources.values()) + resource_groups: Iterator[Tuple[int, ...]] = zip(*resources.values())
768-816: Exception handling uses is_ordinary_string before definition.Line 798 calls
is_ordinary_string(val)but this function is defined later at line 1303. While Python allows this (the function isn't called until runtime whenis_ordinary_stringwill be defined), consider moving the helper function earlier in the file for better code organization.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (22)
src/snakemake/api.py(2 hunks)src/snakemake/cli.py(7 hunks)src/snakemake/common/__init__.py(5 hunks)src/snakemake/dag.py(2 hunks)src/snakemake/exceptions.py(3 hunks)src/snakemake/io/__init__.py(8 hunks)src/snakemake/jobs.py(5 hunks)src/snakemake/logging.py(2 hunks)src/snakemake/resources.py(13 hunks)src/snakemake/rules.py(7 hunks)src/snakemake/scheduling/job_scheduler.py(4 hunks)src/snakemake/script/__init__.py(1 hunks)src/snakemake/settings/types.py(2 hunks)src/snakemake/spawn_jobs.py(2 hunks)src/snakemake/workflow.py(7 hunks)test.txt(1 hunks)tests/common.py(7 hunks)tests/test_default_resources/Snakefile(1 hunks)tests/test_github_issue3271/Snakefile_should_fail(1 hunks)tests/test_group_jobs_resources/Snakefile(2 hunks)tests/test_ioutils/Snakefile(1 hunks)tests/tests.py(19 hunks)
🚧 Files skipped from review as they are similar to previous changes (5)
- src/snakemake/spawn_jobs.py
- test.txt
- src/snakemake/script/init.py
- tests/test_ioutils/Snakefile
- tests/test_group_jobs_resources/Snakefile
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit configuration file
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
src/snakemake/settings/types.pysrc/snakemake/scheduling/job_scheduler.pysrc/snakemake/common/__init__.pytests/common.pysrc/snakemake/workflow.pytests/tests.pysrc/snakemake/exceptions.pysrc/snakemake/jobs.pysrc/snakemake/api.pysrc/snakemake/cli.pysrc/snakemake/io/__init__.pysrc/snakemake/dag.pysrc/snakemake/rules.pysrc/snakemake/resources.pysrc/snakemake/logging.py
🧠 Learnings (22)
📚 Learning: 2025-10-30T15:39:38.224Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3813
File: tests/test_pathvars/Snakefile:22-23
Timestamp: 2025-10-30T15:39:38.224Z
Learning: In Snakemake, `<resources>` is a default pathvar that does not need to be explicitly defined in the pathvars section. It is available for use in rules and functions without requiring declaration.
Applied to files:
src/snakemake/settings/types.pytests/common.pysrc/snakemake/workflow.pytests/tests.pysrc/snakemake/jobs.pysrc/snakemake/api.pysrc/snakemake/cli.pysrc/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-10-04T16:12:18.927Z
Learnt from: lczech
Repo: snakemake/snakemake PR: 3113
File: snakemake/scheduler.py:912-914
Timestamp: 2024-10-04T16:12:18.927Z
Learning: In `snakemake/scheduler.py`, avoid suggesting the use of `asyncio.gather` in the `jobs_rewards` method due to overhead concerns and the need for immediate results.
Applied to files:
src/snakemake/scheduling/job_scheduler.pysrc/snakemake/jobs.py
📚 Learning: 2024-10-13T14:10:37.796Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the `snakemake/cli.py` file, `ExecutorPluginRegistry()` is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.
Applied to files:
src/snakemake/scheduling/job_scheduler.py
📚 Learning: 2024-10-06T14:09:26.494Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Applied to files:
src/snakemake/common/__init__.pysrc/snakemake/rules.py
📚 Learning: 2024-10-06T14:09:54.370Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.
Applied to files:
tests/common.pytests/tests.pytests/test_github_issue3271/Snakefile_should_failtests/test_default_resources/Snakefile
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.
Applied to files:
src/snakemake/workflow.pysrc/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-11-12T20:22:54.184Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.
Applied to files:
tests/tests.pysrc/snakemake/rules.py
📚 Learning: 2024-10-13T14:29:38.902Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.
Applied to files:
tests/tests.py
📚 Learning: 2025-01-17T12:00:09.368Z
Learnt from: leoschwarz
Repo: snakemake/snakemake PR: 3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
Applied to files:
tests/test_github_issue3271/Snakefile_should_failtests/test_default_resources/Snakefile
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2985
File: tests/tests.py:2051-2051
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.
Applied to files:
src/snakemake/io/__init__.pysrc/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.
Applied to files:
src/snakemake/io/__init__.pysrc/snakemake/rules.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.
Applied to files:
src/snakemake/io/__init__.pysrc/snakemake/rules.py
📚 Learning: 2024-10-14T09:42:11.571Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.
Applied to files:
src/snakemake/dag.py
📚 Learning: 2024-11-12T12:08:20.342Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.
Applied to files:
src/snakemake/dag.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2024-11-07T00:32:44.137Z
Learnt from: mbhall88
Repo: snakemake/snakemake PR: 3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2025-08-28T13:15:55.232Z
Learnt from: Hocnonsense
Repo: snakemake/snakemake PR: 3714
File: src/snakemake/modules.py:195-195
Timestamp: 2025-08-28T13:15:55.232Z
Learning: In src/snakemake/modules.py, the line `self.globals["__name__"] = namespace` is intentionally designed to set __name__ to None when namespace is None (i.e., when not inside a module context, for top-level workflows). This is correct behavior and should not be guarded.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2025-08-07T21:51:08.043Z
Learnt from: pvandyken
Repo: snakemake/snakemake PR: 3421
File: src/snakemake/resources.py:606-607
Timestamp: 2025-08-07T21:51:08.043Z
Learning: In Snakemake's Resource.constrain method, the type comparison uses isinstance(self_val, type(other_val)) rather than direct type equality to allow strings to be compared with AnnotatedString objects. This polymorphic behavior is intentional and necessary for proper resource constraint functionality.
Applied to files:
src/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.
Applied to files:
src/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1250-1251
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid commenting on trailing commas in code reviews for the Snakemake repository as per the user's preference.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2024-09-25T10:55:53.963Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.
Applied to files:
src/snakemake/resources.py
📚 Learning: 2024-12-14T13:11:32.988Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.
Applied to files:
src/snakemake/resources.py
🔇 Additional comments (27)
tests/test_github_issue3271/Snakefile_should_fail (1)
1-13: Test structure looks correct.The resource lambda correctly uses
input.size_mb(which returns a numeric value, not a string), and the params lambda properly accessesresources.mem_mb. The division by 1000 on line 9 is appropriate for decimal unit conversion (MB → GB in decimal units). The test file structure appears sound for its intended purpose of validating error handling in issue #3271.tests/tests.py (5)
15-19: Imports align with new Resources API.Looks good; correct modules used for Resource* exceptions and Resources utilities.
688-690: New overwrite-threads test is appropriate.Covers expression handling for --set-threads; no issues.
906-908: Assert helper tightened.Strict dict equality here is fine for the expected resource dicts.
946-964: Resource normalization and validation tests look solid.
- Human-readable/global vs local and duplication checks are well covered.
- Error types match the new exceptions.
Also applies to: 965-990, 1001-1025, 1028-1046, 1050-1055
1067-1070: Fix mem_mib expected value (off-by-one).60000 MB converts to ceil(60000×0.95367431640625) = 57221 MiB, not 57222. Update assertion.
- mem_mib=57222, + mem_mib=57221,Also applies to: 1075-1084
src/snakemake/rules.py (3)
30-31: API/type updates look correct.Imports, TYPE_CHECKING, and attributes align with the new Resources model.
Also applies to: 52-58, 81-83, 97-97, 110-110
1132-1164: Resource expansion and constraint flow LGTM.
- _cores evaluation constrained by max_threads and global _cores.
- skip={"_cores"} avoids re-evaluating threads.
- Error messaging for constraint/conversion is clear.
1085-1164: Tighten exception handling in expand_resources.
- Avoid exception chaining (“from err/e”) to match Snakemake style.
- Catch Exception instead of BaseException to avoid swallowing KeyboardInterrupt/SystemExit.
Apply this diff:
- except ResourceValidationError as err: - raise WorkflowError(err, rule=self) from err + except ResourceValidationError as err: + raise WorkflowError(err, rule=self) @@ - except BaseException as e: - raise InputFunctionException(e, rule=self, wildcards=wildcards) from e + except Exception as e: + raise InputFunctionException(e, rule=self, wildcards=wildcards)To check consistency repo-wide:
Based on learnings
src/snakemake/workflow.py (3)
132-133: Resources integration looks consistent.
- Using ResourceScopes/Resources and .value access on _cores/_nodes aligns with the new model.
- Directly reusing resource_settings.resources for global_resources matches PR goals.
Also applies to: 182-185, 631-631, 635-635
1871-1876: Default resources and threads initialization LGTM.Copies defaults when provided, ensures _nodes=1, and sets _cores with overwrite precedence.
Also applies to: 1878-1885
2009-2011: Handover behavior confirmed.Updating rule.resources from global_resources when unspecified matches the documented intent.
src/snakemake/resources.py (15)
169-170: Verify SizedResources filtering doesn't hide TBDString issues.The condition filters out both
TBDStringinstances andSizedResources, but the order matters. If a resource value is aTBDStringand also has a name inSizedResources, both conditions should be checked. The current logic is correct but could be clearer.
447-459: Helper types and constants look good.The type aliases, constants, and helper function are well-defined and appropriately typed.
493-517: Resource initialization handles validation and conversion well.The
__init__method properly validates input types, converts floats to integers, parses human-friendly values, and sets up evaluators. The logic is sound.
571-602: Constrain method logic is correct.Based on retrieved learnings, the
isinstance(self_val, type(other_val))check on line 592 is intentionally designed to allow polymorphic behavior between strings andAnnotatedStringobjects. The implementation correctly handles None values, TBDString, and type mismatches.Based on learnings
838-847: Human-friendly parsing ensures minimum value of 1.The use of
max(..., 1)on lines 840 and 845 ensures that parsed sizes and timespans are at least 1, preventing zero or negative values from human-readable inputs. This is a good defensive practice.
875-882: Default resource expressions are well-designed.The DEFAULTS dictionary provides sensible defaults:
memscales with input size (2x) with min 1GB and max 8GBdiskscales with input size (2x) with min 1GB, or 50GB if no inputtmpdiruses system temp directoryThese expressions will be evaluated in the context of each rule.
892-903: Size normalization prevents resource duplication.The
_normalize_sizesmethod correctly detects when multiple variants of a size resource (e.g.,mem,mem_mb,mem_mib) are provided and raisesResourceDuplicationError. The standardization ensures only one canonical form is stored.
1006-1015: Good validation of reserved resource name.Explicitly checking for and rejecting
_coresprevents conflicts with Snakemake's internal resource management. The error message is clear and actionable.
1025-1037: Integer validation correctly allows zero.Based on the past review comment discussion, the check
val < 0is correct because zero is a valid value for some contexts (e.g.,--set-resources). The validation only rejects negative integers.Based on learnings
1108-1156: expand_items provides comprehensive resource expansion.This method handles:
- Evaluation of callable resources
- Constraint application
- Expansion of sized resources into both human-friendly and mebibyte forms
The logic correctly yields tuples and handles the skip set and expand_sized flag.
1169-1189: ResourceScopes initialization validates scope values.The
__init__method:
- Normalizes size resource variants (mem/mem_mb/mem_mib)
- Validates that all scopes are either 'local', 'global', or 'excluded'
- Raises
ResourceScopesExceptionwith clear error message if invalidThe implementation is correct.
1202-1204: Sensible default scopes.The
defaults()classmethod sets:
mem_mbanddisk_mbas local (per-job pools)runtimeas excluded (not submitted to cluster)These defaults align with typical Snakemake resource management patterns.
1287-1295: Excluded property correctly expands sized resources.The
excludedproperty adds suffixed versions (_mb,_mib) of any sized resources that have a defined scope. This ensures that when a base resource (e.g.,mem) is excluded, all its variants are also excluded.
1303-1336: is_ordinary_string helper has comprehensive validation.The function uses a well-designed regex pattern to detect:
- Function calls:
func(...)- Dictionary literals:
{...}- Lambda expressions:
lambda x: ...- Math operators:
+,-,*,/,%- Attribute access:
.attributeThis helps distinguish ordinary strings from Python expressions that should be evaluated. Consider adding test coverage for edge cases like escaped quotes or nested structures.
212-213: Potential KeyError when constraint not found.Line 212 calls
constraints.get(name).valuewhich will raise anAttributeErrorifnameis not inconstraints(since.get()returns aResource("", None)per line 1105, and calling.valueon that should work). However, this assumes theResources.get()method returns a defaultResourceobject.Run this script to verify the
Resources.get()behavior:
Yes! |
80e19ad to
e017870
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (3)
src/snakemake/exceptions.py (1)
658-665: Fix the ValueError message to avoid tuple representation.This issue was previously flagged. Passing
duplicate_resourcesas a second positional argument tosuper().__init__()causesstr(ResourceDuplicationError)to display as a tuple('message', [...]). Remove it from the super call and keep only the message string.Apply this diff:
class ResourceDuplicationError(ValueError): def __init__(self, duplicate_resources: Sequence[str]): self.duplicate_resources = duplicate_resources super().__init__( "The following resources are equivalent and cannot be simultaneously " - "provided to a rule:", - duplicate_resources, + "provided to a rule: " + ", ".join(duplicate_resources) )src/snakemake/dag.py (1)
18-18: DuplicateDictin typing import (ruff F811).Remove the second
Dictto prevent redefinition warnings/errors.-from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Union, Dict +from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Unionsrc/snakemake/cli.py (1)
168-193: Past review concerns remain unresolved.Three issues previously flagged in past reviews are still present:
- Grammar error (Line 171): Missing comma in error message—"a positive integer a quoted string" should be "a positive integer, a quoted string".
- Missing validation (Line 183): No check that ":" exists in entry before
split(":", maxsplit=1), which could raiseValueErrorwith an unclear message if malformed.- Incomplete exception handling (Line 192): Only catches
ValueError, butResources.parsecan also raiseWorkflowErrorper the code insrc/snakemake/resources.py. Should catch both and chain them.Apply this consolidated fix:
errmsg = ( "Invalid resource definition: entries have to be defined as " - "RULE:RESOURCE=VALUE, with VALUE being a positive integer a quoted string, or " + "RULE:RESOURCE=VALUE, with VALUE being a positive integer, a quoted string, or " "a Python expression (e.g. min(max(2*input.size_mb, 1000), 8000))." ) if args is None: return {} assignments: Dict[str, List[str]] = defaultdict(list) for entry in args: + if ":" not in entry: + raise ValueError(f"{errmsg} (Unparsable value: {repr(entry)})") rule, assign = entry.split(":", maxsplit=1) assignments[rule].append(assign) try: return { rule: Resources.parse(assigns, allow_expressions=True) for rule, assigns in assignments.items() } - except ValueError as err: + except (ValueError, WorkflowError) as err: raise ValueError(errmsg) from errAlso add the missing import:
from snakemake.exceptions import ( CliException, ResourceScopesException, + WorkflowError, print_exception, )
🧹 Nitpick comments (5)
src/snakemake/resources.py (2)
835-851: Remove redundant int() casts.The
parse_size()andparse_timespan()functions already returnintvalues, making theint()casts on lines 843 and 848 redundant.Apply this diff:
if name in SizedResources: try: - return max(int(math.ceil(parse_size(stripped) / 1e6)), 1) + return max(math.ceil(parse_size(stripped) / 1e6), 1) except InvalidSize as err: raise WorkflowError(err_msg.format(unit="size in MB")) from err elif name in TimeResources: try: - return max(int(round(parse_timespan(stripped) / 60)), 1) + return max(round(parse_timespan(stripped) / 60), 1) except InvalidTimespan as err: raise WorkflowError(err_msg.format(unit="minutes")) from err
878-885: Consider annotating DEFAULTS with ClassVar.Static analysis suggests that mutable class attributes should be annotated with
typing.ClassVarto indicate they are class-level rather than instance-level.+from typing import ClassVar + class Resources(Mapping[str, Resource]): - DEFAULTS = { + DEFAULTS: ClassVar[dict[str, dict[str, str]]] = { "full": { "mem": "min(max(2*input.size_mb, 1000), 8000)", "disk": "max(2*input.size_mb, 1000) if input else 50000", "tmpdir": "system_tmpdir", }, "bare": {"tmpdir": "system_tmpdir"}, }src/snakemake/common/__init__.py (2)
107-113: Use exception chaining for context (raise ... from e).Preserve the original cause instead of passing
eas an argument.- raise NestedCoroutineError( - "Error running coroutine in event loop. Snakemake currently does not " - "support being executed from an already running event loop. " - "If you run Snakemake e.g. from a Jupyter notebook, make sure to spawn a " - "separate process for Snakemake.", - e, - ) + raise NestedCoroutineError( + "Error running coroutine in event loop. Snakemake currently does not " + "support being executed from an already running event loop. " + "If you run Snakemake e.g. from a Jupyter notebook, make sure to spawn a " + "separate process for Snakemake." + ) from e
65-71: Nit: remove redundantint()aroundmath.ceil/math.floor.Both return
intin modern Python.-def mb_to_mib(mb: int): - return int(math.ceil(mb * 0.95367431640625)) +def mb_to_mib(mb: int): + return math.ceil(mb * 0.95367431640625) -def mib_to_mb(mib: int): - return int(math.floor(mib / 0.95367431640625)) +def mib_to_mb(mib: int): + return math.floor(mib / 0.95367431640625)src/snakemake/scheduling/job_scheduler.py (1)
103-111: Add a short comment clarifying why sized/human-friendly resources are filtered.Using
expand_sized=Falseand filteringstrexcludes human-friendly sized resources (e.g., mem/disk) from global limits, aligning with ResourceScopes (mem/disk default to local). A brief comment will prevent future “fixes” that reintroduce them into global constraints.self.global_resources = { name: (sys.maxsize if res is None else res) for name, res in workflow.global_resources.expand_items( constraints={}, evaluate=None, expand_sized=False, ) if not isinstance(res, str) } + # Note: sized resources (mem/disk) are intentionally excluded here: + # they normalize to human‑readable strings and default to local scope, + # hence they must not constrain global scheduling.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (22)
src/snakemake/api.py(2 hunks)src/snakemake/cli.py(7 hunks)src/snakemake/common/__init__.py(5 hunks)src/snakemake/dag.py(2 hunks)src/snakemake/exceptions.py(3 hunks)src/snakemake/io/__init__.py(8 hunks)src/snakemake/jobs.py(5 hunks)src/snakemake/logging.py(2 hunks)src/snakemake/resources.py(13 hunks)src/snakemake/rules.py(7 hunks)src/snakemake/scheduling/job_scheduler.py(4 hunks)src/snakemake/script/__init__.py(1 hunks)src/snakemake/settings/types.py(2 hunks)src/snakemake/spawn_jobs.py(2 hunks)src/snakemake/workflow.py(7 hunks)test.txt(1 hunks)tests/common.py(7 hunks)tests/test_default_resources/Snakefile(1 hunks)tests/test_github_issue3271/Snakefile_should_fail(1 hunks)tests/test_group_jobs_resources/Snakefile(2 hunks)tests/test_ioutils/Snakefile(1 hunks)tests/tests.py(19 hunks)
🚧 Files skipped from review as they are similar to previous changes (5)
- tests/test_ioutils/Snakefile
- src/snakemake/settings/types.py
- test.txt
- src/snakemake/logging.py
- src/snakemake/api.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit configuration file
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
src/snakemake/script/__init__.pysrc/snakemake/rules.pysrc/snakemake/spawn_jobs.pysrc/snakemake/exceptions.pysrc/snakemake/scheduling/job_scheduler.pytests/common.pysrc/snakemake/dag.pytests/tests.pysrc/snakemake/cli.pysrc/snakemake/workflow.pysrc/snakemake/jobs.pysrc/snakemake/resources.pysrc/snakemake/common/__init__.pysrc/snakemake/io/__init__.py
🧠 Learnings (23)
📚 Learning: 2025-10-30T15:39:38.224Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3813
File: tests/test_pathvars/Snakefile:22-23
Timestamp: 2025-10-30T15:39:38.224Z
Learning: In Snakemake, `<resources>` is a default pathvar that does not need to be explicitly defined in the pathvars section. It is available for use in rules and functions without requiring declaration.
Applied to files:
src/snakemake/script/__init__.pysrc/snakemake/rules.pytests/common.pytests/tests.pysrc/snakemake/cli.pysrc/snakemake/workflow.pysrc/snakemake/jobs.pysrc/snakemake/resources.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.
Applied to files:
src/snakemake/rules.pysrc/snakemake/workflow.pysrc/snakemake/resources.py
📚 Learning: 2024-10-06T14:09:26.494Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Applied to files:
src/snakemake/rules.pysrc/snakemake/common/__init__.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2024-11-07T00:32:44.137Z
Learnt from: mbhall88
Repo: snakemake/snakemake PR: 3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.
Applied to files:
src/snakemake/rules.pytests/test_default_resources/Snakefile
📚 Learning: 2024-08-13T16:22:09.641Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-08-13T16:22:09.641Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.
Applied to files:
src/snakemake/rules.pysrc/snakemake/io/__init__.py
📚 Learning: 2025-08-28T13:15:55.232Z
Learnt from: Hocnonsense
Repo: snakemake/snakemake PR: 3714
File: src/snakemake/modules.py:195-195
Timestamp: 2025-08-28T13:15:55.232Z
Learning: In src/snakemake/modules.py, the line `self.globals["__name__"] = namespace` is intentionally designed to set __name__ to None when namespace is None (i.e., when not inside a module context, for top-level workflows). This is correct behavior and should not be guarded.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.
Applied to files:
src/snakemake/rules.pysrc/snakemake/io/__init__.py
📚 Learning: 2024-11-12T20:22:54.184Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.
Applied to files:
src/snakemake/rules.pytests/tests.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2985
File: tests/tests.py:2051-2051
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.
Applied to files:
src/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2025-08-07T21:51:08.043Z
Learnt from: pvandyken
Repo: snakemake/snakemake PR: 3421
File: src/snakemake/resources.py:606-607
Timestamp: 2025-08-07T21:51:08.043Z
Learning: In Snakemake's Resource.constrain method, the type comparison uses isinstance(self_val, type(other_val)) rather than direct type equality to allow strings to be compared with AnnotatedString objects. This polymorphic behavior is intentional and necessary for proper resource constraint functionality.
Applied to files:
src/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.
Applied to files:
src/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1250-1251
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid commenting on trailing commas in code reviews for the Snakemake repository as per the user's preference.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2024-10-04T16:12:18.927Z
Learnt from: lczech
Repo: snakemake/snakemake PR: 3113
File: snakemake/scheduler.py:912-914
Timestamp: 2024-10-04T16:12:18.927Z
Learning: In `snakemake/scheduler.py`, avoid suggesting the use of `asyncio.gather` in the `jobs_rewards` method due to overhead concerns and the need for immediate results.
Applied to files:
src/snakemake/scheduling/job_scheduler.pysrc/snakemake/jobs.py
📚 Learning: 2025-07-29T14:53:04.598Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3676
File: src/snakemake/cli.py:0-0
Timestamp: 2025-07-29T14:53:04.598Z
Learning: In the Snakemake repository, deprecated scheduler interfaces (like scheduler_ilp_solver, --scheduler-solver-path, --scheduler-greediness) should be kept for backward compatibility during the migration to the plugin-based scheduler system, rather than being removed immediately.
Applied to files:
src/snakemake/scheduling/job_scheduler.py
📚 Learning: 2024-10-13T14:10:37.796Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the `snakemake/cli.py` file, `ExecutorPluginRegistry()` is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.
Applied to files:
src/snakemake/scheduling/job_scheduler.py
📚 Learning: 2024-10-06T14:09:54.370Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.
Applied to files:
tests/test_group_jobs_resources/Snakefiletests/test_github_issue3271/Snakefile_should_failtests/tests.pytests/test_default_resources/Snakefile
📚 Learning: 2024-10-14T09:42:11.571Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.
Applied to files:
src/snakemake/dag.py
📚 Learning: 2024-11-12T12:08:20.342Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.
Applied to files:
src/snakemake/dag.py
📚 Learning: 2024-10-13T14:29:38.902Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.
Applied to files:
tests/tests.py
📚 Learning: 2025-01-17T12:00:09.368Z
Learnt from: leoschwarz
Repo: snakemake/snakemake PR: 3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
Applied to files:
tests/test_default_resources/Snakefile
📚 Learning: 2024-09-25T10:55:53.963Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.
Applied to files:
src/snakemake/resources.py
📚 Learning: 2024-12-14T13:11:32.988Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.
Applied to files:
src/snakemake/resources.py
🧬 Code graph analysis (13)
src/snakemake/script/__init__.py (3)
src/snakemake/jobs.py (2)
resources(481-498)resources(1519-1533)src/snakemake/workflow.py (1)
resources(2230-2235)src/snakemake/io/__init__.py (1)
ResourceList(1995-1996)
src/snakemake/rules.py (4)
src/snakemake/io/__init__.py (2)
ResourceList(1995-1996)get(1915-1920)src/snakemake/jobs.py (13)
resources(481-498)resources(1519-1533)wildcards(329-330)wildcards(333-334)input(337-338)input(341-342)input(1540-1545)attempt(471-472)attempt(475-478)attempt(1732-1733)attempt(1736-1741)name(1292-1293)name(1670-1674)src/snakemake/resources.py (7)
Resource(461-851)Resources(854-1158)evaluate(552-569)constrain(571-602)get(1105-1109)value(536-546)expand_items(1111-1158)src/snakemake/exceptions.py (5)
ResourceConstraintError(637-638)ResourceValidationError(633-634)NestedCoroutineError(551-552)ResourceConversionError(641-655)InputFunctionException(250-262)
src/snakemake/exceptions.py (1)
src/snakemake/resources.py (1)
value(536-546)
src/snakemake/scheduling/job_scheduler.py (2)
src/snakemake/resources.py (3)
expand_items(1111-1158)evaluate(552-569)value(536-546)src/snakemake/workflow.py (2)
name(2314-2319)resources(2230-2235)
tests/common.py (1)
src/snakemake/resources.py (5)
ResourceScopes(1164-1302)Resources(854-1158)from_mapping(1060-1080)parse(982-1057)defaults(1205-1206)
src/snakemake/dag.py (1)
src/snakemake/scheduling/job_scheduler.py (1)
stats(187-188)
tests/tests.py (3)
src/snakemake/exceptions.py (2)
ResourceConversionError(641-655)ResourceDuplicationError(658-665)src/snakemake/resources.py (6)
GroupResources(62-444)is_ordinary_string(1305-1338)Resources(854-1158)from_mapping(1060-1080)parse(982-1057)defaults(1205-1206)tests/common.py (1)
run(155-521)
src/snakemake/cli.py (3)
src/snakemake/resources.py (10)
Resource(461-851)Resources(854-1158)from_cli_expression(724-769)args(929-934)parse(982-1057)items(1094-1095)value(536-546)default(944-962)parser_factory(965-979)defaults(1205-1206)src/snakemake/workflow.py (1)
rule(1806-2046)src/snakemake/common/__init__.py (1)
parse_key_value_arg(73-80)
src/snakemake/workflow.py (2)
src/snakemake/exceptions.py (3)
ResourceDuplicationError(658-665)ResourceConversionError(641-655)ResourceValidationError(633-634)src/snakemake/resources.py (8)
Resources(854-1158)value(536-546)copy(1101-1103)get(1105-1109)args(929-934)from_mapping(1060-1080)update(1082-1086)update(1299-1302)
src/snakemake/jobs.py (4)
src/snakemake/io/__init__.py (2)
ResourceList(1995-1996)items(1870-1872)src/snakemake/workflow.py (2)
resources(2230-2235)resource_scopes(535-536)src/snakemake/resources.py (4)
GroupResources(62-444)Resources(854-1158)items(1094-1095)args(929-934)src/snakemake/exceptions.py (1)
ResourceInsufficiencyError(604-623)
src/snakemake/resources.py (6)
src/snakemake/common/__init__.py (3)
get_input_function_aux_params(311-323)mb_to_mib(65-66)mib_to_mb(69-70)src/snakemake/exceptions.py (11)
NestedCoroutineError(551-552)ResourceConstraintError(637-638)ResourceDuplicationError(658-665)ResourceError(600-601)ResourceInsufficiencyError(604-623)ResourceScopesException(626-630)ResourceConversionError(641-655)ResourceValidationError(633-634)is_file_not_found_error(693-700)format(651-655)format_evaluated(643-648)src/snakemake/common/tbdstring.py (1)
TBDString(3-96)src/snakemake/io/__init__.py (11)
AnnotatedString(957-977)items(1870-1872)get(1915-1920)size(176-177)size(651-661)size(1963-1964)is_callable(324-325)is_callable(968-969)is_callable(1186-1191)keys(1903-1904)update(1845-1851)src/snakemake/jobs.py (22)
jobs(1377-1378)jobs(1381-1382)Job(183-1320)resources(481-498)resources(1519-1533)name(1292-1293)name(1670-1674)group(461-464)group(467-468)wildcards(329-330)wildcards(333-334)input(337-338)input(341-342)input(1540-1545)attempt(471-472)attempt(475-478)attempt(1732-1733)attempt(1736-1741)threads(423-424)threads(1725-1726)is_local(1020-1028)is_local(1681-1682)src/snakemake/rules.py (8)
name(152-153)name(156-157)group(182-191)group(194-195)evaluate(1093-1120)input(291-292)input(1403-1425)add(1356-1360)
src/snakemake/common/__init__.py (1)
src/snakemake/exceptions.py (1)
NestedCoroutineError(551-552)
src/snakemake/io/__init__.py (2)
src/snakemake/gui.py (1)
index(84-95)src/snakemake/resources.py (1)
items(1094-1095)
🪛 Ruff (0.14.3)
src/snakemake/rules.py
1106-1106: Unused lambda argument: e
(ARG005)
1133-1133: Avoid specifying long messages outside the exception class
(TRY003)
1146-1150: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
1146-1150: Avoid specifying long messages outside the exception class
(TRY003)
1159-1159: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/dag.py
18-18: Redefinition of unused Dict from line 18
Remove definition: Dict
(F811)
src/snakemake/cli.py
232-232: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
244-244: Avoid specifying long messages outside the exception class
(TRY003)
src/snakemake/workflow.py
1886-1889: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
1886-1889: Avoid specifying long messages outside the exception class
(TRY003)
1894-1894: Avoid specifying long messages outside the exception class
(TRY003)
src/snakemake/jobs.py
1530-1532: Avoid specifying long messages outside the exception class
(TRY003)
src/snakemake/resources.py
386-386: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
399-399: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
542-545: Avoid specifying long messages outside the exception class
(TRY003)
569-569: Use raise without specifying exception name
Remove exception name
(TRY201)
648-650: Avoid specifying long messages outside the exception class
(TRY003)
742-742: Unused function argument: wildcards
(ARG001)
746-746: Unused function argument: rulename
(ARG001)
757-757: Unused function argument: wildcards
(ARG001)
757-757: Unused function argument: rulename
(ARG001)
773-773: Unused static method argument: name
(ARG004)
780-789: Use of possibly insecure function; consider using ast.literal_eval
(S307)
795-795: Do not catch blind exception: Exception
(BLE001)
804-818: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
804-818: Avoid specifying long messages outside the exception class
(TRY003)
843-843: Value being cast to int is already an integer
Remove unnecessary int call
(RUF046)
848-848: Value being cast to int is already an integer
Remove unnecessary int call
(RUF046)
878-885: Mutable class attributes should be annotated with typing.ClassVar
(RUF012)
919-919: Avoid specifying long messages outside the exception class
(TRY003)
941-941: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
941-941: Avoid specifying long messages outside the exception class
(TRY003)
955-955: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
955-955: Avoid specifying long messages outside the exception class
(TRY003)
1010-1013: Avoid specifying long messages outside the exception class
(TRY003)
1048-1050: Avoid specifying long messages outside the exception class
(TRY003)
1187-1191: Avoid specifying long messages outside the exception class
(TRY003)
src/snakemake/common/__init__.py
66-66: Value being cast to int is already an integer
Remove unnecessary int call
(RUF046)
70-70: Value being cast to int is already an integer
Remove unnecessary int call
(RUF046)
107-113: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
107-113: Avoid specifying long messages outside the exception class
(TRY003)
src/snakemake/io/__init__.py
1-1: The file is executable but no shebang is present
(EXE002)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (46)
- GitHub Check: tests (10, ubuntu-latest, py313)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (10, windows-2022, py313)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (9, windows-2022, py313)
- GitHub Check: tests (8, windows-2022, py313)
- GitHub Check: tests (9, ubuntu-latest, py313)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (7, windows-2022, py313)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (7, ubuntu-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (6, windows-2022, py313)
- GitHub Check: tests (8, ubuntu-latest, py313)
- GitHub Check: tests (6, macos-latest, py313)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (6, ubuntu-latest, py313)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py313)
- GitHub Check: tests (5, ubuntu-latest, py313)
- GitHub Check: tests (5, windows-2022, py313)
- GitHub Check: tests (4, windows-2022, py313)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (4, macos-latest, py313)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py313)
- GitHub Check: tests (3, windows-2022, py313)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (3, macos-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py313)
- GitHub Check: tests (1, windows-2022, py313)
- GitHub Check: tests (2, macos-latest, py313)
- GitHub Check: tests (2, windows-2022, py313)
- GitHub Check: tests (1, macos-latest, py313)
- GitHub Check: tests (1, ubuntu-latest, py313)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py311)
- GitHub Check: apidocs
🔇 Additional comments (33)
src/snakemake/resources.py (4)
64-269: LGTM! Well-typed refactor of resource grouping logic.The addition of comprehensive type annotations and the migration to the typed
ResourcesAPI improves type safety. The error handling withResourceInsufficiencyErroris appropriate.
461-716: LGTM! Well-designed Resource class with proper validation.The
Resourceclass provides a clean abstraction for resource values with appropriate validation, evaluation, and conversion methods. The design handles evaluable resources, human-friendly parsing, and unit conversions elegantly.
854-1159: LGTM! Robust Mapping-based Resources container.The
Resourcesclass provides a clean, type-safe container for resource management with proper normalization of sized resources and good support for default resources and parsing.
1164-1303: LGTM! Clean ResourceScopes implementation.The
ResourceScopesclass properly handles resource scope management with appropriate normalization of size variants and clear testing methods.tests/test_group_jobs_resources/Snakefile (2)
18-18: LGTM! Tests human-friendly resource format.The change from
mem_mb=20000tomem="20 GB"appropriately tests the new human-friendly resource parsing feature.
49-49: LGTM! Tests mebibyte resource handling.The addition of
mem_mib=10000tests the new mebibyte resource suffix functionality.src/snakemake/spawn_jobs.py (1)
97-118: LGTM! Simplified resource argument formatting.The direct use of
value.rawaligns with the newResourceAPI and removes unnecessary indirection from the previousget_orig_arghelper. The change maintains the same behavior while being more straightforward.src/snakemake/script/__init__.py (1)
113-113: LGTM! Type annotation updated for compatibility.The change from
io_.Resourcestoio_.ResourceListaligns with the PR's renaming strategy to preserve plugin compatibility while introducing the new typedResourcesAPI.tests/test_github_issue3271/Snakefile_should_fail (1)
7-7: LGTM! Tests input.size_mb in resource function.The change from
attempt * 1000toinput.size_mbappropriately tests the evaluation of resources that depend on input file sizes, which is a key feature of the new resource system.src/snakemake/dag.py (1)
3189-3214: LGTM:stats()implementation and return typing are consistent.Counts and message formatting look correct; sorting by
rule.nameis clear.tests/test_default_resources/Snakefile (1)
9-14: LGTM: exercising mem/disk normalization across mem, mem_mb, mem_mib.The guards correctly validate human-friendly and sized variants before touching outputs.
Also applies to: 21-26
tests/common.py (1)
342-356: LGTM: resources/default_resources normalization toResourcesinstances.
Resources.from_mapping(...)andResources.parse(..., defaults="full", allow_expressions=True)match the new API and keep tests aligned with CLI semantics.src/snakemake/io/__init__.py (1)
1995-1996: Rename verified—no stale imports of old Resources from snakemake.io detected.All usages of Resources correctly import from
snakemake.resources: cli.py, workflow.py, and tests/common.py. The old Resources class has been removed from snakemake.io, replaced by the ResourceList placeholder shown in the code snippet. Refactoring is complete with no import issues.src/snakemake/rules.py (1)
1084-1163: LGTM! Well-structured resource evaluation with proper error handling.The refactored
expand_resourcesmethod correctly:
- Evaluates resources through the new
ResourceAPI- Applies thread constraints from
max_threadsand global_cores- Uses
Resources.expand_itemsfor standardized expansion- Handles
NestedCoroutineErrorfor async evaluation conflicts- Catches and wraps
ResourceConversionErrorwith helpful context- Returns
ResourceListas documented in the AI summaryThe error handling appropriately catches specific exceptions without chaining (per codebase conventions).
src/snakemake/jobs.py (2)
108-116: LGTM! Proper filtering of TBDString resources and scope-aware scheduling.The updated
_get_scheduler_resourcescorrectly:
- Filters out
TBDStringresources (unevaluated resources)- Applies scope filtering for non-local execution via
resource_scopes.is_local()- Maintains consistent resource dictionary construction
1529-1533: LGTM! Proper exception handling for group job resources.The exception handling correctly:
- Catches both
WorkflowErrorandResourceInsufficiencyError- Wraps with contextual group ID information
- Uses
from errfor proper exception chaining (note: the comment about avoiding chaining applies to hiding business logic; here we're preserving the error context appropriately)tests/tests.py (7)
688-690: LGTM! Test verifies expression evaluation in thread overwriting.The test correctly exercises the
--set-threadsCLI option with an expression ('10*2'), validating that thread resource expressions are properly evaluated.
906-907: LGTM! Simplified assertion aligns with new resource API.The simplified
assert_resourcesdirectly compares resource dictionaries, which is appropriate now that resources are returned asResourceList(a dictionary-like structure).
938-963: LGTM! Comprehensive test of human-readable resource handling in groups.The test correctly:
- Exercises group job resource aggregation via spy on
GroupResources.basic_layered- Verifies that
mem_mb=60000is properly expanded to includemem(human-readable) andmem_mibvariants- Validates all resource values including the correctly computed
mem_mib=57221- Uses
default_resources=["mem_mb=0"]to set baseline defaults
966-989: LGTM! Validates resources can be provided in mebibytes.This test correctly verifies that users can specify resources using
mem_mibinstead ofmem_mb, and that the system properly converts and expands all variants (mem,mem_mb,mem_mib).
1028-1047: LGTM! Validates duplicate resource detection.The parametrized test correctly verifies that providing multiple variants of the same sized resource (e.g.,
memandmem_mbtogether) raisesResourceDuplicationError, preventing ambiguous resource specifications.
1050-1057: LGTM! Validates human-readable resources cannot use suffixed names.The test correctly verifies that suffixed resources like
mem_mbmust be integers and cannot accept human-readable strings like"40 Gb", raisingResourceConversionErroras expected.
1542-1567: LGTM! Comprehensive tests for human-readable default resources.The tests correctly validate:
test_default_resources_humanreadable: Human-readable defaults (e.g.,mem='1 Gb') work correctlytest_default_resources_humanreadable_disallowed_for_suffixed: Suffixed resources (mem_mb,disk_mib) cannot use human-readable strings (should fail)test_default_resources_mebibytes: Mebibyte decimal values are handled correctlyThese align with the PR's goal of standardizing human-readable resource handling.
src/snakemake/cli.py (4)
110-123: LGTM: Threads wrapper correctly handles integers and expressions.The wrapper properly distinguishes between integer values (creating
Resource("_cores", orig_value)) and expression strings (usingResource.from_cli_expression), addressing the concern from previous reviews.
223-245: LGTM: Integer parsing and wrapper application is correct.The function properly:
- Attempts integer conversion and validates non-negativity
- Falls back to wrapper for non-integer values when wrapper is provided
- Includes exception chaining at line 244 where appropriate
The static analysis warning (B904) at line 232 about missing exception chaining is intentional per the codebase's backward compatibility requirements.
Based on learnings
533-534: LGTM: Resources initialization for--resourcesis correct.Properly uses
Resources()as the default and delegates parsing toResources.parser_factory(allow_expressions=False), which correctly prevents expressions in global resource constraints.
612-614: LGTM: Default resources parser correctly configured.Uses
Resources.parser_factory(defaults="full", allow_expressions=True), which appropriately:
- Enables Python expressions for default resources (e.g.,
2*input.size_mb)- Includes full default values (mem, disk, tmpdir)
- Aligns with the PR's goal to standardize resource handling
src/snakemake/workflow.py (6)
72-74: LGTM: Resource imports correctly updated.The new exception types (
ResourceDuplicationError,ResourceConversionError,ResourceValidationError) andResourcescontainer class are properly imported to support the centralized resource handling.Also applies to: 132-132
182-184: LGTM: Global resources initialization is correct.Properly initializes
global_resourcesfromresource_settings.resources(now aResourcesobject) and adds the required_coresand_nodesentries.
630-631: LGTM: Property access on Resource objects is correct.The
.valueproperty correctly retrieves concrete resource values. Note that per theResourceclass implementation (lines ~535-545 insrc/snakemake/resources.py),.valueraisesValueErrorif the resource is still evaluable (unevaluated). Since_coresand_nodesare set as concrete values in__post_init__, this is safe.Also applies to: 634-635
687-687: LGTM: Return type annotation improves type safety.The added
-> Rulereturn type annotation enhances code clarity and enables better static analysis.
1870-1909: LGTM: Resource handling is comprehensive and correct.The implementation properly:
- Initializes rule resources from defaults or creates empty
Resources()- Sets the required
_nodesresource- Handles thread overrides and validation (lines 1877-1889)
- Processes rule-specific resources via
Resources.from_mapping(line 1896)- Catches and converts all resource-specific exceptions (
ResourceDuplicationError,ResourceConversionError,ResourceValidationError) intoRuleExceptionwith appropriate context (lines 1897-1903)- Applies resource overwrites from CLI/API settings (lines 1907-1908)
The static analysis warnings about exception chaining (B904) and message formatting (TRY003) are expected—exception chaining is intentionally avoided per backward compatibility requirements.
Based on learnings
2010-2010: LGTM: Handover rule resource allocation is appropriate.Correctly provides handover rules with all available global resources via
rule.resources.update(self.global_resources), enabling them to manage resource allocation for their spawned jobs.
|
@pvandyken, see my message on discord. |
e017870 to
d23bb09
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (3)
src/snakemake/dag.py (1)
18-18: Remove duplicateDictimport to satisfy Ruff and avoid F811.The
typingimport currently includesDicttwice:from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Union, DictThis triggers Ruff F811 and is unnecessary. Please drop the trailing
, Dict:-from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Union, Dict +from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Uniontests/common.py (1)
507-511: Still need to guard againstexception is Nonewhen expecting a specific failure.
shouldfailcan be an exception type, butexceptionmay remainNone(e.g. shell-based runs only setsuccess = False), soraise exceptionwill itself raise aTypeErrorinstead of the expected exception, making the assertion misleading. This is the same issue noted in a prior review.Apply this minimal guard to fail clearly if no exception was captured:
- if shouldfail is not False: - assert not success, "expected error on execution" - if shouldfail is not True: - with pytest.raises(shouldfail): - raise exception + if shouldfail is not False: + assert not success, "expected error on execution" + if shouldfail is not True: + if exception is None: + pytest.fail("Expected failure, but no exception was captured.") + with pytest.raises(shouldfail): + raise exceptionsrc/snakemake/cli.py (1)
168-194:parse_set_resourcesneeds more robust validation and error wrapping.Right now:
- Entries without a
":"will raise a bareValueErrorduring unpacking (rule, assign = entry.split(...)), bypassing thetry/exceptand yielding an unfriendly traceback.Resources.parse(...)can raiseWorkflowError(e.g. invalid names or negative values), which currently escapes instead of being converted into the CLI-friendlyValueError(errmsg).- The error message text is slightly off grammatically (“positive integer a quoted string”).
To harden this, I suggest:
-from snakemake.exceptions import ( - CliException, - ResourceScopesException, - print_exception, -) +from snakemake.exceptions import ( + CliException, + ResourceScopesException, + WorkflowError, + print_exception, +) @@ -def parse_set_resources(args: List[str] | None) -> Dict[str, Resources]: - errmsg = ( - "Invalid resource definition: entries have to be defined as " - "RULE:RESOURCE=VALUE, with VALUE being a positive integer a quoted string, or " - "a Python expression (e.g. min(max(2*input.size_mb, 1000), 8000))." - ) +def parse_set_resources(args: List[str] | None) -> Dict[str, Resources]: + errmsg = ( + "Invalid resource definition: entries have to be defined as " + "RULE:RESOURCE=VALUE, with VALUE being a positive integer, a quoted string, " + "or a Python expression (e.g. min(max(2*input.size_mb, 1000), 8000))." + ) @@ - if args is None: - return {} - - assignments: Dict[str, List[str]] = defaultdict(list) - - for entry in args: - rule, assign = entry.split(":", maxsplit=1) - assignments[rule].append(assign) - - try: - return { - rule: Resources.parse(assigns, allow_expressions=True) - for rule, assigns in assignments.items() - } - - except ValueError as err: - raise ValueError(errmsg) from err + if args is None: + return {} + + assignments: Dict[str, List[str]] = defaultdict(list) + try: + for entry in args: + if ":" not in entry: + raise ValueError(f"Unparsable value: {repr(entry)}") + rule, assign = entry.split(":", maxsplit=1) + assignments[rule].append(assign) + + return { + rule: Resources.parse(assigns, allow_expressions=True) + for rule, assigns in assignments.items() + } + except (ValueError, WorkflowError) as err: + raise ValueError(errmsg) from errThis keeps CLI behavior predictable and ensures all parsing failures are reported via a uniform
ValueErrormessage.
🧹 Nitpick comments (8)
src/snakemake/resources.py (5)
223-232: AlignResourceErrorpayload withResourceInsufficiencyError.excess_resourcesexpectations.Here
_get_layersraisesResourceErrorwith a joined string, and you passerr.args[0]asexcess_resourcesintoResourceInsufficiencyError, whose signature/documentation describesexcess_resourcesas a sequence of resources.If you intend
excess_resourcesto be structured (e.g. one entry per offending resource), consider either:
- Raising
ResourceErrorwith a list of offending specs and passing that list through, or- Relaxing the
ResourceInsufficiencyErrortype hint/docstring to explicitly allow a single formatted string.Functionality is fine; this is mainly about keeping the error API and types self-consistent.
448-455:SizedResources/TimeResourcessets are clear; consider documenting them as shared constants.The
SizedResourcesandTimeResourcessets are central to parsing and normalization acrossResource,Resources, andResourceScopes. It may be worth a brief module‑level comment (or docstring mention) that these define the canonical sized/time resource families so future additions stay consistent.No functional issue; just a small maintainability aid.
879-886:Resources.DEFAULTS+default()interaction aligns with the new Resource API.
DEFAULTS["full"]and"bare"are expressed as CLI‑style expressions, whichdefault()wraps viaResource.from_cli_expression.Resources.__init__then normalizes sized resources so that externally you get_mb-suffixed entries for sizes, as expected.- The docstring for
default()still correctly describes that the resultingResourcesincludes pre‑set memory/disk/tmpdir defaults.The mutable
DEFAULTSdict is effectively constant here; if you want to make that explicit for type checkers, annotating it asClassVar[dict[str, dict[str, str]]]would be a nice‑to‑have, but not required.
945-963:Resources.defaulterror message still mentionsDefaultResources.The logic is fine, but the error message still refers to the old
DefaultResourcesname:raise ValueError(f"Unexpected mode for DefaultResources: {mode}")Given the class is now
Resources, consider updating the wording to avoid confusion for API users.
1086-1114:Resources.updateandgetsemantics are slightly non-standard but intentional.
updateaccepts either anotherResourcesor a plain mapping, in the latter case normalizing viafrom_mapping, which is convenient.getreturningResource("", None)instead ofNonekeeps callers from having to special-caseNonebefore calling.valueor.constrain().Since this is a new public API, it may be worth documenting the
get()behavior explicitly to avoid surprises for consumers expecting a dict-likeNonedefault.src/snakemake/jobs.py (2)
103-120: Scheduler resource filtering correctly respects scopes and excludes placeholders.The updated
_get_scheduler_resources()now (a) dropsTBDStringentries, and (b) for non-local execution, omits resources whose scope is local viaresource_scopes.is_local(k). This ensures only applicable, fully-evaluated resources are exposed to the scheduler.If you want to make debugging easier in future, consider documenting (in a short comment) that
_scheduler_resourcesintentionally excludes local-scoped and unevaluated values.
1374-1378: Grouped resource aggregation and exposure viaGroupJob.resourceslook sound.
GroupJob.__init__clearly treatsglobal_resourcesas aResourcesconstraint set, andresourcesnow:
- Computes grouped resources via
GroupResources.basic_layered(...)with appropriate flags for local execution and additive/sort-by keys.- Wraps
WorkflowErrorandResourceInsufficiencyErrorwith group-id context while chaining the original exception.- Returns a
ResourceList(fromdict=self._resources), giving executors a uniform, Namedlist-like view.You might later consider caching the constructed
ResourceListalongside_resourcesif repeated access toGroupJob.resourcesbecomes performance-sensitive.Also applies to: 1539-1557
src/snakemake/io/__init__.py (1)
1747-1756:Namedlistgenerics andResourceListdefinition align with new resource surface.Making
Namedlistgeneric over key and item types, adding self-typed helper methods, and typingitems()asIterator[Tuple[_TNamedKeys, _TNamedList]]strengthens static checking. DefiningResourceList(Namedlist)here cleanly exposes the list-like{resources}object consumed by jobs and format contexts.If you ever specialize
ResourceListfurther (e.g., to expose convenience accessors), consider parameterizing it explicitly (e.g.,ResourceList(Namedlist[str, Any])) for even clearer intent, though this is not required for correctness.Also applies to: 1763-1769, 1869-1871, 2006-2007
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (22)
src/snakemake/api.py(2 hunks)src/snakemake/cli.py(7 hunks)src/snakemake/common/__init__.py(5 hunks)src/snakemake/dag.py(2 hunks)src/snakemake/exceptions.py(3 hunks)src/snakemake/io/__init__.py(8 hunks)src/snakemake/jobs.py(5 hunks)src/snakemake/logging.py(2 hunks)src/snakemake/resources.py(13 hunks)src/snakemake/rules.py(7 hunks)src/snakemake/scheduling/job_scheduler.py(4 hunks)src/snakemake/script/__init__.py(1 hunks)src/snakemake/settings/types.py(3 hunks)src/snakemake/spawn_jobs.py(2 hunks)src/snakemake/workflow.py(7 hunks)test.txt(1 hunks)tests/common.py(7 hunks)tests/test_default_resources/Snakefile(1 hunks)tests/test_github_issue3271/Snakefile_should_fail(1 hunks)tests/test_group_jobs_resources/Snakefile(2 hunks)tests/test_ioutils/Snakefile(1 hunks)tests/tests.py(19 hunks)
🚧 Files skipped from review as they are similar to previous changes (6)
- src/snakemake/logging.py
- tests/test_ioutils/Snakefile
- tests/test_github_issue3271/Snakefile_should_fail
- tests/tests.py
- src/snakemake/exceptions.py
- tests/test_group_jobs_resources/Snakefile
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit configuration file
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
src/snakemake/scheduling/job_scheduler.pytests/common.pysrc/snakemake/script/__init__.pysrc/snakemake/dag.pysrc/snakemake/spawn_jobs.pysrc/snakemake/settings/types.pysrc/snakemake/cli.pysrc/snakemake/jobs.pysrc/snakemake/workflow.pysrc/snakemake/common/__init__.pysrc/snakemake/api.pysrc/snakemake/rules.pysrc/snakemake/io/__init__.pysrc/snakemake/resources.py
🧠 Learnings (21)
📚 Learning: 2024-10-04T16:12:18.927Z
Learnt from: lczech
Repo: snakemake/snakemake PR: 3113
File: snakemake/scheduler.py:912-914
Timestamp: 2024-10-04T16:12:18.927Z
Learning: In `snakemake/scheduler.py`, avoid suggesting the use of `asyncio.gather` in the `jobs_rewards` method due to overhead concerns and the need for immediate results.
Applied to files:
src/snakemake/scheduling/job_scheduler.pysrc/snakemake/jobs.py
📚 Learning: 2024-10-13T14:10:37.796Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the `snakemake/cli.py` file, `ExecutorPluginRegistry()` is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.
Applied to files:
src/snakemake/scheduling/job_scheduler.py
📚 Learning: 2025-10-30T15:39:38.224Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3813
File: tests/test_pathvars/Snakefile:22-23
Timestamp: 2025-10-30T15:39:38.224Z
Learning: In Snakemake, `<resources>` is a default pathvar that does not need to be explicitly defined in the pathvars section. It is available for use in rules and functions without requiring declaration.
Applied to files:
tests/common.pysrc/snakemake/script/__init__.pysrc/snakemake/spawn_jobs.pysrc/snakemake/cli.pysrc/snakemake/jobs.pysrc/snakemake/workflow.pysrc/snakemake/api.pysrc/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-10-14T09:42:11.571Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.
Applied to files:
src/snakemake/dag.py
📚 Learning: 2024-11-12T12:08:20.342Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.
Applied to files:
src/snakemake/dag.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.
Applied to files:
src/snakemake/workflow.pysrc/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-10-06T14:09:54.370Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.
Applied to files:
tests/test_default_resources/Snakefile
📚 Learning: 2025-01-17T12:00:09.368Z
Learnt from: leoschwarz
Repo: snakemake/snakemake PR: 3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
Applied to files:
tests/test_default_resources/Snakefile
📚 Learning: 2024-10-06T14:09:26.494Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.
Applied to files:
src/snakemake/common/__init__.pysrc/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2024-11-07T00:32:44.137Z
Learnt from: mbhall88
Repo: snakemake/snakemake PR: 3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2024-08-13T16:22:09.641Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-08-13T16:22:09.641Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.
Applied to files:
src/snakemake/rules.pysrc/snakemake/io/__init__.py
📚 Learning: 2025-08-28T13:15:55.232Z
Learnt from: Hocnonsense
Repo: snakemake/snakemake PR: 3714
File: src/snakemake/modules.py:195-195
Timestamp: 2025-08-28T13:15:55.232Z
Learning: In src/snakemake/modules.py, the line `self.globals["__name__"] = namespace` is intentionally designed to set __name__ to None when namespace is None (i.e., when not inside a module context, for top-level workflows). This is correct behavior and should not be guarded.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.
Applied to files:
src/snakemake/rules.pysrc/snakemake/io/__init__.py
📚 Learning: 2024-11-12T20:22:54.184Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2985
File: tests/tests.py:2051-2051
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.
Applied to files:
src/snakemake/rules.pysrc/snakemake/io/__init__.pysrc/snakemake/resources.py
📚 Learning: 2025-08-07T21:51:08.043Z
Learnt from: pvandyken
Repo: snakemake/snakemake PR: 3421
File: src/snakemake/resources.py:606-607
Timestamp: 2025-08-07T21:51:08.043Z
Learning: In Snakemake's Resource.constrain method, the type comparison uses isinstance(self_val, type(other_val)) rather than direct type equality to allow strings to be compared with AnnotatedString objects. This polymorphic behavior is intentional and necessary for proper resource constraint functionality.
Applied to files:
src/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-08-27T20:15:58.699Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-08-27T20:15:58.699Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.
Applied to files:
src/snakemake/rules.pysrc/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1250-1251
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid commenting on trailing commas in code reviews for the Snakemake repository as per the user's preference.
Applied to files:
src/snakemake/rules.py
📚 Learning: 2024-09-25T10:55:53.963Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.
Applied to files:
src/snakemake/resources.py
📚 Learning: 2024-12-14T13:11:32.988Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.
Applied to files:
src/snakemake/resources.py
🧬 Code graph analysis (11)
tests/common.py (1)
src/snakemake/resources.py (3)
from_mapping(1061-1084)parse(983-1058)defaults(1209-1210)
src/snakemake/script/__init__.py (3)
src/snakemake/workflow.py (1)
resources(2245-2250)src/snakemake/jobs.py (2)
resources(482-499)resources(1539-1553)src/snakemake/io/__init__.py (1)
ResourceList(2006-2007)
src/snakemake/dag.py (1)
src/snakemake/scheduling/job_scheduler.py (1)
stats(187-188)
src/snakemake/spawn_jobs.py (1)
src/snakemake/settings/types.py (2)
_parsed_overwrite_resources(372-376)_parsed_overwrite_threads(386-392)
src/snakemake/settings/types.py (1)
src/snakemake/resources.py (5)
Resource(460-852)Resources(855-1162)from_mapping(1061-1084)items(1098-1099)default(945-963)
src/snakemake/workflow.py (3)
src/snakemake/exceptions.py (3)
ResourceDuplicationError(665-672)ResourceConversionError(648-662)ResourceValidationError(640-641)src/snakemake/resources.py (8)
ResourceScopes(1168-1306)Resources(855-1162)value(535-545)copy(1105-1107)get(1109-1113)from_mapping(1061-1084)update(1086-1090)update(1303-1306)src/snakemake/settings/types.py (4)
_parsed_resources(368-369)_parsed_default_resources(379-383)_parsed_overwrite_threads(386-392)_parsed_overwrite_resources(372-376)
src/snakemake/common/__init__.py (1)
src/snakemake/exceptions.py (1)
NestedCoroutineError(558-559)
src/snakemake/api.py (1)
src/snakemake/resources.py (2)
Resources(855-1162)default(945-963)
src/snakemake/rules.py (4)
src/snakemake/io/__init__.py (1)
ResourceList(2006-2007)src/snakemake/resources.py (6)
Resource(460-852)Resources(855-1162)evaluate(551-568)constrain(570-601)value(535-545)expand_items(1115-1162)src/snakemake/exceptions.py (5)
ResourceConstraintError(644-645)ResourceValidationError(640-641)InputOpenException(583-586)NestedCoroutineError(558-559)ResourceConversionError(648-662)src/snakemake/persistence.py (1)
rule(474-475)
src/snakemake/io/__init__.py (1)
src/snakemake/resources.py (1)
items(1098-1099)
src/snakemake/resources.py (3)
src/snakemake/common/__init__.py (3)
get_input_function_aux_params(315-327)mb_to_mib(69-70)mib_to_mb(73-74)src/snakemake/exceptions.py (7)
NestedCoroutineError(558-559)ResourceError(607-608)ResourceInsufficiencyError(611-630)ResourceScopesException(633-637)ResourceConversionError(648-662)format(658-662)format_evaluated(650-655)src/snakemake/io/__init__.py (6)
AnnotatedString(956-976)items(1869-1871)get(1914-1919)is_callable(326-327)is_callable(967-968)is_callable(1185-1190)
🪛 Ruff (0.14.7)
src/snakemake/dag.py
18-18: Redefinition of unused Dict from line 18
Remove definition: Dict
(F811)
src/snakemake/settings/types.py
358-358: Do not perform function call immutables.Map in dataclass defaults
(RUF009)
src/snakemake/cli.py
232-232: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
244-244: Avoid specifying long messages outside the exception class
(TRY003)
src/snakemake/jobs.py
1550-1552: Avoid specifying long messages outside the exception class
(TRY003)
src/snakemake/workflow.py
1889-1892: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
1889-1892: Avoid specifying long messages outside the exception class
(TRY003)
1897-1897: Avoid specifying long messages outside the exception class
(TRY003)
src/snakemake/common/__init__.py
70-70: Value being cast to int is already an integer
Remove unnecessary int call
(RUF046)
74-74: Value being cast to int is already an integer
Remove unnecessary int call
(RUF046)
111-117: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
111-117: Avoid specifying long messages outside the exception class
(TRY003)
src/snakemake/rules.py
1114-1114: Unused lambda argument: e
(ARG005)
1141-1141: Avoid specifying long messages outside the exception class
(TRY003)
1154-1158: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
1154-1158: Avoid specifying long messages outside the exception class
(TRY003)
1167-1167: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
src/snakemake/io/__init__.py
1-1: The file is executable but no shebang is present
(EXE002)
src/snakemake/resources.py
387-387: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
400-400: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
541-544: Avoid specifying long messages outside the exception class
(TRY003)
568-568: Use raise without specifying exception name
Remove exception name
(TRY201)
647-649: Avoid specifying long messages outside the exception class
(TRY003)
741-741: Unused function argument: wildcards
(ARG001)
745-745: Unused function argument: rulename
(ARG001)
756-756: Unused function argument: wildcards
(ARG001)
756-756: Unused function argument: rulename
(ARG001)
772-772: Unused static method argument: name
(ARG004)
781-790: Use of possibly insecure function; consider using ast.literal_eval
(S307)
796-796: Do not catch blind exception: Exception
(BLE001)
805-819: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
805-819: Avoid specifying long messages outside the exception class
(TRY003)
844-844: Value being cast to int is already an integer
Remove unnecessary int call
(RUF046)
849-849: Value being cast to int is already an integer
Remove unnecessary int call
(RUF046)
879-886: Mutable class attributes should be annotated with typing.ClassVar
(RUF012)
920-920: Avoid specifying long messages outside the exception class
(TRY003)
942-942: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
942-942: Avoid specifying long messages outside the exception class
(TRY003)
956-956: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
956-956: Avoid specifying long messages outside the exception class
(TRY003)
1011-1014: Avoid specifying long messages outside the exception class
(TRY003)
1049-1051: Avoid specifying long messages outside the exception class
(TRY003)
1191-1195: Avoid specifying long messages outside the exception class
(TRY003)
🔇 Additional comments (53)
src/snakemake/dag.py (1)
3178-3203:DAG.statstyping change looks consistent with implementation.The updated return annotation
-> Tuple[str, Dict[str, int]]matches the actual return value(message, stats_dict)and aligns with how stats are consumed elsewhere (e.g. schedulers). No behavior change introduced here.src/snakemake/resources.py (18)
492-517:Resource.__init__validation and evaluator detection look sound.The constructor correctly:
- Restricts values to
str | int | float | callable | None | TBDStringand normalizes floats to rounded ints.- Parses human‑friendly strings only for known sized/time resources.
- Detects both plain callables and
AnnotatedStringcallables as evaluators, leaving others as concrete.This matches the intended behavior described in the docstring and should integrate cleanly with
evaluate()andstandardize_size().
570-601:Resource.constraincorrectly enforces type compatibility, including AnnotatedString.The method:
- Bails out for
TBDStringandNonevalues as intended.- Uses
isinstance(self_val, type(other_val))rather than directtype(...)equality, which preserves compatibility between plainstrandAnnotatedString(per prior discussion).- Only applies a
min()constraint when both sides areint, returningselfotherwise.This is a good balance between safety and the polymorphic behavior needed for annotated strings.
603-625:from_mb_to_mibhandles both concrete and evaluable resources correctly.Using
_convert_units(mb_to_mib)combined with renaming to*_mibensures:
- Concrete integer MB values are converted once.
- Evaluable resources are wrapped so the converter is applied post‑evaluation.
- Non‑numeric/evaluable types raise
ResourceConversionErroras expected.This matches the documented behavior and works with later normalization in
Resources.
626-651:format_human_friendlybehavior is reasonable; just be aware it assumes MB units.The method:
- Rejects unevaluated resources with a
TypeError, which is safer than silently formatting.- Delegates to
_convert_units(mb_to_str), so it will also correctly preserveTBDString.- Assumes the stored unit is MB and only strips
_mb, which is consistent with howstandardize_size()normalizes sized resources.No changes needed; just worth noting the MB assumption when adding new sized resource types.
678-715:standardize_size’s normalization flow is coherent withResources._normalize_sizes.The method:
- Validates
_mbresources using an identity converter.- Converts
_mib→_mbviamib_to_mb.- For unsuffixed sized resources, wraps the evaluator (if any) to ensure evaluation and human‑friendly parsing happen before attaching
_mb.This aligns with
Resources._normalize_sizes, which converts any of{name, name_mb, name_mib}into a single_mbentry. Behavior looks correct.
722-769: CLI expression helpers (from_cli_expression/cli_evaluator) are consistent with workflow semantics.
from_cli_expressionbuilds evaluators with the expected signature used by the job machinery (optionally includingthreads), and preserves the raw expression string.cli_evaluatorlimits the eval environment toinput,attempt,system_tmpdir,shutil, and optionallythreads, and falls back to treating values as plain strings onNameError/SyntaxErroror whenis_ordinary_stringreturns True.- File‑not‑found conditions are mapped to
TBDStringin line with the rest of the resource evaluation logic.Given Snakemake’s explicit support for Python expressions in resource specifications, this is an appropriate design.
823-852:parse_human_friendlybehavior and error messages look good.
- Only string values (excluding
TBDString) for known human‑friendly resources are parsed.- Size strings are converted to at least 1 MB; time strings to at least 1 minute, with clear unit‑specific error messages.
- Non‑human‑friendly resources and non‑string values are passed through untouched.
Note that callers like
Resources.parseintentionally swallowWorkflowErrorhere to re‑interpret inputs as plain ints/expressions, which matches the PR’s design notes.
888-908:Resources.__init__and_normalize_sizesimplement sized-resource canonicalization correctly.
- Passing a
Dict[str, Resource]preserves existing instances, whileNoneinitializes an empty container._normalize_sizesenforces that only one of{name, name_mb, name_mib}is present and then normalizes to a single_mbentry viastandardize_size(), raisingResourceDuplicationErrorotherwise.This achieves the “one effective sized resource per name” invariant described in the PR.
937-943:Resources.decode_argprovides minimal but sufficient validation.Splitting on the first
=and raisingValueErrorwhen it’s missing is consistent with the CLI contract that resources must be specified asname=value. Callers higher up already wrap this in Snakemake-specific error handling.
965-981:parser_factorycleanly wrapsparsefor argparse-style usage.The returned lambda closes over
allow_expressionsanddefaultsand simply forwards the list of CLI strings toResources.parse, which matches the intended usage in the CLI layer. No issues here.
983-1057:Resources.parsebehavior matches documented CLI semantics.Key points:
- Enforces valid identifiers and reserves
_cores.- Attempts human‑friendly parsing first, then integer parsing, then (optionally) expression parsing; otherwise it raises
WorkflowErrorwith a clear message.- Accepts zero and positive integers but rejects negatives, in line with earlier discussion about zero being valid for some flags.
- Properly merges defaults when
defaultsis provided, with explicit override semantics.Overall this looks correct and matches the PR’s stated behavior.
1060-1084:Resources.from_mappingcovers both plain mappings and existingResources.
- Short‑circuiting when
mappingis already aResourcesinstance avoids unnecessary wrapping.- For plain mappings, values are eagerly wrapped in
Resource, so all downstream normalization and validation logic applies.This is a straightforward and useful constructor.
1115-1163:Resources.expand_itemscorrectly evaluates, constrains, and expands sized resources.
- Respects an optional
evaluatecallback and skips names inskip.- Applies per-resource constraints via
Resource.constrain.- When
expand_sizedis True, emits three entries for each*_mbsized resource:_mb(int), unsuffixed (human‑friendly string), and_mib(int), which matches the PR’s design that executors can choose the most convenient representation.This should give executors and plugins a coherent, backwards‑compatible view of resources.
1165-1207:ResourceScopesnormalization and validation logic look correct.
- Constructor normalizes sized resource names to their unsuffixed form and raises
ResourceDuplicationErrorfor conflicting variants, mirroringResources.- It allows the internal
"excluded"scope while keeping the error message user‑facing (“local”/“global” only), as intended._normalize_sizescorrectly promotes a single suffixed variant to the base name.No functional issues spotted here.
1208-1219:ResourceScopes.defaultsand_testgive a clear, centralized scoping policy.Defaults of
mem_mb="local",disk_mb="local", andruntime="excluded"align with the intended scheduler behavior, and_testproperly collapses_mb/_mibto the base sized resource for scope checks.
1220-1255: Scope query helpers onResourceScopesare straightforward and consistent.
is_local,is_global, andis_excludedall delegate to_test, and the correspondinglocals,globals, andexcludedproperties expose these as sets, withexcludedalso including all suffixed sized resources when the base is excluded. This matches the intended semantics for scheduler vs. executor handling.
1303-1307:ResourceScopes.updatepreserves normalization on incoming mappings.By wrapping non‑
ResourceScopesinputs in a newResourceScopesinstance, you ensure sized resources in overrides are normalized before merging, preventing conflicting suffix variants from slipping through. Good defensive design.
1309-1342:is_ordinary_stringheuristic is reasonable for distinguishing expressions from “plain” values.The regex specifically filters out function calls, dict literals, lambdas, arithmetic, and attribute access. Combined with the higher-level handling in
cli_evaluator(only used on non‑NameError/SyntaxErrorexceptions), this should avoid misclassifying most “resource-like” strings while still surfacing genuine expression errors.test.txt (1)
1-1: Fixture file looks fine.The
test.txtfile containing3.0is a simple fixture and does not impact runtime behavior.src/snakemake/api.py (1)
66-66: Switch toResources.default("full")maintains previous default-resources behavior.Importing
Resourcesand using:self.workflow_api.resource_settings.default_resources = Resources.default("full")for non-local execution cleanly aligns the API with the new
Resourcesabstraction while preserving the prior semantics of “full” default resources in cluster/cloud mode.Also applies to: 571-575
src/snakemake/script/__init__.py (1)
114-114: LGTM!The type annotation change from
io_.Resourcestoio_.ResourceListaligns with the PR's rename of the oldResourceclass toResourceListfor executor plugin compatibility. This is consistent with the class definition insrc/snakemake/io/__init__.pyand usage insrc/snakemake/jobs.py.src/snakemake/spawn_jobs.py (2)
98-119: LGTM!The refactored resource handling correctly uses the new
_parsed_overwrite_resourcesand_parsed_overwrite_threadsproperties, which returnResourcesandResourceobjects respectively. Accessing.rawon these Resource objects to serialize the original CLI argument is consistent with the new Resource API design.
364-371: LGTM!The change to use
_parsed_default_resourceswithattr="args"correctly serializes default resources for spawned jobs using the new Resources API. Theargsattribute provides the CLI-compatible representation needed for subprocess invocation.src/snakemake/common/__init__.py (3)
69-75: Consider usingraise ... from efor proper exception chaining.The static analysis correctly identifies that in the
async_runfunction (lines 111-117), the exception should be raised with explicit chaining. However, looking at the actual code, theRuntimeErroris already passed as the second argument toNestedCoroutineError. If this is intentional (storing the original exception), this is acceptable. Otherwise, consider usingraise ... from efor standard exception chaining.The
mib_to_mbfunction is a useful addition that complementsmb_to_mib. Theint()calls on lines 70 and 74 are technically redundant sincemath.ceilandmath.flooralready return integers in Python 3, but they serve as explicit type documentation.
111-117: Exception handling looks intentional but verify chaining behavior.The
NestedCoroutineErroris raised with the originalRuntimeErrorpassed as a second argument. This appears intentional, asWorkflowError(the parent class) likely handles this pattern. The static analysis hint (B904) aboutraise ... from erris a false positive in this context since the original error is explicitly passed to preserve the chain.
315-327: LGTM!The updated type annotations for
get_input_function_aux_paramsimprove code clarity. UsingCallable[..., Any]andMapping[str, Any]appropriately describes the function's interface.tests/test_default_resources/Snakefile (2)
9-14: LGTM!The expanded test correctly validates that sized resources now expose all three representations (
mem_mb,mem,mem_mib). The expected values are mathematically consistent: 1 GB = 1000 MB, and 1000 MB ≈ 954 MiB (using the mb_to_mib conversion factor of ~0.9537).
21-26: LGTM!Consistent with rule
a, this test validates the disk resource representations. The shell syntax with backslash line continuations is correct for bash.src/snakemake/scheduling/job_scheduler.py (2)
1-2: LGTM!The import reorganization and TYPE_CHECKING pattern for the
Workflowtype hint follow Python best practices for avoiding circular imports while maintaining type safety.Also applies to: 10-13, 19-19, 39-41
103-119: Resource expansion logic appears intentional per prior discussion.The
expand_sized=Falsesetting and string filtering has been previously discussed, with the author clarifying that docstrings were updated to explain the intended behavior. The change to accessworkflow.global_resources["_nodes"].valuecorrectly uses the new Resource API to check if nodes are explicitly set.Using
copy.copy(self.global_resources)instead ofdict(self.global_resources)is a good practice for explicit intent.tests/common.py (2)
11-20: Imports andis_connectedURLError handling look correct.The additional typing imports,
urllib.errorusage, andResourceScopes/Resourcesimports are consistent with later code; catchingurllib.error.URLErroris the right exception type here.Also applies to: 32-32, 57-65
163-209: Resource normalization inrun()aligns with newResourcesAPI.Wrapping
resourcesviaResources.from_mapping(...)and parsing list-styledefault_resourceswithResources.parse(..., defaults="full", allow_expressions=True)ensures tests exercise the same code paths as the CLI and always handResourceSettingsproperResourcesinstances.Also applies to: 373-399
src/snakemake/jobs.py (2)
41-45: Resource-related imports are consistent with new API usage.Using
ResourceListfromsnakemake.ioalongsideGroupResources,Resources, andResourceInsufficiencyErrormatches how resources and grouped resource errors are handled later in this module.Also applies to: 52-53, 56-62
482-499:Job.resourcesnow returningResourceListis compatible with existing usage.The property still lazily expands rule resources, respects
skip_evaluationfortmpdir, merges_aux_resources, and now exposes them as aResourceList, preserving attribute/item access semantics needed by{resources}in formatting and byJob.threads.src/snakemake/io/__init__.py (2)
1-1: Future import and extended typing imports are appropriate.Enabling postponed evaluation of annotations and pulling in
Generic,Iterator, andTuplematches the new generic/iterator typing later in this module without affecting runtime behavior.Also applies to: 27-41
1489-1497: Type refinements fordo_expandandglob_wildcardsimprove clarity without semantic change.Using
Dict[str, Union[str, collections.abc.Iterable[str]]]andList[str]for wildcard name collections better reflects actual usage and keeps compatibility with older Python versions while leaving the expansion/globbing logic untouched.Also applies to: 1650-1652
src/snakemake/cli.py (4)
13-13: Typing and resource imports are consistent with later usage.Adding
Dictand importingResource,Resources, andResourceScopesmatches how parsing, resource scopes, andResourceSettingsare wired up below.Also applies to: 42-46
110-123:parse_set_threadsand generalizedparse_set_intscorrectly support ints and expressions.The new
wrapperconverts positive integers intoResource("_cores", n)and defers non-ints toResource.from_cli_expression, whileparse_set_ints:
- Enforces non-negativity for integer values.
- Propagates conversion errors as
ValueError(errmsg)when no wrapper is used.- Wraps any wrapper failures with
raise ValueError(... ) from e.This preserves existing integer semantics and adds robust expression handling for
--set-threads.Also applies to: 223-245
529-547: Switching--resourcestoResources.parser_factoryis a good centralization.Using
default=Resources()plusparse_func=Resources.parser_factory(allow_expressions=False)standardizes global resource parsing and validation, and ensures CLI-provided--resourcesvalues flow through the same code paths as the rest of the resource system.
608-614:--default-resourcesnow correctly leveragesResources.parser_factorywith defaults.Wrapping
Resources.parser_factory(defaults="full", allow_expressions=True)inmaybe_base64(...)lets profile/CLI values populateargs.default_resourcesas a fully initializedResourcesinstance with the built-in defaults, including expression support, matching the behavior used elsewhere in the workflow API.src/snakemake/workflow.py (5)
182-184: LGTM!The refactoring to use
_parsed_resourcescentralizes resource handling correctly. The Resources object supports item assignment for_coresand_nodes.
630-635: LGTM!The property accessors correctly extract
.valuefrom the Resource objects now stored inglobal_resources.
687-698: LGTM!Adding the return type annotation improves API clarity.
1873-1912: LGTM!The refactored resource handling correctly:
- Initializes rule resources from parsed default resources with a copy
- Handles thread overwrites with proper validation
- Parses rule-defined resources via
Resources.from_mapping- Translates resource-specific exceptions to
RuleExceptionwith contextBased on learnings, the exception handling pattern without chaining in some places is intentional for backward compatibility.
2011-2018: LGTM!The handover handling correctly uses
Resources.update()to merge global resources wholesale when no explicit resources are provided.src/snakemake/settings/types.py (3)
349-350: LGTM!The
ValidResourcetype alias correctly captures the full range of valid resource types, including the recursive callable case for dynamic resource evaluation.
352-365: LGTM!The field type unions provide flexibility by accepting both raw mappings (for CLI/API compatibility) and pre-parsed Resources objects. The use of
immutables.Map()as default is a safe, pre-existing pattern in this codebase.
367-392: LGTM!The cached property accessors provide clean lazy parsing:
_parsed_resourcesand_parsed_overwrite_resourcescorrectly useResources.from_mapping_parsed_default_resourcesdefaults to "bare" mode whenNone_parsed_overwrite_threadsnormalizes both raw integers and Resource objects to Resource instancessrc/snakemake/rules.py (4)
52-58: LGTM!The new resource-related imports align with the refactored resource handling architecture and are all used appropriately in the file.
97-97: LGTM!The explicit type annotations for
self.workflowandself.resourcesimprove code clarity and enable better IDE support without introducing circular import issues.Also applies to: 109-109
1101-1128: LGTM!The
evaluate()helper function correctly:
- Skips evaluation for resources in
skip_evaluation, returning TBDString- Handles
ResourceValidationErrorby wrapping inWorkflowError- Handles
NestedCoroutineErrorgracefully for async nesting scenarios (well-documented)- Propagates other exceptions as
InputFunctionExceptionThe unused lambda argument
eat line 1114 is required by theincomplete_checkpoint_funccallback signature.
1130-1171: LGTM!The refactored
expand_resourcescorrectly:
- Evaluates and constrains threads via chained
.constrain()calls- Uses
Resources.expand_items()for centralized resource evaluation and expansion- Translates
ResourceConstraintErrorandResourceConversionErrorto contextualWorkflowError- Returns a
ResourceListfor executor plugin compatibilityBased on learnings, the exception handling pattern without explicit chaining is intentional for this codebase.
01d165b to
bf37c8c
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/snakemake/common/__init__.py (1)
16-26:⚠️ Potential issue | 🟠 MajorFix NameError by aligning exception type and docstring with imports.
Line 132 raises
WorkflowError, but this exception is not imported (onlyNestedCoroutineErroris imported on line 25). Switch the raised exception toNestedCoroutineErrorand update the docstrings at lines 119 and 174 to reflect this change.🛠️ Proposed fixes
- WorkflowError - If called from within an already running event loop. + NestedCoroutineError + If called from within an already running event loop.- raise WorkflowError( + raise NestedCoroutineError( "Error running coroutine in event loop. Snakemake currently does not " "support being executed from an already running event loop. " "If you run Snakemake e.g. from a Jupyter notebook, make sure to spawn a " "separate process for Snakemake.", )- WorkflowError - If called from within an already running event loop. + NestedCoroutineError + If called from within an already running event loop.tests/tests.py (1)
1057-1084:⚠️ Potential issue | 🟠 MajorFix expected
mem_mibvalue for 60000 MB (Line 1077).Other tests in this file already assert
mem_mib=57221for 60000 MB. Keeping57222here will diverge and likely fail with the shared conversion logic.🐛 Proposed fix
- mem_mib=57222, + mem_mib=57221,
🤖 Fix all issues with AI agents
In `@src/snakemake/io/__init__.py`:
- Line 1: The package init module (snakemake.io/__init__.py) is marked
executable but contains only imports; either remove the executable bit from this
module file (make it non-executable) if it’s intended only for import, or add a
proper shebang line (e.g., #!/usr/bin/env python3) at the top to declare it
executable; update the file permission or add the shebang consistently with
project conventions and CI/tooling expectations.
In `@src/snakemake/workflow.py`:
- Around line 1901-1940: Remove exception chaining when re-raising RuleException
for resource errors: in the block handling ruleinfo.resources, change the two
raises that use "from err" so they simply raise RuleException(err, rule=rule)
and raise RuleException(msg + str(err), rule=rule) respectively; keep the same
catch clauses for ResourceDuplicationError, ResourceConversionError, and
ResourceValidationError and preserve the original messages and rule parameter
but do not append "from err" to the raise statements (affecting the raise in the
ResourceDuplicationError and ResourceConversionError handlers).
🧹 Nitpick comments (1)
src/snakemake/resources.py (1)
890-897: Consider addingClassVarannotation for the class constant.The
DEFAULTSdictionary is a class-level constant that should not be mutated. While this works correctly as-is, adding aClassVarannotation would make the intent clearer and satisfy type checkers.+from typing import ClassVar + DEFAULTS: ClassVar[dict[str, dict[str, str]]] = { - DEFAULTS = {
This centralizes the code for resource handling into the resource.py
file under a new Resource class. Notably, DefaultResources has been
removed, it's functionality rolled into the new class. The same class is
now used for handling global resource limits, default resources, and
rule resource constraints
As a result of this standarization, the following bugs/inconsistencies are fixed:
1) humanreadable resources can now be specified in a group job in
cluster submit without crashing the program
2) if mem or disk are used as global resource limits, they will
correctly restrict mem_mb, mem_mib, disk_mb, and disk_mib
3) similar to above when setting a global scope for mem and disk
4) humanreadable resources can now be used for global resource limits
5) disk_mib and mem_mib can be set in rule resource specifications,
global limits, and in default resources, and will correctly set the
_mb variants.
6) Providing the same effective resource twice (e.g. mem, mem_mb and/or
mem_mib) in the same rule or in the global resource specification
raises an error.
7) QOL: Resources in tests can be set using mappings
8) Runtime is handled as a human-readable resource
9) Original raw resource specifications are maintained to pass on to
subprocesses
10) Fix handling of input-based resource lambdas
Previously, lambdas with an input parameter were skipped by inspecting
the arguments of the lambda via introspection. This did not work well
with the new refactor, which involves potentially deep nesting of these
lambdas. Instead, always run these input functions, relying on the fact
that missing input files will already result in TBDStrings, the same
outcome as the previous skip_evaluation.
Currently limited by second bug, where input file sizes cannot be
evaulated in the initial dag creation routine due to nesting of
asynchronous routines. For now, simply catch these nesting errors and
routine TBDString when they occur
11) Move resource expansion logic to resources.py
Better separation of concerns: in particular, handling of "magic"
resources "mem" and "disk" and their suffixed versions is handled
totally within `Resources`. `Rule.expand_resources` now only has logic
that requires direct knowledge of the rule.
12) Move humanfriendly from expr to main parse block
Previously only evaluated in the context of epxressions, preventing use
of human friendly resources with --resources. No reason for this, as
they can be parsed deterministically.
13) Sized resources are stored without a suffix
This makes much of the logic more straightforward, and facilitates
allowing callables on mem and disk to return strings which can then be
parsed as human-readable
14) Remove positive_integer arguments, CLI should always validate
that integers are positive
15) Exclude the string versions of sized resources from the group layer
function, then add in after
bf37c8c to
8157552
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In `@src/snakemake/resources.py`:
- Around line 225-233: Remove exception chaining when wrapping ResourceError and
WorkflowError: in the block that calls cls._get_layers (where ResourceError is
caught and you currently raise ResourceInsufficiencyError(additive_resources,
err.args[0]) from err) drop the "from err" so you simply raise
ResourceInsufficiencyError(additive_resources, err.args[0]). Apply the same
change for the two places that re-raise WorkflowError (the raise
WorkflowError(...) from err occurrences referenced at lines ~857 and ~862) so
those become plain raises without "from err". Ensure the caught exception
variable (err) is still used to extract the message but do not chain the
original exception.
In `@src/snakemake/rules.py`:
- Around line 1094-1169: In the evaluate(...) inner function, remove exception
chaining when re-raising ResourceValidationError and BaseException: change the
ResourceValidationError handler that does "raise WorkflowError(err, rule=self)
from err" to just "raise WorkflowError(err, rule=self)" and change the
BaseException handler that does "raise InputFunctionException(e, rule=self,
wildcards=wildcards) from e" to "raise InputFunctionException(e, rule=self,
wildcards=wildcards)"; keep all other behavior the same so evaluate,
ResourceValidationError and InputFunctionException usages are unchanged.
In `@test.txt`:
- Line 1: The new fixture file test.txt containing "3.0" appears unused; either
wire it into the test suite or remove it: search for any tests or helper modules
that should consume this fixture (tests that mention "test.txt" or use
fixture-loading utilities) and update the appropriate test(s) or loader function
to import/read test.txt (e.g., add a fixture path reference or call the existing
loadFixture utility), or delete test.txt if no consumer is required; ensure the
change is reflected in the test setup so CI no longer has a stray artifact.
🧹 Nitpick comments (1)
src/snakemake/logging.py (1)
68-73: Consider aligning the error message with the broader Mapping acceptance.
Right now the ValueError says “dict or Namedlist,” but you now accept any Mapping. Updating the message keeps debugging clearer.🔧 Suggested tweak
- raise ValueError( - "bug: format_dict applied to something neither a dict nor a Namedlist" - ) + raise ValueError( + "bug: format_dict applied to something neither a Mapping nor a Namedlist" + )
8157552 to
ff3b8c2
Compare
🤖 I have created a release *beep* *boop* --- ## [9.16.3](v9.16.2...v9.16.3) (2026-02-05) ### Bug Fixes * standardize handling of resources ([#3421](#3421)) ([4cada18](4cada18)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
|
Thank you for yet another contribution to I used to use the following in my I just upgraded to
I'm sure there are good reasons for this, but can you suggest a workaround to achieve the same effect as the 'hack' above? |
|
@pvandyken great work and thanks for not giving up on this PR! 😄 |
Fixes dynamic resource parsing regression introduced in #3421. Closes #3970 ### QC <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Preserve unresolved placeholder resource values during evaluation so pending resource expressions remain intact and are not incorrectly converted. * **Tests** * Added a test validating a minimal sequential workflow to confirm resource handling. * Updated an existing test to adjust embedded shell invocation structure for more robust execution. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Johannes Koester <johannes.koester@uni-due.de>
This centralizes the code for resource handling into the
resource.pyfile under a newResourceclass. Notably,DefaultResourceshas been removed, its functionality rolled into the new class. The same class is now used for handling global resource limits, default resources, and rule resource constraintsAs a result of this standarization, the following bugs/inconsistencies are fixed:
mem,mem_mb, andmem_mibcan now be used interchangeably across--resources,--set-resource-scopes,--default-resources,--set-resources, and theresourcekeyword within aSnakefile, as the names are internally standardized.--resources)mem_mbandmem_mibcan no longer be provided a string or a function that returns a string within a rule.{resources}object available within a shell command or parameter context now has access to all versions of a sized resource, e.g.mem,mem_mb,mem_mib.memis now always a string with a unit.A few additional changes:
Resourceclass in snakemake/io.py (that subclassesNamedlistwas renamed toResourceListto prevent confusion. This class is still used within theJobclass and is ultimately passed onto the executor. This ensures compatibility with the existing plugin interface.Outstanding issues (not to be resolved here)
--set-threadswith a python expressionGotchas
QC
docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).Summary by CodeRabbit
Release Notes
New Features
Bug Fixes
Refactor