Skip to content

fix: standardize handling of resources#3421

Merged
johanneskoester merged 4 commits intosnakemake:mainfrom
pvandyken:fix/human-friendly-units-groups
Feb 5, 2026
Merged

fix: standardize handling of resources#3421
johanneskoester merged 4 commits intosnakemake:mainfrom
pvandyken:fix/human-friendly-units-groups

Conversation

@pvandyken
Copy link
Copy Markdown
Contributor

@pvandyken pvandyken commented Mar 13, 2025

This centralizes the code for resource handling into the resource.py file under a new Resource class. Notably, DefaultResources has been removed, its functionality rolled into the new class. The same class is now used for handling global resource limits, default resources, and rule resource constraints

As a result of this standarization, the following bugs/inconsistencies are fixed:

  1. humanreadable resources can now be specified in a group job in cluster submit without crashing the program (fix: group resources #3367)
  2. mem, mem_mb, and mem_mib can now be used interchangeably across --resources, --set-resource-scopes, --default-resources, --set-resources, and the resource keyword within a Snakefile, as the names are internally standardized.
  3. humanreadable resources can now be used in all resource contexts (e.g. --resources)
  4. Providing the same effective resource twice (e.g. mem, mem_mb and/or mem_mib) in the same rule or in the global resource specification raises an error.
  5. BREAKING mem_mb and mem_mib can no longer be provided a string or a function that returns a string within a rule.
  6. The {resources} object available within a shell command or parameter context now has access to all versions of a sized resource, e.g. mem, mem_mb, mem_mib. mem is now always a string with a unit.

A few additional changes:

  • The old Resource class in snakemake/io.py (that subclasses Namedlist was renamed to ResourceList to prevent confusion. This class is still used within the Job class and is ultimately passed onto the executor. This ensures compatibility with the existing plugin interface.
  • I added the ability to watch for a specific Exception class within the tests to help increase the resolution of some of my "failure" tests. This is completely backwards-compatible
  • Additional typing annotations were added throughout the code I touched.
  • A small rearrangement in the code within snakemake/workflow.py to group the resource-related code together.

Outstanding issues (not to be resolved here)

  1. It was, and remains, possible to sneak a string value as a thread using --set-threads with a python expression
  2. --resources cannot be given an expression, per design. Currently, though, nothing stops one from from providing an expression directly via the api. This problem preexists this PR.
  3. There is currently no testing of the cli parsing functions. This should be written in a new PR.
  4. The check for positive integers is only done for the CLI. Resources requested within rules will not be so validated. This is unchanged from previous

Gotchas

  1. This does represent a breaking change for API use, as the dataclasses will not perform any automatic conversion. Specifically, ResourceSettings.resources, .overwrite_threads, .overwrite_resources, and .default_resources will be broken in dependent apps.

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Summary by CodeRabbit

Release Notes

  • New Features

    • Support for human-readable resource specifications (e.g., "1 GB", "1 GiB")
    • New resource units: mem_mib, disk_mib alongside existing mem_mb, disk_mb
  • Bug Fixes

    • Improved resource handling and validation with better error messages
    • Enhanced resource constraint detection and reporting
  • Refactor

    • Unified resource management system for simplified configuration
    • Streamlined CLI resource parsing and argument handling

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 13, 2025

📝 Walkthrough

Walkthrough

This pull request introduces a comprehensive refactoring of Snakemake's resource management system. It replaces legacy DefaultResources and ParsedResource classes with new Resource and Resources classes offering unified API for resource handling across CLI parsing, workflow execution, job scheduling, and rule expansion. New exception types are added for resource-related errors, and existing APIs are updated to use the new resource abstractions.

Changes

Cohort / File(s) Summary
Resource System Core
src/snakemake/resources.py
Complete rewrite introducing Resource class for individual resource management and Resources class as a Mapping-like container with parsing, evaluation, and normalization capabilities. Adds ResourceScopes enhancements for scope predicates (is_local, is_global, is_excluded). Includes extensive public API: from_cli_expression, parse, parser_factory, default, from_mapping and unit conversion helpers.
Exception Hierarchy
src/snakemake/exceptions.py
Adds new resource-related exception classes: NestedCoroutineError, ResourceError, ResourceInsufficiencyError, ResourceValidationError, ResourceConstraintError, ResourceConversionError, ResourceDuplicationError with specialized constructors and classmethods for error formatting.
Public API Updates
src/snakemake/api.py, src/snakemake/script/__init__.py
Updates to use new Resources.default("full") instead of DefaultResources(mode="full"); parameter type changed from io_.Resources to io_.ResourceList in Snakemake constructor.
CLI Parsing
src/snakemake/cli.py, src/snakemake/settings/types.py
Refactored resource and thread parsing to use Resources.parser_factory and Resources.parse. Added parse_set_threads wrapper converting ints to Resource("_cores", n). Updated resource settings with ValidResource type alias and cached property parsing for _parsed_resources, _parsed_overwrite_resources, etc.
IO Type System
src/snakemake/io/__init__.py
Renamed Resources class to ResourceList (Namedlist subclass). Enhanced with generics: Namedlist(Generic[_TNamedKeys, _TNamedList]) with updated items() return type annotation.
Common Utilities
src/snakemake/common/__init__.py, src/snakemake/logging.py
Added mib_to_mb conversion function. Updated mb_to_mib signature with type annotation. Enhanced get_input_function_aux_params to accept Callable and Mapping types. Replaced WorkflowError with NestedCoroutineError in async context. Updated format_dict to accept Mapping type.
Job and Resource Management
src/snakemake/jobs.py, src/snakemake/scheduling/job_scheduler.py, src/snakemake/spawn_jobs.py
Job.resources now returns ResourceList. GroupJob handles resource aggregation with both WorkflowError and ResourceInsufficiencyError. JobScheduler expands workflow.global_resources via expand_items(...) and uses value.raw for CLI formatting. Removed ParsedResource import and _get_resources_to_skip helper.
Workflow and Rules
src/snakemake/workflow.py, src/snakemake/rules.py, src/snakemake/dag.py
Workflow.global_resources sourced from _parsed_resources; resource initialization and merging consolidated via Resources.from_mapping. Rule.expand_resources refactored to use expand_items with refined error handling for ResourceConstraintError and ResourceConversionError. Added type hints for Workflow, Resources attributes. DAG.stats signature updated to use capitalized typing aliases.
Tests
tests/common.py, tests/tests.py, tests/test_*.../Snakefile*
Updated test framework to import and use Resources instead of DefaultResources. Added tests for new exceptions (ResourceConversionError, ResourceDuplicationError). Updated Snakefile tests to use new resource formats (mem="1 GB", mem_mib, disk="1 GB") and assertions validating Resources parsing and expansion.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120+ minutes

Possibly related PRs

Suggested reviewers

  • johanneskoester
🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 29.45% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title 'fix: standardize handling of resources' accurately reflects the main objective of the PR, which centralizes resource handling through a new Resource class and standardizes how mem, mem_mb, and mem_mib are handled across the codebase.
Description check ✅ Passed The PR description is comprehensive and well-structured. It clearly explains the changes, lists specific fixes and behavioral changes, documents breaking changes, and acknowledges outstanding issues and gotchas. Both QC checkboxes are marked as complete.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@pvandyken pvandyken added the bug Something isn't working label Mar 13, 2025
@pvandyken pvandyken force-pushed the fix/human-friendly-units-groups branch 2 times, most recently from ab20f60 to f98ac7b Compare March 13, 2025 18:00
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

Actionable comments posted: 1

🧹 Nitpick comments (22)
src/snakemake/workflow.py (1)

166-166: Validate the implications of assigning resources by reference.

self.global_resources = self.resource_settings.resources creates a direct reference rather than a copy. If unintentional shared state is not desired later, consider using a copy; otherwise, this is fine if continued synchronization is intended.

- self.global_resources = self.resource_settings.resources
+ self.global_resources = deepcopy(self.resource_settings.resources)
src/snakemake/rules.py (1)

1027-1031: Consider chaining the original exception.

According to B904, raising a new exception in an except block can be improved by chaining the old exception (raise ... from err). If the original context is unnecessary, you could also use from None.

-except ResourceValidationError:
-    raise WorkflowError(
+except ResourceValidationError as err:
+    raise WorkflowError(
🧰 Tools
🪛 Ruff (0.8.2)

1027-1031: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/cli.py (1)

183-202: Consider removing obsolete code.

The commented-out code appears fully replaced by the new resource parsing logic. To keep the codebase clean, consider removing it rather than commenting it out.

-    # (Old commented-out code here)
-    # ...
+    # Remove this entire commented section to avoid confusion
tests/common.py (1)

442-446: Clarity in test failure logic.

The new checks handle boolean vs. exception usage correctly. The nested condition can be slightly confusing but remains logically sound.

src/snakemake/settings/types.py (1)

353-354: Default resource mode.

Setting Resources.default("bare") is likely correct for minimal resource usage. Ensure it’s intentional versus "full" elsewhere for consistency.

src/snakemake/jobs.py (1)

122-132: Use the local variable v for consistency.
Within the dictionary comprehension, you already unpack (k, v). For clarity and readability, prefer referencing v instead of self.resources[k].

-        res_dict = {
-            k: v
-            for k, v in self.resources.items()
-            if not self.dag.workflow.resource_scopes.is_local(k)
-            and not isinstance(self.resources[k], TBDString)
-        }
+        res_dict = {
+            k: v
+            for k, v in self.resources.items()
+            if not self.dag.workflow.resource_scopes.is_local(k)
+            and not isinstance(v, TBDString)
+        }
tests/tests.py (4)

800-801: Clarify removed assertion snippet.

The commented-out assertion in line 801 might still be relevant or helpful for debugging. Either remove it entirely or convert it into an explanatory comment if it contains important context:

-# assert {res: resources[res] for res in expected_resources} == expected_resources
+# If needed in debugging, consider providing some clarifying context here

1054-1058: Eliminate ambiguous variable name and optimize comparison.

Instead of l and not l == "\n", rename the loop variable to something descriptive and use != for clarity. Example fix:

- lines = [l for l in f.readlines() if not l == "\n"]
+ lines = [line_content for line_content in f.readlines() if line_content != "\n"]
🧰 Tools
🪛 Ruff (0.8.2)

1054-1054: Ambiguous variable name: l

(E741)


1054-1054: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)


1131-1131: Remove or refine explanatory comment.

"Implicitly tests the normalization of mem_mb and disk_mb..." can be clarified further or removed if self-explanatory. Comment clarity aids future maintainers.


1148-1148: Consider verifying all suffixed forms.

You assert combined resource fields (mem=60000, disk=3000). Adding checks for _mib or subtle edge cases would ensure that normalizations remain robust across future code changes.

src/snakemake/resources.py (12)

2-29: Remove unused imports.

Imports such as TypeVar, UserDict, Dict, Generic, Tuple, Union, isin appear unused and can be removed to keep the code clean.

Apply an example diff:

- from ast import TypeVar
- from collections import UserDict, defaultdict
+ from collections import defaultdict
...
- from typing import TYPE_CHECKING, Any, Callable, Dict, ...
+ from typing import TYPE_CHECKING, Any, Callable, ...
...
- from numpy import isin
🧰 Tools
🪛 Ruff (0.8.2)

2-2: ast.TypeVar imported but unused

Remove unused import: ast.TypeVar

(F401)


3-3: collections.UserDict imported but unused

Remove unused import: collections.UserDict

(F401)


16-16: typing.Dict imported but unused

Remove unused import

(F401)


17-17: typing.Generic imported but unused

Remove unused import

(F401)


23-23: typing.Tuple imported but unused

Remove unused import

(F401)


25-25: typing.Union imported but unused

Remove unused import

(F401)


29-29: numpy.isin imported but unused

Remove unused import: numpy.isin

(F401)


135-137: Proper initialization of total_resources and blocks.

Using defaultdict(int) for total_resources plus an ordinary list for blocks is consistent. Confirm that these data structures are reset for each new group job evaluation to avoid stale state if the method is reused.


141-144: Check for empty sibling blocks.

block_resources, job_resources, and pipe_resources are all set up well. Provide test coverage for empty sibling sets or no pipe groups to confirm stable handling of degenerate cases.


190-190: Centralize local variable naming.

resource_types = list(set(it.chain(*job_resources))) is clear though a bit nested. Consider a short comment clarifying it merges all resource keys from multiple jobs for readability.


219-219: Validate sortby usage.

Bringing in sortby from the constraints ensures layered scheduling. Consider logging or debugging prints if resource layering leads to unexpected group job expansions.


386-390: Clarify _proportion and _highest_proportion docstrings.

These helper functions are crucial for layering logic. Add docstrings or comments to highlight that 0 is returned if there is no constraint.


394-394: Initializing rows.

rows: list[list[tuple[int, ...]]] = [[]] works for the layering approach. Include a quick comment clarifying why you start with a single empty row rather than an empty list for better maintainability.


431-433: Warn on single-resource overshoot.

Exceeding constraints in pipe jobs is flagged here. The current messaging is succinct. If possible, mention the job or pipe group name for easier debugging.


596-596: Check type comparisons.

Instead of if type(self_val) != type(other_val):, prefer if not isinstance(self_val, type(other_val)): for clarity and best practice:

-if type(self_val) != type(other_val):
+if not isinstance(self_val, type(other_val)):
🧰 Tools
🪛 Ruff (0.8.2)

596-596: Use is and is not for type comparisons, or isinstance() for isinstance checks

(E721)


754-765: Raise exceptions with explicit cause.

Pythonic style recommends chaining exceptions for clarity:

-except Exception as e:
-    raise WorkflowError(...) 
+except Exception as e:
+    raise WorkflowError(...) from e
🧰 Tools
🪛 Ruff (0.8.2)

754-765: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


805-805: Use 'is None' checks.

Line 805 compares if mapping == None:; prefer if mapping is None: to comply with Python style.

🧰 Tools
🪛 Ruff (0.8.2)

805-805: Comparison to None should be cond is None

Replace with cond is None

(E711)


856-856: Consider using exception chaining.

When re-raising a caught exception, do:

raise ValueError("...") from err
🧰 Tools
🪛 Ruff (0.8.2)

856-856: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🛑 Comments failed to post (1)
src/snakemake/resources.py (1)

293-299: 🛠️ Refactor suggestion

Avoid mutable defaults in function parameters.

skip: list[str] = [] and methods: dict[str, Callable[[list[int]], int]] = {} can introduce subtle bugs. Use None and initialize inside the function:

-def _merge_resource_dict(
-    cls,
-    resources: list[dict[str, str | int]],
-    skip: list[str] = [],
-    methods: dict[str, Callable[[list[int]], int]] = {},
-    default_method: Callable[[list[int]], int] = max,
-):
+def _merge_resource_dict(
+    cls,
+    resources: list[dict[str, str | int]],
+    skip: Optional[list[str]] = None,
+    methods: Optional[dict[str, Callable[[list[int]], int]]] = None,
+    default_method: Callable[[list[int]], int] = max,
+):
+    if skip is None:
+        skip = []
+    if methods is None:
+        methods = {}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    def _merge_resource_dict(
        cls,
        resources: list[dict[str, str | int]],
        skip: Optional[list[str]] = None,
        methods: Optional[dict[str, Callable[[list[int]], int]]] = None,
        default_method: Callable[[list[int]], int] = max,
    ):
        if skip is None:
            skip = []
        if methods is None:
            methods = {}
🧰 Tools
🪛 Ruff (0.8.2)

296-296: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


297-297: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (12)
src/snakemake/workflow.py (2)

1764-1777: Enhanced threads handling with proper error handling.

The try-except block catches ResourceValidationError with a clear error message. The conditional logic for thread assignment is well-structured and clear.

However, there's a potential improvement for the exception handling:

-            except ResourceValidationError:
-                raise RuleException(
+            except ResourceValidationError as err:
+                raise RuleException(
                    "Threads value has to be an integer, float, or a callable.",
                    rule=rule,
-                )
+                ) from err

Adding from err to the exception would preserve the exception chain for better debugging.

🧰 Tools
🪛 Ruff (0.8.2)

1773-1776: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1778-1791: Improved resource validation.

The updated resource handling code properly validates resources and provides clear error messages. The error message includes the type of the problematic resource.

Similarly to the previous comment, consider preserving the exception chain:

-                    raise RuleException(
+                    raise RuleException(
                        "Resources values have to be integers, strings, or callables "
                        f"(functions): type({err.args[0]} : {err.args[1]}) == "
                        f"{type(err.args[1])}",
                        rule=rule,
-                    )
+                    ) from err
🧰 Tools
🪛 Ruff (0.8.2)

1785-1790: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/rules.py (2)

1031-1058: Improved resource evaluation with clear error handling.

The new evaluate function is a significant improvement:

  1. It properly handles resource evaluation with optional skipping
  2. Provides clear error messages for validation failures
  3. Uses the new resource types consistently

The error handling is comprehensive, capturing both ResourceValidationError and general exceptions.

Consider preserving the exception chain for clearer debugging in these exception handlers:

-            except ResourceValidationError:
-                raise WorkflowError(
+            except ResourceValidationError as err:
+                raise WorkflowError(
                    f"Resource {resource} is neither int, float (would be rounded to "
                    "nearest int), str, or None.",
                    rule=self,
-                )
+                ) from err

And similarly for the BaseException handler.

🧰 Tools
🪛 Ruff (0.8.2)

1049-1053: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1068-1094: Robust resource evaluation and constraint checking.

This section properly evaluates resources, handles None values, and applies constraints from global resources with appropriate type checking. The error message for type mismatches is clear and informative.

Again, consider preserving the exception chain:

-            except ResourceConstraintError:
+            except ResourceConstraintError as err:
                global_val = global_res.value
                global_type = (
                    "an int" if isinstance(global_val, int) else type(global_val)
                )
-                raise WorkflowError(
+                raise WorkflowError(
                    f"Resource {resource} is of type {type(val.value).__name__} but "
                    f"global resource constraint defines {global_type} with value "
                    f"{global_val}. Resources with the same name need to have the same "
                    "types (int, float, or str are allowed).",
                    rule=self,
-                )
+                ) from err
🧰 Tools
🪛 Ruff (0.8.2)

1086-1092: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

tests/tests.py (1)

1136-1136: Fix spelling error in comment

There's a typo in the comment: "Implicitely" should be "Implicitly".

-    # Implicitely tests the normalization of mem_mb and disk_mb to mem and disk
+    # Implicitly tests the normalization of mem_mb and disk_mb to mem and disk
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 1136-1136:
Implicitely ==> Implicitly

🪛 GitHub Actions: Codespell

[error] 1136-1136: codespell: Implicitely ==> Implicitly

src/snakemake/resources.py (7)

1-29: Clean up unused imports

Several imports are declared but never used in the code, which can lead to maintainability issues.

from __future__ import annotations
-from ast import TypeVar
from collections import UserDict, defaultdict
import copy
from dataclasses import dataclass
from humanfriendly import InvalidTimespan, InvalidSize, parse_size, parse_timespan
import itertools as it
import operator as op
import re
import tempfile
import math
from typing import (
    TYPE_CHECKING,
    Any,
    Callable,
-    Dict,
-    Generic,
    Iterator,
    Literal,
    Mapping,
    Self,
    Sequence,
-    Tuple,
    TypeAlias,
-    Union,
    cast,
)

-from numpy import isin
🧰 Tools
🪛 Ruff (0.8.2)

2-2: ast.TypeVar imported but unused

Remove unused import: ast.TypeVar

(F401)


3-3: collections.UserDict imported but unused

Remove unused import: collections.UserDict

(F401)


16-16: typing.Dict imported but unused

Remove unused import

(F401)


17-17: typing.Generic imported but unused

Remove unused import

(F401)


23-23: typing.Tuple imported but unused

Remove unused import

(F401)


25-25: typing.Union imported but unused

Remove unused import

(F401)


29-29: numpy.isin imported but unused

Remove unused import: numpy.isin

(F401)


525-538: Replace if-else with ternary in repr method

A small optimization opportunity in the repr method.

def __repr__(self):
-    if self.is_evaluable():
-        value = "function(...)"
-    else:
-        value = self._value
+    value = "function(...)" if self.is_evaluable() else self._value
    return f'Resource("{self.name}", {value})'
🧰 Tools
🪛 Ruff (0.8.2)

526-529: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


566-577: Fix typo in docstring

There's a typo in the docstring for the evaluate method.

def evaluate(self, *args: Any, **kwargs: Any):
    """Evaluate the resource with given args.

    This method ensures the returned resource is correctly returned in a
-    new ``Resource`` class, validated, and standardized. Resources should thus
-    only be evaulated via this method.
+    new ``Resource`` class, validated, and standardized. Resources should thus
+    only be evaluated via this method.
    """
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 571-571:
evaulated ==> evaluated


674-720: Fix typo in docstring

There's a typo in the docstring for the from_cli_expression method.

@classmethod
def from_cli_expression(
    cls, name: str, value: str, *, with_threads_arg: bool = True
):
    """Create a new evaluable resource based on a python expression.

    Threads can optionally be included in the expression environment

    Arguments
    =========
    name:
        name of the resource
    value:
-        python expression to be evaulated
+        python expression to be evaluated
    with_threads_arg: boolean
        If True, include ``threads`` as an argument in the returned function
    """
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 687-687:
evaulated ==> evaluated


898-965: Fix typo in docstring

There's a typo in the docstring for the parse method.

@classmethod
def parse(
    cls,
    exprs: list[str],
    *,
    allow_expressions: bool = False,
    only_positive_integers: bool = False,
    defaults: None | Literal["bare"] | Literal["full"] = None,
):
    """Parse a series of CLI-style string resource assigments.
-    """
+    """Parse a series of CLI-style string resource assignments.
+    """
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 907-907:
assigments ==> assignments


487-487: Fix typo in docstring

There's a typo in the docstring for the Resource class.

"""Standardized representation of a resource name and value.

In addition to ``int`` and ``str``, the standard resource types, ``Resource`` can
store ``None``, signifying an unset resource, and a callable, signifying a resource
to be calculated in the context of a rule. Via the ``.value`` property, ``Resource``
-ensures a resource has been computed before its value is retreived.
+ensures a resource has been computed before its value is retrieved.
"""
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 487-487:
retreived ==> retrieved


492-492: Fix typo in docstring

There's a typo in the docstring for the Resource class.

The class additionally handles input validation and standardization.
``float``s get rounded into ``int``. If the resource name corresponds to a size
resource ("disk" and "mem") or a timespan resource ("runtime"), strings values
-will be intepreted as a human readable resource and converted to an integer of
+will be interpreted as a human readable resource and converted to an integer of
appropriate denomination (Mb for size, hr for timespan).
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 492-492:
intepreted ==> interpreted

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between fd41744 and f98ac7b.

📒 Files selected for processing (18)
  • src/snakemake/api.py (2 hunks)
  • src/snakemake/cli.py (5 hunks)
  • src/snakemake/common/__init__.py (1 hunks)
  • src/snakemake/exceptions.py (2 hunks)
  • src/snakemake/io.py (5 hunks)
  • src/snakemake/jobs.py (5 hunks)
  • src/snakemake/logging.py (2 hunks)
  • src/snakemake/resources.py (16 hunks)
  • src/snakemake/rules.py (6 hunks)
  • src/snakemake/scheduler.py (5 hunks)
  • src/snakemake/script/__init__.py (1 hunks)
  • src/snakemake/settings/types.py (2 hunks)
  • src/snakemake/workflow.py (7 hunks)
  • tests/common.py (6 hunks)
  • tests/test_all_temp/Snakefile (0 hunks)
  • tests/test_default_resources/Snakefile (1 hunks)
  • tests/test_group_jobs_resources/Snakefile (2 hunks)
  • tests/tests.py (17 hunks)
💤 Files with no reviewable changes (1)
  • tests/test_all_temp/Snakefile
🚧 Files skipped from review as they are similar to previous changes (10)
  • tests/test_default_resources/Snakefile
  • tests/test_group_jobs_resources/Snakefile
  • src/snakemake/script/init.py
  • src/snakemake/common/init.py
  • src/snakemake/logging.py
  • src/snakemake/api.py
  • src/snakemake/scheduler.py
  • tests/common.py
  • src/snakemake/io.py
  • src/snakemake/jobs.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • src/snakemake/workflow.py
  • src/snakemake/exceptions.py
  • src/snakemake/rules.py
  • src/snakemake/settings/types.py
  • src/snakemake/resources.py
  • tests/tests.py
  • src/snakemake/cli.py
🪛 Ruff (0.8.2)
src/snakemake/workflow.py

1773-1776: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1785-1790: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/rules.py

1049-1053: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1086-1092: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/resources.py

2-2: ast.TypeVar imported but unused

Remove unused import: ast.TypeVar

(F401)


3-3: collections.UserDict imported but unused

Remove unused import: collections.UserDict

(F401)


16-16: typing.Dict imported but unused

Remove unused import

(F401)


17-17: typing.Generic imported but unused

Remove unused import

(F401)


23-23: typing.Tuple imported but unused

Remove unused import

(F401)


25-25: typing.Union imported but unused

Remove unused import

(F401)


29-29: numpy.isin imported but unused

Remove unused import: numpy.isin

(F401)


296-296: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


297-297: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


526-529: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


596-596: Use is and is not for type comparisons, or isinstance() for isinstance checks

(E721)


754-765: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


805-805: Comparison to None should be cond is None

Replace with cond is None

(E711)


856-856: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


870-870: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

tests/tests.py

1059-1059: Ambiguous variable name: l

(E741)


1059-1059: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)


1085-1085: Ambiguous variable name: l

(E741)


1085-1085: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py

[failure] 487-487:
retreived ==> retrieved


[failure] 492-492:
intepreted ==> interpreted


[failure] 571-571:
evaulated ==> evaluated


[failure] 687-687:
evaulated ==> evaluated


[failure] 907-907:
assigments ==> assignments

tests/tests.py

[failure] 1136-1136:
Implicitely ==> Implicitly

🪛 GitHub Actions: Codespell
tests/tests.py

[error] 1136-1136: codespell: Implicitely ==> Implicitly

⏰ Context from checks skipped due to timeout of 90000ms (5)
  • GitHub Check: tests (9, windows-latest, py311, bash)
  • GitHub Check: tests (7, windows-latest, py312, bash)
  • GitHub Check: tests (7, ubuntu-latest, py312, bash)
  • GitHub Check: tests (4, windows-latest, py312, bash)
  • GitHub Check: tests (4, ubuntu-latest, py312, bash)
🔇 Additional comments (73)
src/snakemake/exceptions.py (4)

10-10: Update to type imports is appropriate.

Adding Sequence from typing module is necessary for proper type annotation in the new exception classes.


609-610: Good addition of ResourceValidationError class.

This custom exception type helps distinguish resource validation errors from other type errors, making error handling more specific.


613-614: ResourceConstraintError provides clear error categorization.

This exception helps distinguish resource constraint violations from general type errors, improving error clarity.


617-624: Well-structured ResourceDuplicationError with helpful error message.

This exception properly captures duplicate resource information and constructs a clear error message, making debugging easier for users.

src/snakemake/workflow.py (7)

58-61: Appropriate import of new ResourceValidationError class.

Adding the import for the new exception class is necessary for proper error handling in resource validation.


118-118: Updated import for Resources components.

Using the new Resources import path is consistent with the architectural changes made to the resource handling system.


166-166: Simplified global_resources initialization.

Directly assigning self.resource_settings.resources is more straightforward than creating a new dictionary. This reflects the shift to using the Resources class as a container.


569-569: Updated cores retrieval to use the value property.

This change is consistent with the new Resources class structure, which encapsulates resource values with additional metadata.


573-573: Updated nodes retrieval to use the value property.

Similar to the cores update, this change correctly accesses the encapsulated value from the Resources class.


1758-1761: Improved resource initialization.

The change from deepcopy to simple copy for default resources is appropriate, likely improving performance. When no default resources are provided, properly initializing with an empty Resources object maintains type consistency.


1793-1794: Simplified resource overwrite logic.

Using the update method of the Resources class provides a cleaner approach to applying rule-specific resource overwrites.

src/snakemake/rules.py (6)

28-28: Updated import for ResourceList.

Using ResourceList from the io module is appropriate. This reflects the architectural changes in resource handling.


48-53: Consolidated resource imports.

Importing the necessary resource-related classes and exceptions from the new resource module improves code organization and makes dependencies explicit.


74-76: Added typing for workflow import.

Adding type checking for the Workflow import helps with static type checking and IDE support.


102-102: Updated resources type annotation.

Changing the type annotation to Resources | None properly reflects the new resource implementation.


1059-1066: Well-implemented cores resource constraint.

The cores resource is properly evaluated and constrained against the maximum threads setting. The type check ensures thread counts are integers.


1096-1102: Proper handling of memory and disk resources.

This adds support for human-readable resources with automatic conversion between units (MB and MiB). The final return of a ResourceList maintains type consistency.

src/snakemake/settings/types.py (4)

33-33: Updated import for Resources class.

Using the new Resources import is consistent with the architectural changes in resource handling.


344-344: Improved resource type and initialization.

Changing the resources attribute to use the Resources class and providing a default factory ensures proper initialization and type consistency.


348-349: Updated resource type annotations.

The updated type annotations for overwrite_resources and default_resources reflect the architectural changes, ensuring type safety throughout the codebase.


353-353: Use of Resources.default factory method.

Using the static default method of the Resources class to create default resources is cleaner than direct instantiation and likely encapsulates initialization logic.

src/snakemake/cli.py (3)

156-181: Improved resource parsing with cleaner implementation

The function has been completely rewritten to leverage the new Resources class parser, resulting in a more concise and maintainable implementation. The new approach correctly collects multiple assignments for each rule and processes them together, which aligns with the PR objective to centralize resource handling.

@@ Original function parsed key-value pairs and evaluated expressions manually
+  New implementation uses Resources.parse() and handles expressions consistently

515-517: Updated default value and parser for resources argument

The default value for --resources is now a Resources object instead of an empty dictionary, and the parser uses the new Resources.parser_factory() method. This is consistent with the centralized resource handling approach.


594-596: Standardized default resources parsing

The default resources parser now uses the Resources.parser_factory with parameters for defaults and expression allowance, ensuring consistent resource handling throughout the codebase.

tests/tests.py (7)

805-806: Simplified assertion function for resources

The assert_resources function has been simplified to directly compare the entire resources dictionary with expected values, improving clarity and making test failures more informative.


846-870: Added test for MiB resource specifications

New test verifies that resources can be provided in mebibytes (MiB) rather than just megabytes (MB), which is part of the PR objective to support both mem_mb and mem_mib variants.


873-897: Added test for human-readable resource specifications

New test ensures that global resources can be specified in human-readable formats (like "60 Gb"), fulfilling the PR objective to support human-readable resources in various contexts.


900-920: Added validation test for resource duplication

This parametrized test ensures that specifying both prefixed and unprefixed variants of the same resource (e.g., both "mem" and "mem_mb") raises an appropriate error, improving resource specification clarity.


922-945: Added validation tests for suffixed resource formats

These tests verify that suffixed resources (mem_mb, mem_mib, etc.) cannot be provided in human-readable format in constraints or declarations, enforcing consistent resource specifications.


1153-1154: Updated assertion to include normalized resources

The assertion now correctly includes the normalized resource names (mem, disk) along with the original resource names, ensuring consistency with the new resource normalization behavior.


1386-1415: Added tests for human-readable and mebibyte resource specifications

These new tests verify that default resources can be specified in both human-readable format and mebibytes, supporting the PR's goal of standardizing resource handling across different formats.

src/snakemake/resources.py (42)

55-63: Good job adding proper type annotations to GroupResources.basic_layered

The changes to the function signature with proper type annotations improve type safety and make the code more self-documenting. The use of Resources type instead of a generic dictionary is a good improvement.


135-145: Improved variable annotations enhance code readability

The added type annotations for variables improve the readability and type safety of the code.


190-202: Good improvement in type safety with proper casting

The use of cast() to properly type the variables improves type safety and makes the code more robust.


205-207: Resources class property access pattern is consistent

Using the .get() method with .value property access ensures consistent handling of resources.


218-220: Types are properly passed to _get_layers method

The changes properly pass value list to the _get_layers method, improving type consistency.


256-259: Type annotations added to _get_saturated_resource_error method

The method signature now has proper type annotations, which improves code readability and type safety.


274-276: Type annotation for _is_string_resource improves clarity

Adding proper type annotations for parameters helps document the expected input types.


300-325: Improved type safety in _merge_resource_dict method

The addition of type annotations for variables in the method body improves type safety and code readability.


328-333: Enhanced type annotations for _merge_resource_layer

The detailed type annotations provide better documentation and compile-time checking of the method's parameters.


347-349: Type annotations added to _check_constraint method

Proper type annotations for parameters improve code readability and type safety.


368-373: Well-defined type annotations for _get_layers method

The method signature now has clear type annotations, which improves code maintainability.


386-391: Good use of local function type annotations

The inner functions _proportion and _highest_proportion have proper type annotations, enhancing code clarity.


393-400: Type annotation for rows and resource_groups improves clarity

Adding proper type annotations helps clarify the data structures used in the algorithm.


430-433: Fixed type annotation for too_high variable

The precise type annotation for the too_high list makes the code more maintainable.


447-474: Well-implemented helper functions for unit conversion

The helper functions evaluable_from_mb_to_mib and evaluable_from_mib_to_mb are well-implemented and handle type checking properly.


476-479: Good use of type alias and constants

The ValidResource type alias and SizedResources set make the code more maintainable and self-documenting.


481-524: Resource class is well-designed with comprehensive validation

The Resource class provides standardized representation with proper validation and human-readable resource handling.

🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 487-487:
retreived ==> retrieved


[failure] 492-492:
intepreted ==> interpreted


539-551: Good implementation of the value property

The value property correctly ensures that only evaluated resources are accessible, preventing potential errors.


552-565: Well-designed is_evaluable and _get_evaluator methods

These methods provide a clean API for determining if a resource needs evaluation and retrieving the evaluator function.


603-629: Well-implemented to_mib conversion method

The to_mib method properly handles different resource types and provides appropriate error messages.


630-656: Well-implemented to_mb conversion method

The to_mb method properly handles different resource types and provides appropriate error messages.


657-673: Clean implementation of without_suffix and with_name methods

These utility methods provide a clean way to manipulate resource names and handle suffix conversions.


721-767: Good error handling in cli_evaluator method

The method properly handles different error cases with descriptive error messages.

🧰 Tools
🪛 Ruff (0.8.2)

754-765: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


768-777: Well-implemented _parse_human_friendly method

The method properly handles parsing human-readable resource values for both sized resources and runtime.


779-803: Resources class is well-designed with clear documentation

The class provides a standardized container for resources with proper documentation of its functionality.


812-822: Good implementation of _normalize_sizes method

The method properly handles size resource normalization and raises appropriate errors for duplications.


823-836: Good implementation of mapping methods

The basic mapping methods are well-implemented with proper type checking.


837-842: Good implementation of setitem method

The method properly handles different input types.


843-850: Well-implemented args property

The property correctly filters non-evaluable, non-None resources for argument representation.


851-857: Good error handling in decode_arg method

The method properly handles invalid input formats with a descriptive error message.

🧰 Tools
🪛 Ruff (0.8.2)

856-856: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


858-878: Good implementation of default method

The method provides a clean way to initialize resources with default values.

🧰 Tools
🪛 Ruff (0.8.2)

870-870: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


879-897: Well-designed parser_factory method

The method provides a clean way to create parser functions with preset arguments.


966-975: Good implementation of from_mapping method

The method provides a clean way to initialize resources from a mapping.


976-1008: Well-implemented utility methods

The update, keys, values, items, unwrapped_items, unwrapped_nonstr_items, copy, and get methods provide a comprehensive API.


1009-1013: Good use of type alias for ValidScope

The ValidScope type alias makes the code more maintainable and self-documenting.


1013-1041: Well-documented ResourceScopes class

The class provides clear documentation and proper validation of resource scopes.


1042-1052: Good implementation of _normalize_sizes method

The method properly handles size resource normalization and raises appropriate errors for duplications.


1053-1056: Good implementation of defaults method

The method provides sensible default scopes for common resources.


1057-1064: Good implementation of _test method

The method provides a clean way to test if a resource has a specific scope.


1065-1100: Well-documented is_local, is_global, and is_excluded methods

These methods have clear documentation explaining their purpose.


1101-1147: Good implementation of locals, globals, and excluded properties

These properties provide a clean way to get resources by scope.


1148-1152: Good implementation of update method

The method properly handles updating from another ResourceScopes or mapping.

@fgvieira
Copy link
Copy Markdown
Contributor

Fixes #3367

@pvandyken pvandyken force-pushed the fix/human-friendly-units-groups branch from 08bd255 to de55285 Compare March 14, 2025 09:57
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (12)
src/snakemake/workflow.py (2)

1767-1780: Improved thread resource handling

The thread resource handling has been refactored with a try-except block for ResourceValidationError, providing better error messages when thread values are invalid.

However, there's a minor issue with the error handling:

-            except ResourceValidationError:
-                raise RuleException(
-                    "Threads value has to be an integer, float, or a callable.",
-                    rule=rule,
-                )
+            except ResourceValidationError as err:
+                raise RuleException(
+                    "Threads value has to be an integer, float, or a callable.",
+                    rule=rule,
+                ) from err

Adding from err preserves the exception chain, which helps with debugging by showing the original cause.

🧰 Tools
🪛 Ruff (0.8.2)

1776-1779: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1781-1795: Improved resource validation and handling

Added proper error handling for resource parsing with informative error messages. Now resources are handled through the Resources class API with Resources.from_mapping() rather than direct dictionary operations.

The error handling could be improved by preserving the exception chain:

-                except KeyError as err:
-                    raise RuleException(
-                        "Resources values have to be integers, strings, or callables "
-                        f"(functions): type({err.args[0]} : {err.args[1]}) == "
-                        f"{type(err.args[1])}",
-                        rule=rule,
-                    )
+                except KeyError as err:
+                    raise RuleException(
+                        "Resources values have to be integers, strings, or callables "
+                        f"(functions): type({err.args[0]} : {err.args[1]}) == "
+                        f"{type(err.args[1])}",
+                        rule=rule,
+                    ) from err
🧰 Tools
🪛 Ruff (0.8.2)

1788-1793: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/rules.py (1)

1068-1094: Enhanced resource evaluation and constraint application

Improved the handling of all resources with proper evaluation and constraint checking:

  1. Skips _cores which is handled separately
  2. Only evaluates resources that have evaluation functions
  3. Properly handles None values
  4. Applies constraints against global resources with proper error messages
  5. Includes type checking to ensure resources are of compatible types

This makes resource handling more robust and consistent.

The exception handling at line 1086 could benefit from using from err to preserve the exception chain:

-                raise WorkflowError(
-                    f"Resource {resource} is of type {type(val.value).__name__} but "
-                    f"global resource constraint defines {global_type} with value "
-                    f"{global_val}. Resources with the same name need to have the same "
-                    "types (int, float, or str are allowed).",
-                    rule=self,
-                )
+                raise WorkflowError(
+                    f"Resource {resource} is of type {type(val.value).__name__} but "
+                    f"global resource constraint defines {global_type} with value "
+                    f"{global_val}. Resources with the same name need to have the same "
+                    "types (int, float, or str are allowed).",
+                    rule=self,
+                ) from err
🧰 Tools
🪛 Ruff (0.8.2)

1086-1092: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

tests/tests.py (1)

1093-1093: Consider using != operator and improving variable name clarity.

The line uses a negated equality check (not l == "\n") which could be more clearly written as l != "\n". Additionally, the variable name l is ambiguous and could be renamed to something more descriptive like line.

- lines = [l for l in f.readlines() if not l == "\n"]
+ lines = [line for line in f.readlines() if line != "\n"]
🧰 Tools
🪛 Ruff (0.8.2)

1093-1093: Ambiguous variable name: l

(E741)


1093-1093: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

src/snakemake/resources.py (8)

219-220: Use from err when re-raising exceptions for better traceability

When re-raising an exception from a caught exception, use raise ... from err to preserve the original exception's traceback, which helps with debugging.

            except ResourceError as err:
-                raise ResourceInsufficiencyError(additive_resources, err.args[0])
+                raise ResourceInsufficiencyError(additive_resources, err.args[0]) from err
🧰 Tools
🪛 Ruff (0.8.2)

220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


508-514: Consider using a ternary operator for better readability

You can simplify the if-else block with a ternary operator for a more concise representation.

    def __repr__(self):
-        if self.is_evaluable():
-            value = "function(...)"
-        else:
-            value = self._value
+        value = "function(...)" if self.is_evaluable() else self._value
        return f'Resource("{self.name}", {value})'
🧰 Tools
🪛 Ruff (0.8.2)

509-512: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


578-580: Use isinstance() instead of type comparison

Using isinstance() is preferred over direct type comparison as it handles inheritance properly. The current implementation could fail if other_val is a subclass of the type of self_val.

        self_val = self.value
-        if not isinstance(self_val, type(other_val)):
+        if not isinstance(self_val, type(other_val)) or not isinstance(other_val, type(self_val)):
            raise ResourceConstraintError(self_val, other_val)

586-638: Consider reducing code duplication between to_mib and to_mb methods

The to_mib and to_mb methods share very similar structure and logic. Consider refactoring to reduce duplication by creating a helper method that handles the common logic.

You could create a private helper method like _convert_units that both methods use:

def _convert_units(self, target_suffix, conversion_func, evaluable_wrapper):
    if isinstance(self._value, TBDString):
        value = self._value
    elif self.is_evaluable():
        value = evaluable_wrapper(self.name, self._get_evaluator())
    elif not isinstance(self._value, int):
        errmsg = (
            f"Resource must be of type 'int' to convert to {target_suffix}. {self.name} == "
            f"{self._value} (type {type(self._value)})"
        )
        raise TypeError(errmsg)
    else:
        value = conversion_func(self._value)
    return Resource(f"{self.name.removesuffix(target_suffix == 'mib' and '_mb' or '_mib')}_{target_suffix}", value)

def to_mib(self):
    """Convert the resource to mebibytes..."""
    return self._convert_units('mib', mb_to_mib, evaluable_from_mb_to_mib)

def to_mb(self):
    """Convert the resource to megabytes..."""
    return self._convert_units('mb', mib_to_mb, evaluable_from_mib_to_mb)

735-748: Use from err or from None when re-raising exceptions

When re-raising an exception from a caught exception, use raise ... from err to preserve the original exception's traceback, or from None to suppress it intentionally.

            raise WorkflowError(
                "Failed to evaluate resources value "
                f"'{val}'.\n"
                "    String arguments may need additional "
                "quoting. E.g.: --default-resources "
                "\"tmpdir='/home/user/tmp'\" or "
                "--set-resources \"somerule:someresource='--nice=100'\". "
                "This also holds for setting resources inside of a profile, where "
                "you might have to enclose them in single and double quotes, "
                "i.e. someresource: \"'--nice=100'\".",
-                e,
+                e, 
+            ) from e
🧰 Tools
🪛 Ruff (0.8.2)

737-748: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


835-839: Use from err when re-raising exceptions

When re-raising an exception from a caught exception, include from err to preserve the traceback for better debugging.

    def decode_arg(cls, arg: str) -> Tuple[str, str]:
        try:
            return cast(Tuple[str, str], tuple(arg.split("=", maxsplit=1)))
        except ValueError:
-            raise ValueError("Resources have to be defined as name=value pairs.")
+            raise ValueError("Resources have to be defined as name=value pairs.") from None
🧰 Tools
🪛 Ruff (0.8.2)

839-839: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


845-854: Use from err when re-raising exceptions

Include from None when re-raising an exception to clarify that this is a new exception with no traceback linkage to a previous one.

        try:
            default_resources = cls.DEFAULTS[mode or "full"]
        except KeyError:
-            raise ValueError(f"Unexpected mode for DefaultResources: {mode}")
+            raise ValueError(f"Unexpected mode for DefaultResources: {mode}") from None
🧰 Tools
🪛 Ruff (0.8.2)

853-853: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


778-786: Consider moving defaults to a configuration file

Hardcoded default configurations can be difficult to maintain and change. Consider moving these defaults to a configuration file that can be loaded at runtime.

This would make it easier to update default resources without modifying code, and potentially allow users to customize defaults without code changes.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f98ac7b and de55285.

📒 Files selected for processing (19)
  • src/snakemake/api.py (2 hunks)
  • src/snakemake/cli.py (6 hunks)
  • src/snakemake/common/__init__.py (1 hunks)
  • src/snakemake/dag.py (2 hunks)
  • src/snakemake/exceptions.py (2 hunks)
  • src/snakemake/io.py (7 hunks)
  • src/snakemake/jobs.py (5 hunks)
  • src/snakemake/logging.py (2 hunks)
  • src/snakemake/resources.py (14 hunks)
  • src/snakemake/rules.py (6 hunks)
  • src/snakemake/scheduler.py (5 hunks)
  • src/snakemake/script/__init__.py (1 hunks)
  • src/snakemake/settings/types.py (2 hunks)
  • src/snakemake/workflow.py (7 hunks)
  • tests/common.py (6 hunks)
  • tests/test_all_temp/Snakefile (0 hunks)
  • tests/test_default_resources/Snakefile (1 hunks)
  • tests/test_group_jobs_resources/Snakefile (2 hunks)
  • tests/tests.py (17 hunks)
💤 Files with no reviewable changes (1)
  • tests/test_all_temp/Snakefile
✅ Files skipped from review due to trivial changes (1)
  • src/snakemake/dag.py
🚧 Files skipped from review as they are similar to previous changes (7)
  • tests/test_group_jobs_resources/Snakefile
  • src/snakemake/logging.py
  • src/snakemake/script/init.py
  • src/snakemake/api.py
  • src/snakemake/jobs.py
  • src/snakemake/common/init.py
  • tests/test_default_resources/Snakefile
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • src/snakemake/scheduler.py
  • src/snakemake/settings/types.py
  • src/snakemake/workflow.py
  • src/snakemake/rules.py
  • src/snakemake/exceptions.py
  • src/snakemake/io.py
  • src/snakemake/cli.py
  • src/snakemake/resources.py
  • tests/common.py
  • tests/tests.py
🪛 Ruff (0.8.2)
src/snakemake/workflow.py

1776-1779: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1788-1793: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/rules.py

1086-1092: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/resources.py

220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


509-512: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


737-748: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


839-839: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


853-853: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

tests/common.py

196-196: Undefined name List

(F821)

tests/tests.py

1093-1093: Ambiguous variable name: l

(E741)


1093-1093: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

🔇 Additional comments (59)
src/snakemake/scheduler.py (5)

9-9: Adding copy module to support shallow copying of resources

The copy module is now needed for creating a shallow copy of resources instead of creating a new dictionary, which aligns with the move to the new Resources class.


17-17: Improving type safety with conditional imports

Adding TYPE_CHECKING for importing types only during static type checking is a good practice that prevents circular imports at runtime while maintaining proper type checking.

Also applies to: 31-33


60-60: Enhanced type annotation for workflow parameter

Adding explicit type hints for the workflow parameter improves code clarity and enables better IDE support and static type checking.


90-90: Updated resource access method

Changed from using workflow.global_resources.items() to workflow.global_resources.unwrapped_nonstr_items(), which is part of the new Resources class API. This method likely returns the primitive values of resources rather than the Resource objects themselves.


99-99: Simplified resource initialization with shallow copy

Using copy.copy(self.global_resources) rather than manually creating a new dictionary indicates a shift to treating resources as a cohesive object rather than individual key-value pairs. This simplifies the code and ensures all properties of the Resources class are preserved.

src/snakemake/workflow.py (6)

58-60: Added ResourceValidationError to exceptions

Added import for the new ResourceValidationError exception which is used for handling validation errors with resources.


118-119: Updated resource-related imports

Now importing ParsedResource, ResourceScopes, and Resources from the new snakemake.resources module, which is part of the centralized resource handling refactoring.


166-169: Simplified global_resources assignment

Now directly assigning self.resource_settings.resources to global_resources instead of creating a dictionary from it, which is cleaner and preserves the object-oriented nature of the Resources class.


568-569: Updated resource value access

Now accessing resource values through the .value property instead of directly, which is consistent with the new Resource object model where values are wrapped in objects with additional metadata.

Also applies to: 572-573


629-629: Added return type annotation to get_rule

Explicitly annotating the return type of get_rule as Rule improves type checking and documentation.


1761-1762: Optimized resource copying

Using .copy() method instead of deepcopy() for resources, which is more efficient when a shallow copy is sufficient. This is appropriate here since the Resources class likely handles its internal state properly during copying.

src/snakemake/rules.py (9)

28-28: Updated import to include ResourceList

Now importing ResourceList from snakemake.io, which is used as the return type for the refactored expand_resources method.


48-53: Added imports for resource-related classes

Added imports for Resource, ResourceConstraintError, ResourceValidationError, and Resources from the new snakemake.resources module, supporting the centralized resource handling approach.


74-76: Added conditional import for Workflow type

Added conditional import of Workflow for type checking, preventing circular imports at runtime.


90-91: Added type annotation for workflow attribute

Added type annotation for the workflow attribute, improving type safety and documentation.


102-103: Added type annotation for resources attribute

Added type annotation for the resources attribute, specifying it can be either a Resources object or None.


1031-1058: Implemented standalone evaluate function for resource handling

Replaced the nested apply function with a standalone evaluate function that provides improved handling of resource values:

  1. Properly handles skipping evaluation based on the skip_evaluation parameter
  2. Returns a Resource with a TBDString for skipped resources
  3. Uses proper error handling for ResourceValidationError with detailed error messages
  4. Handles input function application correctly

This is a significant improvement in the resource evaluation logic.

However, similar to the previous files, the exception handling could preserve the exception chain:

-            except ResourceValidationError as err:
-                raise WorkflowError(
-                    f"Resource {resource} is neither int, float (would be rounded to "
-                    "nearest int), str, or None.",
-                    rule=self,
-                ) from err

This is correctly using from err to preserve the exception chain, which is good practice.


1059-1067: Updated thread resource evaluation

Now properly evaluates thread resources and applies constraints based on the workflow's max_threads setting. The type checking ensures that thread values are integers, providing better error handling.


1095-1102: Added special handling for memory and disk resources

Improved resource handling by adding automatic conversion between different memory and disk size units:

  1. For both mem and disk resources, automatically sets the _mb variant with the same value
  2. Also sets the _mib variant using a conversion method, ensuring consistency across different units

This allows for human-readable resource specifications while maintaining backward compatibility.


1102-1102: Changed return type to ResourceList

Now returning a ResourceList object instead of a plain dictionary, which is more consistent with the object-oriented approach to resource handling and provides additional methods for working with resources.

tests/common.py (4)

154-154: Good enhancement to shouldfail parameter.

The type annotation has been improved from just bool to bool | type[Exception], enabling more precise testing by allowing specific exception types to be validated.


195-196: Better type annotations for resource parameters.

The type annotations have been updated to reflect the more structured approach to resource handling:

  • resources now uses a clearer Mapping[str, Any] | None type
  • default_resources now accepts a dedicated Resources object or a list of strings
🧰 Tools
🪛 Ruff (0.8.2)

196-196: Undefined name List

(F821)


336-349: Good implementation of resource handling with the new Resources class.

The implementation now properly uses the new Resources class methods:

  1. Resources.from_mapping() to convert mapping to resources object
  2. Resources.parse() for parsing list-based resource definitions
  3. Appropriate conditional handling for different input types

This aligns well with the broader refactoring of resource management.


442-446: Improved exception testing mechanism.

The change from a simple truthy check to an explicit not False check is more precise. Now when shouldfail is an exception type rather than just True, the code properly uses pytest's raises context manager to verify the specific exception type.

src/snakemake/settings/types.py (4)

33-33: Updated import for the new Resources class.

Import now references the new consolidated Resources class instead of the previous resource implementation.


344-344: Good implementation of Resources default factory.

Using field(default_factory=Resources) is a clean way to initialize the resources attribute with an empty Resources instance.


348-349: Consistent type updates for resource-related attributes.

The type annotations have been updated to use the new Resources class:

  • overwrite_resources now mapped to Resources objects instead of nested dictionaries
  • default_resources now explicitly uses Optional[Resources]

This makes the type system more consistent and easier to understand.


353-353: Good replacement of DefaultResources with Resources.default.

The initialization of default_resources now uses the static method Resources.default("bare") instead of creating a DefaultResources instance, maintaining the same functionality while using the new unified resource class.

src/snakemake/cli.py (5)

12-12: Updated typing imports.

The import statement now includes a more comprehensive set of typing annotations, adding Dict, List, and Union which are used in the updated code.


43-43: Updated import for the Resources class.

Now using the new consolidated Resources class instead of the previous implementation.


156-182: Significantly simplified resource parsing.

The parse_set_resources function has been completely rewritten to use the new Resources.parse method, which simplifies the code considerably:

  1. Better type annotation (List[str] | None)
  2. Clearer error handling with specific error messages
  3. More straightforward logic for parsing rule/resource assignments
  4. Delegating the actual parsing to Resources.parse

This refactoring reduces complexity while maintaining functionality.


515-516: Updated resource argument defaults.

The default value for the --res argument now uses Resources() directly, and the parsing function uses Resources.parser_factory(), making it consistent with the new resource handling approach.


594-596: Enhanced default resources parsing.

The --default-resources argument now uses Resources.parser_factory with specific parameters for defaults and expression evaluation, providing better consistency and flexibility.

src/snakemake/exceptions.py (5)

10-10: Added Sequence type import.

Added Sequence from typing module to support type annotations in the new exception classes.


602-603: Added base ResourceError class.

Good addition of a base exception class for all resource-related errors, allowing for more specific error catching.


606-625: Well-designed ResourceInsufficiencyError class.

This exception provides detailed information about resource insufficiency errors:

  1. Takes specific resource types (additive vs excess) as parameters
  2. Constructs a clear error message explaining the issue
  3. Properly handles singular/plural grammar in the message
  4. Stores the resource lists as attributes for potential programmatic access

The error message is particularly helpful as it explains the context (pipe groups) and how resources are calculated.


635-640: Added resource validation exception classes.

Added two specialized exception types:

  • ResourceValidationError for type validation issues
  • ResourceConstraintError for constraint violations

These will help with more specific error handling and better error messages.


643-650: Well-implemented ResourceDuplicationError.

This exception handles the case when equivalent resources are specified multiple times:

  1. Accepts and stores a list of duplicate resources
  2. Provides a clear error message explaining why this is problematic
  3. Inherits from ValueError which is appropriate for this type of error

This will help users understand and fix resource duplication issues more easily.

src/snakemake/io.py (7)

30-32: Appropriate addition of new imports needed for generic types.

The addition of Generic and Iterator from the typing module is necessary to support the improvements to the type annotations in the Namedlist class.


35-35: Good addition of Tuple import.

Adding Tuple to the imports is needed for the return type annotation of the items() method later in the code.


1635-1639: Improved type variable declarations for stronger typing.

The changes to _TNamedList and the addition of _TNamedKeys improve type safety and flexibility:

  1. Removing the bound to Namedlist from _TNamedList allows for more generic usage
  2. The new type variable _TNamedKeys enables proper typing of dictionary-like operations
  3. Clear docstrings explain their purpose for self-returning methods

These changes prepare for making Namedlist properly generic.


1642-1642: Enhanced Namedlist with proper generic typing.

The Namedlist class now properly inherits from list[_TNamedList] and uses Generic[_TNamedKeys, _TNamedList], which significantly improves type safety. This will provide better IDE support and static type checking capabilities.


1651-1651: Improved type annotation for fromdict parameter.

The fromdict parameter now has a more precise type annotation Optional[Dict[_TNamedKeys, _TNamedList]] which clearly indicates that the dictionary keys and values should match the generic types of the Namedlist.


1755-1755: Added return type annotation for items() method.

The return type annotation Iterator[Tuple[_TNamedKeys, _TNamedList]] for the items() method clarifies that it returns an iterator of key-value pairs that match the generic types of the Namedlist. This improves type safety and documentation.


1869-1870: Renamed Resources class to ResourceList with appropriate generic parameters.

Renaming Resources to ResourceList indicates the actual purpose of this class as a list of resources, which avoids confusion with the new Resources class mentioned in the PR objectives. The generic type parameters [str, str | int] correctly specify that keys are strings and values can be either strings or integers.

tests/tests.py (12)

16-16: Added import for ResourceDuplicationError.

This import is needed for the new tests that verify the correct handling of duplicated resource specifications.


18-18: Updated resources imports to use the new Resources class.

The import has been updated to include both GroupResources and the new unified Resources class, replacing what was likely a DefaultResources import previously, aligning with the PR objective of centralizing resource handling.


814-814: Simplified assertion for better readability.

The assert_resources function has been simplified to directly compare the resources dictionary with expected resources, making the code more straightforward and easier to maintain.


835-851: Updated resource handling to use the new Resources interface.

This test has been updated to:

  1. Use a list of resource strings instead of DefaultResources
  2. Add new resource types (mem_mib, disk, disk_mb, disk_mib)
  3. Use a simpler assertion with direct dictionary comparison

These changes align with the PR objectives of standardizing resource handling and supporting various resource formats.


854-878: Added test for resources provided in mebibytes (MiB).

This new test ensures that resources can be correctly specified in mebibytes (MiB) and are properly converted to their megabyte (MB) counterparts, as mentioned in the PR objectives. The test verifies that specifying mem_mib results in correct values for both mem_mb and mem_mib.


881-905: Added test for human-readable resource specifications.

This test verifies that global resources can be specified in human-readable format (e.g., "60 Gb") and are correctly converted to their numeric equivalents. This directly addresses one of the PR objectives of supporting human-readable resources in global resource limits.


908-927: Added test to prevent specifying the same resource in multiple formats.

This parametrized test ensures that an error is raised when the same effective resource is specified in multiple formats (e.g., both mem and mem_mb), directly addressing the PR objective to raise an error in such cases.


930-940: Added test to prevent human-readable format for suffixed resources.

This test verifies that an appropriate error is raised when attempting to specify a suffixed resource constraint (mem_mb) in human-readable format, which helps enforce consistent resource handling.


942-952: Added test for preventing human-readable resources in rule declarations.

This test ensures that rule resource specifications cannot use human-readable formats for suffixed resources, maintaining consistency in how resources are defined throughout the codebase.


966-968: Updated multiple tests to use the new Resources.parse method.

The tests have been updated to use the new Resources.parse method instead of the previous DefaultResources class, ensuring that all tests consistently use the new resource handling mechanism. The calls now include parameters like defaults="full" and allow_expressions=True which provide more flexibility in resource specification.

Also applies to: 996-998, 1035-1037, 1061-1063, 1087-1089, 1113-1115, 1135-1136, 1156-1158, 1179-1181, 1206-1208, 1237-1239


1406-1424: Added test for human-readable default resources.

This test verifies that default resources can be specified in human-readable format (e.g., "1 Gb"), ensuring this feature works consistently across the codebase.


1073-1096: Added test to verify resource prefix handling in scope overwrite.

This test ensures that overwriting a resource scope works correctly regardless of whether the base resource name or a prefixed version is used (e.g., mem vs mem_mb), which is important for consistent resource handling.

🧰 Tools
🪛 Ruff (0.8.2)

1093-1093: Ambiguous variable name: l

(E741)


1093-1093: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

src/snakemake/resources.py (2)

276-281: Good refactoring of mutable default arguments

Replacing mutable default arguments with None defaults and initializing them within the function is a good practice that avoids unexpected behavior.


459-1130: Excellent implementation of the new Resource management system

The comprehensive refactoring of resource handling with the new Resource, Resources, and ResourceScopes classes provides a robust and type-safe system. The implementation correctly handles:

  1. Human-readable resources
  2. Proper enforcement of memory and disk limits
  3. Conversion between MB and MiB units
  4. Error cases for duplicate resource specifications
  5. Default resource handling

The extensive type annotations also improve code clarity and maintainability.

🧰 Tools
🪛 Ruff (0.8.2)

509-512: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


737-748: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


839-839: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


853-853: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

@pvandyken pvandyken changed the title fix:standardize handling of resources fix: standardize handling of resources Mar 14, 2025
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (3)
src/snakemake/resources.py (3)

273-282: 🛠️ Refactor suggestion

Fix mutable default arguments

Using mutable default arguments (empty lists or dictionaries) can lead to unexpected behavior when the function is called multiple times.

def _merge_resource_dict(
    cls,
    resources: List[Dict[str, str | int]],
-    skip: Optional[List[str]] = None,
-    methods: Optional[Dict[str, Callable[[List[int]], int]] = None,
+    skip: Optional[List[str]] = None,
+    methods: Optional[Dict[str, Callable[[List[int]], int]] = None,
    default_method: Callable[[List[int]], int] = max,
):
    skip = skip or []
    methods = methods or {}

589-595: 🛠️ Refactor suggestion

Use isinstance() instead of type comparison

The constrain method uses direct type comparison, which should be replaced with isinstance().

def constrain(self, other: Resource | int | None):
    """Use ``other`` as the maximum value for ``Self``, but only if both are integers.

    Returns self unaltered if either resource is ``None`` or if both are the same
    non-integer type.

    Errors if ``other`` has a different type from ``Self``, or if either are
    evaluable resources.
    """
    if isinstance(self._value, TBDString):
        return self

    other_val = other.value if isinstance(other, Resource) else other

    if self._value is None or other_val is None:
        return self

    self_val = self.value
-    if not isinstance(self_val, type(other_val)):
+    if not isinstance(self_val, type(other_val)) or not isinstance(other_val, type(self_val)):
        raise ResourceConstraintError(self_val, other_val)
    if isinstance(self_val, int):
        assert isinstance(other_val, int)
        return Resource(self.name, min(other_val, self_val))
    return self

799-802: 🛠️ Refactor suggestion

Use "is None" instead of "== None"

Using is None is the proper way to check for None in Python.

def __init__(self, mapping: Dict[str, Resource] | None = None):
-    if mapping == None:
+    if mapping is None:
        self._data: Dict[str, Resource] = {}
        return
    self._data = mapping
🧹 Nitpick comments (8)
src/snakemake/resources.py (8)

29-29: Remove unused imports

The get_function_params and overwrite_function_params are imported but not used in this file.

-from snakemake.common import get_function_params, get_input_function_aux_params, mb_to_mib, mib_to_mb, overwrite_function_params
+from snakemake.common import get_input_function_aux_params, mb_to_mib, mib_to_mb
🧰 Tools
🪛 Ruff (0.8.2)

29-29: snakemake.common.get_function_params imported but unused

Remove unused import

(F401)


29-29: snakemake.common.overwrite_function_params imported but unused

Remove unused import

(F401)


477-477: Fix typo in docstring

The word "retreived" is misspelled and should be "retrieved".

-    ensures a resource has been computed before its value is retreived.
+    ensures a resource has been computed before its value is retrieved.
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 477-477:
retreived ==> retrieved

🪛 GitHub Actions: Codespell

[error] 477-477: codespell: 'retreived' should be 'retrieved'.


482-482: Fix typo in docstring

The word "intepreted" is misspelled and should be "interpreted".

-    will be intepreted as a human readable resource and converted to an integer of
+    will be interpreted as a human readable resource and converted to an integer of
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 482-482:
intepreted ==> interpreted


564-564: Fix typo in docstring

The word "evaulated" is misspelled and should be "evaluated".

-    standardized. Resources should thus only be evaulated via this method.
+    standardized. Resources should thus only be evaluated via this method.
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 564-564:
evaulated ==> evaluated


680-680: Fix typo in docstring

The word "evaulated" is misspelled and should be "evaluated".

-    value: python expression to be evaulated
+    value: python expression to be evaluated
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 680-680:
evaulated ==> evaluated


901-903: Fix typo in docstring

The word "assigments" is misspelled and should be "assignments".

-    """Parse a series of CLI-style string resource assigments.
+    """Parse a series of CLI-style string resource assignments.
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 901-901:
assigments ==> assignments


220-220: Improve exception chain handling

When raising an exception in an except block, you should use raise ... from err to preserve the exception chain for better debugging.

        try:
            layers = cls._get_layers(
                int_resources, list(sorted_constraints.values()), sortby
            )
        except ResourceError as err:
-            raise ResourceInsufficiencyError(additive_resources, err.args[0])
+            raise ResourceInsufficiencyError(additive_resources, err.args[0]) from err
🧰 Tools
🪛 Ruff (0.8.2)

220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


529-533: Use ternary operator for conditional assignment

You can simplify this conditional assignment using a Python ternary operator.

def __repr__(self):
-    if self.is_evaluable():
-        value = "function(...)"
-    else:
-        value = self._value
+    value = "function(...)" if self.is_evaluable() else self._value
    return f'Resource("{self.name}", {value})'
🧰 Tools
🪛 Ruff (0.8.2)

529-532: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 05edae6 and 4a09b5d.

📒 Files selected for processing (4)
  • src/snakemake/common/__init__.py (3 hunks)
  • src/snakemake/io.py (8 hunks)
  • src/snakemake/resources.py (14 hunks)
  • tests/common.py (6 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/common.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • src/snakemake/common/__init__.py
  • src/snakemake/io.py
  • src/snakemake/resources.py
🪛 Ruff (0.8.2)
src/snakemake/resources.py

29-29: snakemake.common.get_function_params imported but unused

Remove unused import

(F401)


29-29: snakemake.common.overwrite_function_params imported but unused

Remove unused import

(F401)


220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


529-532: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


748-759: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


850-850: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


864-864: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py

[failure] 477-477:
retreived ==> retrieved


[failure] 482-482:
intepreted ==> interpreted


[failure] 564-564:
evaulated ==> evaluated


[failure] 680-680:
evaulated ==> evaluated


[failure] 901-901:
assigments ==> assignments

🪛 GitHub Actions: Codespell
src/snakemake/resources.py

[error] 477-477: codespell: 'retreived' should be 'retrieved'.

⏰ Context from checks skipped due to timeout of 90000ms (40)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (10, windows-latest, py311)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (9, windows-latest, py311)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (8, windows-latest, py311)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (7, windows-latest, py312)
  • GitHub Check: tests (7, windows-latest, py311)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (6, windows-latest, py312)
  • GitHub Check: tests (6, windows-latest, py311)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (5, windows-latest, py311)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (4, windows-latest, py311)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (3, windows-latest, py311)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (2, windows-latest, py311)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (1, windows-latest, py312)
  • GitHub Check: tests (1, windows-latest, py311)
  • GitHub Check: tests (1, ubuntu-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py311)
🔇 Additional comments (12)
src/snakemake/resources.py (4)

430-457: Well-implemented resource unit conversion

These helper functions properly handle the conversion between megabytes and mebibytes for evaluable resources, including appropriate error handling for invalid return types.


471-670: Well-designed Resource class

The Resource class provides a comprehensive solution for standardizing resource representation, validation, and conversion. It supports different resource types (int, str, callable) and provides proper error handling.

🧰 Tools
🪛 Ruff (0.8.2)

529-532: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)

🪛 GitHub Check: Check for spelling errors

[failure] 477-477:
retreived ==> retrieved


[failure] 482-482:
intepreted ==> interpreted


[failure] 564-564:
evaulated ==> evaluated

🪛 GitHub Actions: Codespell

[error] 477-477: codespell: 'retreived' should be 'retrieved'.


773-996: Well-implemented Resources container class

The Resources class provides a standardized container for resources with proper handling of resource types, parsing of CLI-style resource assignments, and management of default resources. The implementation follows the Mapping protocol correctly.

🧰 Tools
🪛 Ruff (0.8.2)

850-850: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


864-864: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors

[failure] 901-901:
assigments ==> assignments


1007-1146: Well-structured ResourceScopes class

The ResourceScopes class appropriately manages resource scopes (local, global, excluded) with comprehensive methods for checking scopes and properties for retrieving resources by scope.

src/snakemake/common/__init__.py (2)

64-70: Good implementation of megabyte-mebibyte conversion functions

The implementation of mb_to_mib and mib_to_mb functions provides a clear and consistent way to convert between these units, with proper rounding behavior (ceiling for mb to mib, flooring for mib to mb).


308-320: Improved type annotations for auxiliary parameter extraction

The get_input_function_aux_params function now has proper type annotations, which improves code readability and type safety. The function correctly filters parameters based on what the provided function can accept.

src/snakemake/io.py (6)

1-1: Good addition of future annotations import

Adding the __future__ import for annotations allows for more flexible type hinting throughout the file.


1636-1640: Improved type variable declarations

The type variables _TNamedList and _TNamedKeys are now properly documented with clear comments explaining their purpose.


1643-1643: Enhanced type safety with generic Namedlist

The Namedlist class now properly uses generics to ensure type safety when working with keys and values.

class Namedlist(list[_TNamedList], Generic[_TNamedKeys, _TNamedList]):

1652-1652: Improved parameter type annotation

The fromdict parameter in the Namedlist.__init__ method now has proper type annotation, improving clarity about expected input types.


1756-1756: Enhanced method return type annotation

The items() method now has an explicit return type annotation, which improves code clarity and IDE support.


1870-1871: Appropriate class renaming

Renaming the class from Resources to ResourceList is a good change as it more accurately describes its purpose and avoids confusion with the new Resources class in resources.py.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (2)
src/snakemake/resources.py (2)

589-594: 🛠️ Refactor suggestion

Use isinstance() instead of type comparison.

Direct type comparison should be replaced with isinstance() for more robust type checking.

-        if not isinstance(self_val, type(other_val)):
+        if not isinstance(self_val, type(other_val)) or not isinstance(other_val, type(self_val)):
            raise ResourceConstraintError(self_val, other_val)

275-282: 🛠️ Refactor suggestion

Fix mutable default arguments.

Using mutable default arguments (empty lists or dictionaries) can lead to unexpected behavior when the function is called multiple times.

    @classmethod
    def _merge_resource_dict(
        cls,
        resources: List[Dict[str, str | int]],
-        skip: Optional[List[str]] = None,
-        methods: Optional[Dict[str, Callable[[List[int]], int]]] = None,
+        skip: Optional[List[str]] = None,
+        methods: Optional[Dict[str, Callable[[List[int]], int]]] = None,
        default_method: Callable[[List[int]], int] = max,
    ):
        skip = skip or []
        methods = methods or {}
🧹 Nitpick comments (11)
src/snakemake/resources.py (11)

29-29: Remove unused imports.

The imports get_function_params and overwrite_function_params are not used in this file.

-from snakemake.common import get_function_params, get_input_function_aux_params, mb_to_mib, mib_to_mb, overwrite_function_params
+from snakemake.common import get_input_function_aux_params, mb_to_mib, mib_to_mb
🧰 Tools
🪛 Ruff (0.8.2)

29-29: snakemake.common.get_function_params imported but unused

Remove unused import

(F401)


29-29: snakemake.common.overwrite_function_params imported but unused

Remove unused import

(F401)


471-477: Good class documentation with spelling error.

The class documentation clearly explains the purpose and behavior of the Resource class, but contains a spelling error in line 477.

-    ensures a resource has been computed before its value is retreived.
+    ensures a resource has been computed before its value is retrieved.
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 477-477:
retreived ==> retrieved

🪛 GitHub Actions: Codespell

[error] 477-477: codespell: 'retreived' is a misspelling. Correct spelling is 'retrieved'.


480-484: Fix spelling error in documentation.

There's a spelling error in the documentation explaining the class behavior.

-    will be intepreted as a human readable resource and converted to an integer of
+    will be interpreted as a human readable resource and converted to an integer of
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 482-482:
intepreted ==> interpreted


529-534: Simplify code with ternary operator.

This if-else block can be simplified using a ternary operator.

-    def __repr__(self):
-        if self.is_evaluable():
-            value = "function(...)"
-        else:
-            value = self._value
-        return f'Resource("{self.name}", {value})'
+    def __repr__(self):
+        value = "function(...)" if self.is_evaluable() else self._value
+        return f'Resource("{self.name}", {value})'
🧰 Tools
🪛 Ruff (0.8.2)

529-532: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


564-564: Fix spelling error in documentation.

There's a spelling error in the documentation for the evaluate method.

-        This method ensures the returned resource is correctly returned in a
-        new ``Resource`` class, validated, and standardized. Resources should thus
-        only be evaulated via this method.
+        This method ensures the returned resource is correctly returned in a
+        new ``Resource`` class, validated, and standardized. Resources should thus
+        only be evaluated via this method.
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 564-564:
evaulated ==> evaluated


680-680: Fix spelling error in documentation.

There's a spelling error in the documentation for the from_cli_expression method.

-        value:
-            python expression to be evaulated
+        value:
+            python expression to be evaluated
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 680-680:
evaulated ==> evaluated


748-759: Add exception chaining with 'from' clause.

When raising a new exception from a caught exception, use the from clause to preserve the traceback.

-            raise WorkflowError(
+            raise WorkflowError(
                "Failed to evaluate resources value "
                f"'{val}'.\n"
                "    String arguments may need additional "
                "quoting. E.g.: --default-resources "
                "\"tmpdir='/home/user/tmp'\" or "
                "--set-resources \"somerule:someresource='--nice=100'\". "
                "This also holds for setting resources inside of a profile, where "
                "you might have to enclose them in single and double quotes, "
                "i.e. someresource: \"'--nice=100'\".",
-                e,
+                e,
+            ) from e
🧰 Tools
🪛 Ruff (0.8.2)

748-759: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


900-902: Fix spelling error in documentation.

There's a spelling error in the documentation for the parse method.

-        """Parse a series of CLI-style string resource assigments.
+        """Parse a series of CLI-style string resource assignments.
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 901-901:
assigments ==> assignments


220-220: Add exception chaining with 'from' clause.

When raising a new exception from a caught exception, use the from clause to preserve the traceback.

-            except ResourceError as err:
-                raise ResourceInsufficiencyError(additive_resources, err.args[0])
+            except ResourceError as err:
+                raise ResourceInsufficiencyError(additive_resources, err.args[0]) from err
🧰 Tools
🪛 Ruff (0.8.2)

220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


850-850: Add exception chaining with 'from' clause.

When raising a new exception from a caught exception, use the from clause to preserve the traceback.

-        except ValueError:
-            raise ValueError("Resources have to be defined as name=value pairs.")
+        except ValueError as e:
+            raise ValueError("Resources have to be defined as name=value pairs.") from e
🧰 Tools
🪛 Ruff (0.8.2)

850-850: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


864-864: Add exception chaining with 'from' clause.

When raising a new exception from a caught exception, use the from clause to preserve the traceback.

-        except KeyError:
-            raise ValueError(f"Unexpected mode for DefaultResources: {mode}")
+        except KeyError as e:
+            raise ValueError(f"Unexpected mode for DefaultResources: {mode}") from e
🧰 Tools
🪛 Ruff (0.8.2)

864-864: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4a09b5d and 7f75879.

📒 Files selected for processing (3)
  • src/snakemake/io.py (8 hunks)
  • src/snakemake/resources.py (14 hunks)
  • tests/common.py (6 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/common.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • src/snakemake/io.py
  • src/snakemake/resources.py
🪛 Ruff (0.8.2)
src/snakemake/resources.py

29-29: snakemake.common.get_function_params imported but unused

Remove unused import

(F401)


29-29: snakemake.common.overwrite_function_params imported but unused

Remove unused import

(F401)


220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


529-532: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


748-759: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


850-850: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


864-864: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py

[failure] 477-477:
retreived ==> retrieved


[failure] 482-482:
intepreted ==> interpreted


[failure] 564-564:
evaulated ==> evaluated


[failure] 680-680:
evaulated ==> evaluated


[failure] 901-901:
assigments ==> assignments

🪛 GitHub Actions: Codespell
src/snakemake/resources.py

[error] 477-477: codespell: 'retreived' is a misspelling. Correct spelling is 'retrieved'.

⏰ Context from checks skipped due to timeout of 90000ms (31)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (7, windows-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (6, windows-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (1, windows-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py311)
  • GitHub Check: apidocs
🔇 Additional comments (11)
src/snakemake/io.py (7)

1-1: Good use of future annotations import.

Adding from __future__ import annotations enables postponed evaluation of type annotations, which is a modern Python practice that helps with circular references in type hints.


31-37: Appropriate type imports added.

The additional imports for Generic, Iterator, and Tuple types support the enhanced type safety introduced in this file.


1636-1640: Improved type variable definitions with clear documentation.

The modification of _TNamedList to remove its bound to Namedlist allows for more flexible usage, and the addition of _TNamedKeys enhances type safety for self-returning methods. Both have clear docstrings explaining their purpose.


1643-1643: Enhanced type safety for Namedlist class.

Updating the Namedlist class to inherit from list[_TNamedList] and Generic[_TNamedKeys, _TNamedList] improves type checking and enables better IDE support.


1652-1652: Improved parameter typing.

Specifying the type of fromdict as Optional[Dict[_TNamedKeys, _TNamedList]] clarifies the expected structure of the input parameter.


1756-1756: Enhanced method return type annotation.

Adding the return type annotation Iterator[Tuple[_TNamedKeys, _TNamedList]] to the items method improves type safety and code clarity.


1870-1871: Appropriate class renaming.

Renaming Resources to ResourceList and specifying its inheritance as Namedlist[str, str | int] helps avoid confusion with the new Resources class in resources.py, while clearly indicating its purpose and structure.

src/snakemake/resources.py (4)

1-26: Good use of type hints with comprehensive imports.

The import structure is well-organized, using future annotations and a thorough set of typing imports that support the implementation of type-safe resource handling.


430-457: Well-implemented conversion functions with proper error handling.

The helper functions evaluable_from_mb_to_mib and evaluable_from_mib_to_mb provide clear and type-safe conversions between different memory unit formats, with appropriate error handling for invalid types.


773-787: Well-designed Resource container with clear documentation.

The Resources class provides a standardized container for resources with excellent handling of suffixed and unsuffixed versions of the same size resource. The docstring clearly explains its purpose and initialization methods.


1010-1035: Good implementation of resource scopes with validation.

The ResourceScopes class effectively defines and enforces valid scopes for resources, with appropriate validation and normalization of size resources.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (14)
src/snakemake/io.py (1)

1636-1638: Consider converting the standalone string literal to a comment or docstring.
If the string on line 1637 is intended as documentation (rather than just an unused literal), consider using a code comment or adding it to a docstring block for clarity.

+# This type variable is for self-returning methods on Namedlist-deriving classes.
tests/tests.py (1)

835-836: Optional debug line can be removed or documented.
Line 836 is commented out. If it is no longer needed, you may remove it to keep the codebase clean, or add a short explanation.

-    default_resources=(["mem_mb=0"]),
-    # cleanup=False
+    default_resources=(["mem_mb=0"]),
src/snakemake/resources.py (12)

29-29: Remove unused imports.

The imports get_function_params and overwrite_function_params appear to be unused in this file.

-from snakemake.common import get_function_params, get_input_function_aux_params, mb_to_mib, mib_to_mb, overwrite_function_params
+from snakemake.common import get_input_function_aux_params, mb_to_mib, mib_to_mb
🧰 Tools
🪛 Ruff (0.8.2)

29-29: snakemake.common.get_function_params imported but unused

Remove unused import

(F401)


29-29: snakemake.common.overwrite_function_params imported but unused

Remove unused import

(F401)


220-220: Use explicit exception chaining.

When raising exceptions inside an except block, you should use raise ... from err to maintain the exception chain for better debugging.

-            raise ResourceInsufficiencyError(additive_resources, err.args[0])
+            raise ResourceInsufficiencyError(additive_resources, err.args[0]) from err
🧰 Tools
🪛 Ruff (0.8.2)

220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


479-479: Fix typo in docstring.

There's a spelling error in the documentation.

-    ensures a resource has been computed before its value is retreived.
+    ensures a resource has been computed before its value is retrieved.
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 479-479:
retreived ==> retrieved

🪛 GitHub Actions: Codespell

[error] 479-479: codespell: 'retreived' is a misspelling. Suggested correction: 'retrieved'.


484-484: Fix typo in docstring.

There's a spelling error in the documentation.

-    will be intepreted as a human readable resource and converted to an integer of
+    will be interpreted as a human readable resource and converted to an integer of
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 484-484:
intepreted ==> interpreted


531-534: Use ternary operator for conciseness.

Consider using a ternary operator to improve code readability and conciseness.

-        if self.is_evaluable():
-            value = "function(...)"
-        else:
-            value = self._value
+        value = "function(...)" if self.is_evaluable() else self._value
🧰 Tools
🪛 Ruff (0.8.2)

531-534: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


566-566: Fix typo in docstring.

There's a spelling error in the documentation.

-        This method ensures the returned resource is correctly returned in a
-        new ``Resource`` class, validated, and standardized. Resources should thus
-        only be evaulated via this method.
+        This method ensures the returned resource is correctly returned in a
+        new ``Resource`` class, validated, and standardized. Resources should thus
+        only be evaluated via this method.
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 566-566:
evaulated ==> evaluated


590-596: Improve type comparison logic.

The current type comparison isn't symmetric. Consider using a more clear approach to check if types match.

-        if not isinstance(self_val, type(other_val)):
+        if type(self_val) != type(other_val):
            raise ResourceConstraintError(self_val, other_val)
        if isinstance(self_val, int):
            assert isinstance(other_val, int)
            return Resource(self.name, min(other_val, self_val))
        return self

682-682: Fix typo in docstring.

There's a spelling error in the documentation.

-            python expression to be evaulated
+            python expression to be evaluated
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 682-682:
evaulated ==> evaluated


750-761: Use explicit exception chaining.

When raising exceptions inside an except block, you should use raise ... from e to maintain the exception chain for better debugging.

             raise WorkflowError(
                 "Failed to evaluate resources value "
                 f"'{val}'.\n"
                 "    String arguments may need additional "
                 "quoting. E.g.: --default-resources "
                 "\"tmpdir='/home/user/tmp'\" or "
                 "--set-resources \"somerule:someresource='--nice=100'\". "
                 "This also holds for setting resources inside of a profile, where "
                 "you might have to enclose them in single and double quotes, "
                 "i.e. someresource: \"'--nice=100'\".",
-                e,
+                e) from e
🧰 Tools
🪛 Ruff (0.8.2)

750-761: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


852-852: Use explicit exception chaining.

When raising exceptions inside an except block, you should use raise ... from None to explicitly suppress the original exception while maintaining clarity.

-            raise ValueError("Resources have to be defined as name=value pairs.")
+            raise ValueError("Resources have to be defined as name=value pairs.") from None
🧰 Tools
🪛 Ruff (0.8.2)

852-852: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


866-866: Use explicit exception chaining.

When raising exceptions inside an except block, you should use raise ... from None to explicitly suppress the original exception while maintaining clarity.

-            raise ValueError(f"Unexpected mode for DefaultResources: {mode}")
+            raise ValueError(f"Unexpected mode for DefaultResources: {mode}") from None
🧰 Tools
🪛 Ruff (0.8.2)

866-866: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


903-903: Fix typo in docstring.

There's a spelling error in the documentation.

-        """Parse a series of CLI-style string resource assigments.
+        """Parse a series of CLI-style string resource assignments.
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 903-903:
assigments ==> assignments

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7f75879 and f0bf3ef.

📒 Files selected for processing (3)
  • src/snakemake/io.py (8 hunks)
  • src/snakemake/resources.py (14 hunks)
  • tests/tests.py (18 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • src/snakemake/io.py
  • tests/tests.py
  • src/snakemake/resources.py
🪛 Ruff (0.8.2)
tests/tests.py

1095-1095: Ambiguous variable name: l

(E741)


1095-1095: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

src/snakemake/resources.py

29-29: snakemake.common.get_function_params imported but unused

Remove unused import

(F401)


29-29: snakemake.common.overwrite_function_params imported but unused

Remove unused import

(F401)


220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


531-534: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


750-761: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


852-852: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


866-866: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py

[failure] 479-479:
retreived ==> retrieved


[failure] 484-484:
intepreted ==> interpreted


[failure] 566-566:
evaulated ==> evaluated


[failure] 682-682:
evaulated ==> evaluated


[failure] 903-903:
assigments ==> assignments

🪛 GitHub Actions: Codespell
src/snakemake/resources.py

[error] 479-479: codespell: 'retreived' is a misspelling. Suggested correction: 'retrieved'.

⏰ Context from checks skipped due to timeout of 90000ms (31)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (7, windows-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (6, windows-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (1, windows-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py311)
  • GitHub Check: apidocs
🔇 Additional comments (20)
src/snakemake/io.py (5)

1-1: Good use of postponed evaluation of type hints.
This import is standard when forward references or advanced typing features are needed and can help avoid circular dependency issues.


31-36: No issues with newly added type-related imports.
These cleanly align with Python's typing best practices (e.g., Generic, Iterator, Tuple) and are used consistently in the updated code below.


1643-1643: Use of generics in Namedlist.
Defining Namedlist(list, Generic[_TNamedKeys, _TNamedList]) appropriately leverages Python’s generic system to enforce type-checked usage of _TNamedKeys & _TNamedList.


1652-1652: Clearer parameter type annotation in the constructor.
Specifying fromdict: Optional[Dict[_TNamedKeys, _TNamedList]] = None is concise and precise, ensuring better type safety when converting dicts into Namedlist.


1757-1757: Proper return type annotation for the items method.
Returning an Iterator[Tuple[_TNamedKeys, _TNamedList]] is consistent with your generics usage and helps IDEs and type checkers infer usage more accurately.

tests/tests.py (11)

15-19: Validated imports for new exception and resource classes.
These imports (e.g. ResourceDuplicationError, GroupResources, Resources) ensure code references are well-defined and consistent with the new resource-management functionality.


814-814: Simplified assertion logic in assert_resources.
Directly comparing dictionaries (assert resources == expected_resources) is both more readable and robust than partial dictionary checks.


842-851: Verifying resource dictionary with extra fields.
These newly added lines (mem, mem_mib, disk, disk_mib, etc.) appear consistent with the new test scenario validating resource translation. Everything looks correct for the numeric checks.


854-879: test_resources_can_be_provided_in_mib thoroughly exercises Mebibyte-based resource usage.
The test sets {"mem_mib": 57221} and confirms successful translation to standard MB values. This scenario is well-covered, ensuring correctness in resource conversion.


881-905: test_global_resources_can_be_human_readable validates intuitive user inputs.
Verifying “60 Gb” for memory helps ensure the parser correctly handles human-readable formats. The test coverage is strong, and the approach is straightforward.


908-927: Correctly preventing mixed resource prefixes (e.g. mem vs mem_mb).
These lines confirm that specifying both unprefixed and suffixed resource variants together raises a ResourceDuplicationError, reinforcing resource definition integrity.


930-941: Ensuring human-readable resources are invalid when a suffix is required.
This covers the scenario where mem_mb must be numeric only, preventing strings like “60 GB.” Good job raising WorkflowError for invalid usage.


943-955: Rejecting human-readable formats in declarations with suffixed resource keys.
This test enforces consistency for resources such as mem_mb, ensuring that only strict integer or numeric inputs are allowed.


967-969: Precise usage of Resources.parse with expression flags.
Allowing expressions (e.g. mem_mb=0) in default_resources is consistent with the rest of the refactoring. This usage is correct and well-tested.


1409-1414: test_default_resources_humanreadable ensures fractional Gigabyte values are rounded properly.
Great job testing real-world memory designations like '1 Gb' and '0.001 TB'. This confirms proper scaling.


1419-1425: test_default_resources_mebibytes ensures partial Mebibyte values are correctly handled.
The test checks for non-integer inputs (e.g., 953.67) and verifies rounding or conversion logic. Nicely done.

src/snakemake/resources.py (4)

276-282: Good handling of mutable default arguments.

The code correctly handles mutable default arguments by using None and initializing them inside the function body.


775-1007: Great implementation of the Resources class.

The Resources class is well-designed with a clear interface and strong type safety. It effectively replaces the old DefaultResources class and provides a standardized container for working with resources.

Key strengths:

  • Good handling of resource normalization
  • Clear implementation of the Mapping interface
  • Helpful class methods for different initialization scenarios
  • Proper type annotations throughout
🧰 Tools
🪛 Ruff (0.8.2)

852-852: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


866-866: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors

[failure] 903-903:
assigments ==> assignments


473-773: Well-structured Resource class with comprehensive functionality.

The Resource class is well-designed with clear responsibilities:

  • Standardizing resource representation
  • Handling human-readable resource parsing
  • Supporting resource evaluation and constraints
  • Providing conversion between different resource units (MB/MiB)

This implementation will significantly improve resource handling consistency across the codebase.

🧰 Tools
🪛 Ruff (0.8.2)

531-534: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


750-761: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors

[failure] 479-479:
retreived ==> retrieved


[failure] 484-484:
intepreted ==> interpreted


[failure] 566-566:
evaulated ==> evaluated


[failure] 682-682:
evaulated ==> evaluated

🪛 GitHub Actions: Codespell

[error] 479-479: codespell: 'retreived' is a misspelling. Suggested correction: 'retrieved'.


1012-1151: Good enhancement of the ResourceScopes class.

The ResourceScopes class has been effectively enhanced to inherit from Dict, making it more Pythonic and easier to use. The implementation of methods for checking and normalizing resource scopes is clean and well-documented.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
src/snakemake/resources.py (1)

599-600: 🛠️ Refactor suggestion

Improve type checking logic for resource constraint

The current implementation has a potential logical issue in the type comparison. It's comparing types with direct equality which may not work correctly for inheritance hierarchies.

    self_val = self.value
-    if not isinstance(self_val, type(other_val)):
+    if not isinstance(self_val, type(other_val)) or not isinstance(other_val, type(self_val)):
        raise ResourceConstraintError(self_val, other_val)
🧹 Nitpick comments (12)
src/snakemake/workflow.py (3)

1755-1756: More efficient resource copying.

Using the copy() method instead of deepcopy() is more efficient and appropriate here as Resources objects don't contain nested mutable objects that would require deep copying.


1761-1774: Improved error handling for thread resources.

The try-except block with specific ResourceValidationError handling provides better error reporting. However, the raised exception could be improved.

Apply the following change to improve the exception chain:

        try:
            if overwrite_threads is not None:
                rule.resources["_cores"] = overwrite_threads
            elif ruleinfo.threads is not None:
                rule.resources["_cores"] = ruleinfo.threads
            else:
                rule.resources["_cores"] = 1
        except ResourceValidationError:
-            raise RuleException(
+            raise RuleException(
                "Threads value has to be an integer, float, or a callable.",
                rule=rule,
-            )
+            ) from None

This makes it clear that you're explicitly replacing the original exception with a more user-friendly one.

🧰 Tools
🪛 Ruff (0.8.2)

1770-1773: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1779-1788: Enhanced resource validation with Resources.from_mapping.

The use of Resources.from_mapping provides structured validation, but the exception handling should be improved.

Apply the following change to improve the exception chain:

            try:
                resources = Resources.from_mapping(resources)
            except KeyError as err:
                raise RuleException(
                    "Resources values have to be integers, strings, or callables "
                    f"(functions): type({err.args[0]} : {err.args[1]}) == "
                    f"{type(err.args[1])}",
                    rule=rule,
-                )
+                ) from err

This preserves the original exception in the chain, making debugging easier.

🧰 Tools
🪛 Ruff (0.8.2)

1782-1787: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/cli.py (1)

211-229: Consider raising from the original exception for clarity.

When wrapping an exception at line 227, you can preserve the original traceback by adding from e. For example:

-                raise ValueError(f"{errmsg} Cause: {e}")
+                raise ValueError(f"{errmsg} Cause: {e}") from e

This helps distinguish errors in exception handling from the underlying exception.

🧰 Tools
🪛 Ruff (0.8.2)

220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


227-227: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

tests/tests.py (1)

1097-1097: Use a clearer variable name and avoid not l == "\n".

Reading lines with:

lines = [line for line in f if line != "\n"]

would improve readability and satisfy the lint suggestion (!= instead of not ... ==), while also avoiding the single-letter variable name l.

🧰 Tools
🪛 Ruff (0.8.2)

1097-1097: Ambiguous variable name: l

(E741)


1097-1097: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

src/snakemake/resources.py (7)

4-4: Remove unused imports

There are several imports that are declared but never used in the code:

  • dataclasses.dataclass (line 4)
  • snakemake.common.get_function_params (line 30)
  • snakemake.common.overwrite_function_params (line 34)
-from dataclasses import dataclass
 
 from humanfriendly import InvalidTimespan, InvalidSize, parse_size, parse_timespan
 from snakemake.common import (
-    get_function_params,
     get_input_function_aux_params,
     mb_to_mib,
     mib_to_mb,
-    overwrite_function_params,
 )

Also applies to: 30-30, 34-34

🧰 Tools
🪛 Ruff (0.8.2)

4-4: dataclasses.dataclass imported but unused

Remove unused import: dataclasses.dataclass

(F401)


220-220: Use raise ... from err for proper exception chaining

When raising an exception inside an except block, it's a best practice to use raise ... from err to maintain the exception chain for better debugging.

            try:
                layers = cls._get_layers(
                    int_resources, list(sorted_constraints.values()), sortby
                )
            except ResourceError as err:
-                raise ResourceInsufficiencyError(additive_resources, err.args[0])
+                raise ResourceInsufficiencyError(additive_resources, err.args[0]) from err
🧰 Tools
🪛 Ruff (0.8.2)

220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


479-479: Fix spelling errors in docstrings

There are several spelling errors in the docstrings:

    """Standardized representation of a resource name and value.

    In addition to ``int`` and ``str``, the standard resource types, ``Resource`` can
    store ``None``, signifying an unset resource, and a callable, signifying a resource
-    to be calculated in the context of a rule. Via the ``.value`` property, ``Resource``
-    ensures a resource has been computed before its value is retreived.
+    to be calculated in the context of a rule. Via the ``.value`` property, ``Resource``
+    ensures a resource has been computed before its value is retrieved.

    The class additionally handles input validation and standardization.
    ``float``s get rounded into ``int``. If the resource name corresponds to a size
    resource ("disk" and "mem") or a timespan resource ("runtime"), strings values
-    will be intepreted as a human readable resource and converted to an integer of
+    will be interpreted as a human readable resource and converted to an integer of
    appropriate denomination (Mb for size, hr for timespan).
    """Evaluate the resource with given args.

    This method ensures the returned resource is correctly returned in a
    new ``Resource`` class, validated, and standardized. Resources should thus
-    only be evaulated via this method.
+    only be evaluated via this method.
    """
    """Create a new evaluable resource based on a python expression.

    Threads can optionally be included in the expression environment

    Arguments
    =========
    name:
        name of the resource
    value:
-        python expression to be evaulated
+        python expression to be evaluated
    with_threads_arg: boolean
        If True, include ``threads`` as an argument in the returned function
    """
-    """Parse a series of CLI-style string resource assigments.
+    """Parse a series of CLI-style string resource assignments.

    Arguments
    =========
    exprs: list of str
        Each expr is in the form RESOURCE=VALUE, where RESOURCE is a valid python
        identifier, and VALUE is a number, a quoted string, or a valid python
        expression.

Also applies to: 484-484, 572-572, 690-690, 911-911

🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 479-479:
retreived ==> retrieved


420-422: Add more context to resource error message

The error message when a resource constraint is exceeded could be more informative to help users understand what went wrong.

                error_text = [
                    f"\t{res}: {amount}/{constraint}"
                    for res, amount, constraint in too_high
                ]
-                raise ResourceError("\n".join(error_text))
+                raise ResourceError(f"Resource requirement exceeds constraint:\n" + "\n".join(error_text))

537-541: Simplify condition with ternary operator

This can be simplified using a ternary operator for better readability.

    def __repr__(self):
-        if self.is_evaluable():
-            value = "function(...)"
-        else:
-            value = self._value
+        value = "function(...)" if self.is_evaluable() else self._value
        return f'Resource("{self.name}", {value})'
🧰 Tools
🪛 Ruff (0.8.2)

537-540: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


758-769: Use raise ... from e for proper exception chaining

When raising a new exception inside an except block, using raise ... from e maintains the exception chain for better debugging.

            raise WorkflowError(
                "Failed to evaluate resources value "
                f"'{val}'.\n"
                "    String arguments may need additional "
                "quoting. E.g.: --default-resources "
                "\"tmpdir='/home/user/tmp'\" or "
                "--set-resources \"somerule:someresource='--nice=100'\". "
                "This also holds for setting resources inside of a profile, where "
                "you might have to enclose them in single and double quotes, "
                "i.e. someresource: \"'--nice=100'\".",
-                e,
+                e,
+            ) from e
🧰 Tools
🪛 Ruff (0.8.2)

758-769: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


725-770: Ensure consistent error handling for resource evaluation

The cli_evaluator method catches exceptions during resource evaluation but uses different approaches for different error types. It would be better to have a consistent approach to exception handling.

        try:
            value = eval(
                val,
                {
                    "input": input,
                    "attempt": attempt,
                    "system_tmpdir": tempfile.gettempdir(),
                    **({"threads": threads} if threads is not None else {}),
                },
            )
        # Triggers for string arguments like n1-standard-4
        except NameError:
            return val
        except Exception as e:
            try:
                # check if resource is parsable as human friendly (given the correct
                # name and formatted value). If it is, we return the parsed value to
                # save a step later.
                if name in HumanFriendlyResources:
                    return Resource._parse_human_friendly(name, val)
-            except (InvalidSize, InvalidTimespan):
+            except (InvalidSize, InvalidTimespan) as parse_err:
                pass
            if is_file_not_found_error(e, input):
                return TBDString()

            raise WorkflowError(
                "Failed to evaluate resources value "
                f"'{val}'.\n"
                "    String arguments may need additional "
                "quoting. E.g.: --default-resources "
                "\"tmpdir='/home/user/tmp'\" or "
                "--set-resources \"somerule:someresource='--nice=100'\". "
                "This also holds for setting resources inside of a profile, where "
                "you might have to enclose them in single and double quotes, "
                "i.e. someresource: \"'--nice=100'\".",
                e,
-            )
+            ) from e
🧰 Tools
🪛 Ruff (0.8.2)

758-769: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f0bf3ef and 873ada9.

⛔ Files ignored due to path filters (1)
  • tests/test_set_resources_complex/expected-results/test.out is excluded by !**/*.out
📒 Files selected for processing (6)
  • src/snakemake/cli.py (7 hunks)
  • src/snakemake/resources.py (13 hunks)
  • src/snakemake/settings/types.py (2 hunks)
  • src/snakemake/spawn_jobs.py (2 hunks)
  • src/snakemake/workflow.py (7 hunks)
  • tests/tests.py (19 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • src/snakemake/workflow.py
  • src/snakemake/spawn_jobs.py
  • src/snakemake/settings/types.py
  • src/snakemake/resources.py
  • tests/tests.py
  • src/snakemake/cli.py
🪛 Ruff (0.8.2)
src/snakemake/workflow.py

1770-1773: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1782-1787: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/resources.py

4-4: dataclasses.dataclass imported but unused

Remove unused import: dataclasses.dataclass

(F401)


30-30: snakemake.common.get_function_params imported but unused

Remove unused import

(F401)


34-34: snakemake.common.overwrite_function_params imported but unused

Remove unused import

(F401)


220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


537-540: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


758-769: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


860-860: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


874-874: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

tests/tests.py

1097-1097: Ambiguous variable name: l

(E741)


1097-1097: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

src/snakemake/cli.py

220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


227-227: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py

[failure] 479-479:
retreived ==> retrieved


[failure] 484-484:
intepreted ==> interpreted


[failure] 572-572:
evaulated ==> evaluated


[failure] 690-690:
evaulated ==> evaluated


[failure] 911-911:
assigments ==> assignments

⏰ Context from checks skipped due to timeout of 90000ms (31)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (7, windows-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (6, windows-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (1, windows-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py312)
  • GitHub Check: apidocs
  • GitHub Check: tests (1, ubuntu-latest, py311)
🔇 Additional comments (34)
src/snakemake/spawn_jobs.py (2)

95-96: Resource value access simplified.

The code now directly accesses the raw attribute of the resource value instead of using a helper function. This change is aligned with the new Resources class that standardizes resource handling.


105-106: Using consistent resource value access pattern.

The same pattern is now applied to threads values, maintaining consistency in how resource values are accessed.

src/snakemake/workflow.py (8)

58-61: Added ResourceValidationError to import statement.

The import of ResourceValidationError supports the new resource handling system that provides better error messaging for resource validation issues.


118-118: Updated resource import.

Importing Resources from snakemake.resources reflects the centralization of resource handling, which improves code organization.


166-166: Simplified global resources initialization.

Direct assignment of self.resource_settings.resources removes redundant code and makes the implementation cleaner.


568-568: Updated _cores property to access value attribute.

Accessing the .value attribute of resources is consistent with the new Resources class approach.


572-572: Updated nodes property to access value attribute.

Similar to the _cores property, this change maintains consistency with the new resource handling approach.


629-629: Added return type annotation to get_rule method.

Adding the return type annotation improves code clarity and assists with type checking.


1775-1778: Validation for unnamed resources.

The code correctly validates that resources must be named, providing a clear error message.


1790-1791: Properly applying overwritten resources.

The code correctly updates rule resources with any overwritten values provided via command-line arguments.

src/snakemake/cli.py (6)

12-12: No issues found with the new typing import.


42-44: Importing Resource and Resources looks good.

This aligns with the project's refactoring from DefaultResources to the new resource classes. No concerns here.


156-181: Refactored parse_set_resources function is well-structured.

  • The logic for splitting and grouping resource assignments into a dictionary is clear.
  • Error handling with raise ValueError(errmsg) from err preserves the original exception context, which is best practice.

No further issues spotted; everything appears correct.


516-517: Initialization of resource arguments is a good change.

Using Resources() as the default instead of a bare dictionary clarifies the nature of these constraints.


596-597: Approach for parsing default resources with expressions looks fine.

The Resources.parser_factory call is consistent with the new resource design.


969-971: Configured default resources with parsing logic.

Specifying default expressions (like "mem_mb=0") is appropriate if that aligns with your workflow’s requirements.

src/snakemake/settings/types.py (2)

33-33: Imported Resource and Resources successfully.

This replacement is consistent with the ongoing resource-management refactoring.


345-353: Enhanced ResourceSettings class initialization.

  • Using field(default_factory=Resources) and storing more structured resource data is a clear improvement over raw mappings.
  • The fallback to Resources.default("bare") at line 353 is a straightforward way to ensure consistent defaults.

No red flags identified.

tests/tests.py (9)

15-18: Importing new exceptions and resource classes for testing.

Bringing in WorkflowError, ResourceDuplicationError, and GroupResources, Resources is aligned with the refactored codebase.


816-817: assert_resources helper is concise and clear.

Simply checking for equality is sufficient for these tests.


829-854: Refined test logic for test_group_jobs_resources.

  • Setting default_resources=(["mem_mb=0"]) ensures explicit control over resource defaults in the test.
  • The final assert_resources(...) calls validate mem/disk conversions thoroughly.

All changes look good.


856-881: test_resources_can_be_provided_in_mib is well-structured.

  • Verifies that specifying memory in MiB is processed correctly (mem_mib=57221mem_mb=60000).
  • Good coverage of the MiB→MB conversion logic.

No concerns here.


883-908: test_global_resources_can_be_human_readable coverage is excellent.

This confirms that human-readable resource strings (e.g., "60 Gb") are converted properly.


910-930: test_resources_cannot_provide_prefixed_and_unprefixed_together: well-defined negative test.

Ensuring that mixing e.g. mem with mem_mb raises ResourceDuplicationError prevents ambiguous resource definitions.


932-943: test_suffixed_resource_constraints_cannot_be_human_readable: solid negative test.

Correctly expecting a WorkflowError for “60 GB” with _mb constraints avoids confusion of scale units.


945-957: test_suffixed_resource_declarations_cannot_be_human_readable is consistent with the preceding logic.

Validates that you can’t define a _mb resource with “Gb” strings. Great catch to avoid mismatch.


1076-1101: test_resource_prefix_does_not_affect_scope_overwrite: thorough scenario.

  • Validates correct overrides for resource scopes like “global_res” and “mem” vs. “mem_mb”.
  • Confirms that matching resource prefixes does not inadvertently skip the scope rules.

All looks fine.

🧰 Tools
🪛 Ruff (0.8.2)

1097-1097: Ambiguous variable name: l

(E741)


1097-1097: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

src/snakemake/resources.py (7)

273-282: LGTM: Proper handling of mutable default arguments

The code correctly avoids using mutable default arguments by setting defaults to None and initializing them in the function body.


430-443: LGTM: Well-implemented unit conversion functions

The implementation of unit conversion functions for evaluable resources (MB to MiB and vice versa) is well done. The functions properly handle type checking and conversions.

Also applies to: 445-457


607-632: LGTM: Resource unit conversion methods

The to_mib and to_mb methods are well designed with proper error handling and support for converting both evaluated and unevaluated resources.

Also applies to: 633-658


783-1015: LGTM: Well-designed Resources class

The Resources class is well-designed as a standardized container for resource objects. It properly handles unit conversion, resource normalization, and provides convenient methods for resource manipulation.

The implementation of the Mapping interface methods and additional utility methods is thorough and follows best practices.

🧰 Tools
🪛 Ruff (0.8.2)

860-860: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


874-874: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors

[failure] 911-911:
assigments ==> assignments


1020-1154: LGTM: Improved ResourceScopes class

The ResourceScopes class is well-implemented with clear semantics for resource scope handling. The methods for checking resource scopes (is_local, is_global, is_excluded) and the properties for retrieving scoped resources are well-designed.


54-253: LGTM: Comprehensive implementation of resource handling for group jobs

The GroupResources.basic_layered method is a well-documented and robust implementation for calculating resource requirements for group jobs. The implementation handles different resource types, constraints, and optimization strategies.

The code is well-structured with clear separation of concerns between resource parsing, constraint checking, and layer optimization.

🧰 Tools
🪛 Ruff (0.8.2)

220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


824-825:

Details

❓ Verification inconclusive

Verify error handling for resource duplication

The code raises a ResourceDuplicationError when multiple variants of the same sized resource are specified. Verify that this error is properly caught and handled in the calling code.


🏁 Script executed:

#!/bin/bash
# Check if ResourceDuplicationError is handled in the caller
rg -A 5 "ResourceDuplicationError" --include="*.py" --exclude="resources.py"

Length of output: 516


Below is a revised shell script command that correctly restricts the search to Python files (excluding the file under review) and looks for explicit exception handling of ResourceDuplicationError. This will help verify whether callers are catching the error in a try/except block.

Please execute the following command and share the output so we can confirm the intended error handling is in place:


#!/bin/bash
# Check for explicit try/except blocks catching ResourceDuplicationError in the codebase (excluding resources.py)
rg -A 5 "except.*ResourceDuplicationError" -g "*.py" -g "!resources.py"

Once you’ve run this script, we can proceed further with the verification.

**

@fgvieira fgvieira mentioned this pull request May 25, 2025
2 tasks
@pvandyken pvandyken force-pushed the fix/human-friendly-units-groups branch from 873ada9 to c13583a Compare May 29, 2025 16:28
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🔭 Outside diff range comments (4)
src/snakemake/common/__init__.py (1)

106-112: ⚠️ Potential issue

Consider preserving the exception chain for better debugging.

While the change to NestedCoroutineError is correct, consider using raise ... from e to preserve the exception chain as suggested by static analysis:

-        raise NestedCoroutineError(
+        raise NestedCoroutineError(
             "Error running coroutine in event loop. Snakemake currently does not "
             "support being executed from an already running event loop. "
             "If you run Snakemake e.g. from a Jupyter notebook, make sure to spawn a "
             "separate process for Snakemake.",
-            e,
-        )
+        ) from e
🧰 Tools
🪛 Ruff (0.11.9)

106-112: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/workflow.py (2)

783-795: 🛠️ Refactor suggestion

Fix exception chaining for better error diagnostics.

The exception handling pattern is correct, but should use proper exception chaining for better debugging.

Apply this diff to improve exception chaining:

-            except ResourceValidationError:
+            except ResourceValidationError as e:
                 raise RuleException(
                     "Threads value has to be an integer, float, or a callable.",
                     rule=rule,
-                )
+                ) from e

797-810: 🛠️ Refactor suggestion

Fix exception chaining and approve the resource mapping approach.

The use of Resources.from_mapping() is the correct approach for converting rule resources. However, the exception handling needs proper chaining.

Apply this diff to improve exception chaining:

                 except KeyError as err:
                     raise RuleException(
                         "Resources values have to be integers, strings, or callables "
                         f"(functions): type({err.args[0]} : {err.args[1]}) == "
                         f"{type(err.args[1])}",
                         rule=rule,
-                    )
+                    ) from err
tests/tests.py (1)

2516-2516: ⚠️ Potential issue

Update to use the new Resources.parse pattern for consistency.

This line still uses the old DefaultResources class instead of the new Resources.parse pattern used throughout the rest of the file.

Apply this diff to maintain consistency with the resource system refactoring:

-        default_resources=DefaultResources(["mem_mb=120"]),
+        default_resources=Resources.parse(
+            ["mem_mb=120"], defaults="full", allow_expressions=True
+        ),
🧰 Tools
🪛 Ruff (0.11.9)

2516-2516: Undefined name DefaultResources

(F821)

🧹 Nitpick comments (8)
src/snakemake/rules.py (1)

1089-1089: Fix spelling error.

"inital" should be "initial" in the comment.

-                # dag construction routine are run as independent asynchronous loops.
+                # dag construction routine are run as independent asynchronous loops.

Wait, let me check the actual line again. The error is "inital" -> "initial":

-                # If input.size_mb is run in an input method, the loops will be nested
+                # If input.size_mb is run in an input method, the loops will be nested

Actually, looking at line 1089-1091:

-                # Need to catch this because both input.size_mb and the inital
+                # Need to catch this because both input.size_mb and the initial
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 1089-1089:
inital ==> initial

src/snakemake/cli.py (1)

226-236: Good enhancement but improve exception handling.

The wrapper function support is well-implemented and maintains backward compatibility. However, consider improving exception chaining as suggested by static analysis.

Apply this diff to improve exception handling:

             if wrapper is None:
-                raise ValueError(errmsg)
+                raise ValueError(errmsg) from None
             if wrapper is not None:
                 try:
                     assignments[key] = wrapper(value)
                 except Exception as e:
-                    raise ValueError(f"{errmsg} Cause: {e}")
+                    raise ValueError(f"{errmsg} Cause: {e}") from e
🧰 Tools
🪛 Ruff (0.11.9)

227-227: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


234-234: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/jobs.py (1)

37-37: LGTM on import updates, but clean up unused import.

The import changes align well with the resource management refactoring, introducing the new ResourceList, GroupResources, Resources, and ResourceInsufficiencyError classes.

However, there's an unused import that should be removed:

-from snakemake.io import (
-    _IOFile,
-    IOFile,
-    ResourceList,
-    is_callable,
-    Wildcards,
-    is_flagged,
-    get_flag_value,
-    wait_for_files,
-)
+from snakemake.io import (
+    _IOFile,
+    IOFile,
+    ResourceList,
+    Wildcards,
+    is_flagged,
+    get_flag_value,
+    wait_for_files,
+)

Also applies to: 45-45, 51-51

tests/tests.py (2)

1146-1146: Fix code quality issues: ambiguous variable name and comparison style.

Two improvements needed:

  1. The variable name l is ambiguous and can be confused with 1 or I
  2. Use != instead of not == for better readability
-lines = [l for l in f.readlines() if not l == "\n"]
+lines = [line for line in f.readlines() if line != "\n"]
🧰 Tools
🪛 Ruff (0.11.9)

1146-1146: Ambiguous variable name: l

(E741)


1146-1146: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)


1146-1147: Improve variable naming and comparison style.

The static analysis tools correctly identified some style issues:

  1. The variable name l is ambiguous and could be confused with 1 or I
  2. The comparison not l == "\n" is less clear than using !=

Apply this diff to improve readability:

-        lines = [l for l in f.readlines() if not l == "\n"]
+        lines = [line for line in f.readlines() if line != "\n"]
🧰 Tools
🪛 Ruff (0.11.9)

1146-1146: Ambiguous variable name: l

(E741)


1146-1146: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

src/snakemake/resources.py (3)

472-494: Fix spelling errors in docstring.

Multiple spelling errors need correction in the class docstring.

    """Standardized representation of a resource name and value.

    In addition to ``int`` and ``str``, the standard resource types, ``Resource`` can
    store ``None``, signifying an unset resource, and a callable, signifying a resource
-    to be calculated in the context of a rule. Via the ``.value`` property, ``Resource``
-    ensures a resource has been computed before its value is retreived.
+    to be calculated in the context of a rule. Via the ``.value`` property, ``Resource``
+    ensures a resource has been computed before its value is retrieved.

    The class additionally handles input validation and standardization.
    ``float``s get rounded into ``int``. If the resource name corresponds to a size
    resource ("disk" and "mem") or a timespan resource ("runtime"), strings values
-    will be intepreted as a human readable resource and converted to an integer of
+    will be interpreted as a human readable resource and converted to an integer of
    appropriate denomination (Mb for size, hr for timespan).

    Arguments
    =========
    name: str
        Name of the resource (note that "disk", "mem", and "runtime" has special
        behaviour, as described above).
    value: int, str, Callable, None
        The concrete or unevaluated value
    """
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 478-478:
retreived ==> retrieved


[failure] 483-483:
intepreted ==> interpreted

🪛 GitHub Actions: Codespell

[error] 478-478: codespell: 'retreived' is a misspelling of 'retrieved'


681-698: Fix spelling error in docstring.

    @classmethod
    def from_cli_expression(
        cls, name: str, value: str, *, with_threads_arg: bool = True
    ):
        """Create a new evaluable resource based on a python expression.

        Threads can optionally be included in the expression environment

        Arguments
        =========
        name:
            name of the resource
        value:
-            python expression to be evaulated
+            python expression to be evaluated
        with_threads_arg: boolean
            If True, include ``threads`` as an argument in the returned function
        """
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 694-694:
evaulated ==> evaluated


1-1166: Outstanding refactoring of the resource management system!

This comprehensive rewrite significantly improves Snakemake's resource handling:

  • Type safety: Strong typing throughout with proper validation
  • Human-friendly: Support for parsing "60 Gb", "2 hours" etc.
  • Unit conversion: Seamless handling of MB/MiB conversions
  • Error handling: Clear, specific exceptions with helpful messages
  • Clean architecture: Well-separated concerns between Resource, Resources, and ResourceScopes

The implementation successfully addresses all the bugs mentioned in the PR objectives.

Would you like me to help with any documentation updates or additional test cases for edge cases?

🧰 Tools
🪛 Ruff (0.11.9)

4-4: dataclasses.dataclass imported but unused

Remove unused import: dataclasses.dataclass

(F401)


219-219: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


536-539: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


765-776: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


867-867: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


881-881: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors

[failure] 478-478:
retreived ==> retrieved


[failure] 483-483:
intepreted ==> interpreted


[failure] 694-694:
evaulated ==> evaluated


[failure] 918-918:
assigments ==> assignments

🪛 GitHub Actions: Codespell

[error] 478-478: codespell: 'retreived' is a misspelling of 'retrieved'

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 873ada9 and c13583a.

📒 Files selected for processing (21)
  • src/snakemake/api.py (2 hunks)
  • src/snakemake/cli.py (7 hunks)
  • src/snakemake/common/__init__.py (5 hunks)
  • src/snakemake/dag.py (2 hunks)
  • src/snakemake/exceptions.py (3 hunks)
  • src/snakemake/io/__init__.py (8 hunks)
  • src/snakemake/jobs.py (5 hunks)
  • src/snakemake/logging.py (2 hunks)
  • src/snakemake/resources.py (14 hunks)
  • src/snakemake/rules.py (6 hunks)
  • src/snakemake/scheduler.py (5 hunks)
  • src/snakemake/script/__init__.py (1 hunks)
  • src/snakemake/settings/types.py (2 hunks)
  • src/snakemake/spawn_jobs.py (2 hunks)
  • src/snakemake/workflow.py (7 hunks)
  • tests/common.py (6 hunks)
  • tests/test_all_temp/Snakefile (0 hunks)
  • tests/test_default_resources/Snakefile (1 hunks)
  • tests/test_github_issue3271/Snakefile_should_fail (1 hunks)
  • tests/test_group_jobs_resources/Snakefile (2 hunks)
  • tests/tests.py (20 hunks)
💤 Files with no reviewable changes (1)
  • tests/test_all_temp/Snakefile
🚧 Files skipped from review as they are similar to previous changes (11)
  • src/snakemake/spawn_jobs.py
  • src/snakemake/api.py
  • src/snakemake/script/init.py
  • tests/test_default_resources/Snakefile
  • src/snakemake/logging.py
  • tests/test_group_jobs_resources/Snakefile
  • src/snakemake/scheduler.py
  • src/snakemake/dag.py
  • src/snakemake/exceptions.py
  • src/snakemake/settings/types.py
  • tests/common.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest type annotations for functions that are defined inside of functions or methods. Do not suggest type annotation of the `s...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • src/snakemake/io/__init__.py
  • tests/tests.py
  • src/snakemake/resources.py
  • src/snakemake/cli.py
  • src/snakemake/common/__init__.py
  • src/snakemake/jobs.py
  • src/snakemake/rules.py
  • src/snakemake/workflow.py
🧬 Code Graph Analysis (7)
src/snakemake/io/__init__.py (2)
src/snakemake/gui.py (1)
  • index (84-95)
src/snakemake/resources.py (1)
  • items (999-1000)
tests/tests.py (3)
src/snakemake/exceptions.py (1)
  • ResourceDuplicationError (640-647)
src/snakemake/resources.py (5)
  • GroupResources (53-426)
  • Resources (790-1021)
  • from_mapping (979-985)
  • parse (910-976)
  • defaults (1068-1069)
tests/common.py (1)
  • run (152-511)
src/snakemake/resources.py (6)
src/snakemake/common/__init__.py (3)
  • get_input_function_aux_params (308-320)
  • mb_to_mib (64-65)
  • mib_to_mb (68-69)
src/snakemake/exceptions.py (6)
  • ResourceConstraintError (636-637)
  • ResourceError (599-600)
  • ResourceInsufficiencyError (603-622)
  • ResourceScopesException (625-629)
  • ResourceValidationError (632-633)
  • is_file_not_found_error (675-682)
src/snakemake/common/tbdstring.py (1)
  • TBDString (3-96)
src/snakemake/io/__init__.py (11)
  • AnnotatedString (944-964)
  • Wildcards (1942-1943)
  • items (1836-1838)
  • get (1881-1886)
  • inner (1498-1518)
  • is_callable (87-87)
  • is_callable (325-326)
  • is_callable (955-956)
  • is_callable (1152-1157)
  • keys (1869-1870)
  • update (1811-1817)
src/snakemake/jobs.py (22)
  • jobs (1346-1347)
  • jobs (1350-1351)
  • Job (177-1287)
  • resources (466-483)
  • resources (1491-1505)
  • name (1259-1260)
  • name (1642-1646)
  • group (446-449)
  • group (452-453)
  • wildcards (314-315)
  • wildcards (318-319)
  • input (322-323)
  • input (326-327)
  • input (1512-1517)
  • attempt (456-457)
  • attempt (460-463)
  • attempt (1713-1714)
  • attempt (1717-1722)
  • threads (408-409)
  • threads (1706-1707)
  • is_local (1014-1022)
  • is_local (1662-1663)
src/snakemake/workflow.py (8)
  • resources (2151-2156)
  • name (2235-2240)
  • group (2165-2170)
  • func (1630-1636)
  • input (1983-1988)
  • attempt (383-387)
  • threads (2130-2135)
  • is_local (676-680)
src/snakemake/cli.py (2)
src/snakemake/resources.py (9)
  • Resource (472-787)
  • Resources (790-1021)
  • from_cli_expression (682-727)
  • args (855-860)
  • parse (910-976)
  • value (550-560)
  • default (870-888)
  • parser_factory (891-907)
  • defaults (1068-1069)
src/snakemake/common/__init__.py (1)
  • parse_key_value_arg (72-79)
src/snakemake/common/__init__.py (1)
src/snakemake/exceptions.py (1)
  • NestedCoroutineError (550-551)
src/snakemake/rules.py (4)
src/snakemake/io/__init__.py (3)
  • ResourceList (1950-1951)
  • items (1836-1838)
  • get (1881-1886)
src/snakemake/jobs.py (15)
  • resources (466-483)
  • resources (1491-1505)
  • threads (408-409)
  • threads (1706-1707)
  • name (1259-1260)
  • name (1642-1646)
  • wildcards (314-315)
  • wildcards (318-319)
  • input (322-323)
  • input (326-327)
  • input (1512-1517)
  • attempt (456-457)
  • attempt (460-463)
  • attempt (1713-1714)
  • attempt (1717-1722)
src/snakemake/resources.py (9)
  • Resource (472-787)
  • Resources (790-1021)
  • evaluate (566-583)
  • constrain (585-608)
  • value (550-560)
  • items (999-1000)
  • is_evaluable (562-564)
  • get (1017-1021)
  • to_mib (610-635)
src/snakemake/exceptions.py (4)
  • ResourceConstraintError (636-637)
  • ResourceValidationError (632-633)
  • NestedCoroutineError (550-551)
  • InputFunctionException (250-262)
src/snakemake/workflow.py (4)
src/snakemake/exceptions.py (1)
  • ResourceValidationError (632-633)
src/snakemake/jobs.py (6)
  • resources (466-483)
  • resources (1491-1505)
  • name (1259-1260)
  • name (1642-1646)
  • threads (408-409)
  • threads (1706-1707)
src/snakemake/resources.py (8)
  • Resources (790-1021)
  • value (550-560)
  • copy (1013-1015)
  • get (1017-1021)
  • args (855-860)
  • from_mapping (979-985)
  • update (987-991)
  • update (1162-1165)
src/snakemake/rules.py (3)
  • name (137-138)
  • name (141-142)
  • Rule (83-1327)
🪛 Ruff (0.11.9)
tests/tests.py

1146-1146: Ambiguous variable name: l

(E741)


1146-1146: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

src/snakemake/resources.py

4-4: dataclasses.dataclass imported but unused

Remove unused import: dataclasses.dataclass

(F401)


219-219: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


536-539: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


765-776: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


867-867: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


881-881: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/cli.py

227-227: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


234-234: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/common/__init__.py

106-112: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/jobs.py

38-38: snakemake.io.is_callable imported but unused

Remove unused import: snakemake.io.is_callable

(F401)

src/snakemake/rules.py

1125-1131: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/workflow.py

1792-1795: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1804-1809: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py

[failure] 478-478:
retreived ==> retrieved


[failure] 483-483:
intepreted ==> interpreted


[failure] 694-694:
evaulated ==> evaluated


[failure] 918-918:
assigments ==> assignments

src/snakemake/rules.py

[failure] 1089-1089:
inital ==> initial

🪛 GitHub Actions: Codespell
src/snakemake/resources.py

[error] 478-478: codespell: 'retreived' is a misspelling of 'retrieved'

⏰ Context from checks skipped due to timeout of 90000ms (30)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (9, macos-latest, py312)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (8, macos-latest, py312)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (7, macos-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (6, macos-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (5, macos-latest, py312)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py311)
🔇 Additional comments (53)
tests/test_github_issue3271/Snakefile_should_fail (1)

7-7: LGTM! The resource specification correctly tests the new resource evaluation system.

The change from attempt * 1000 to input.size_mb aligns with the PR's objective of standardizing resource handling and tests the enhanced resource evaluation capabilities.

src/snakemake/common/__init__.py (4)

15-15: Imports correctly updated for type annotations and new exception handling.

The addition of Any and Mapping supports the new type annotations, and the switch to NestedCoroutineError aligns with the new exception hierarchy.

Also applies to: 24-24


64-64: Good addition of type annotation for clarity.

The explicit int type annotation improves code clarity and type safety.


68-70: Correct implementation of MiB to MB conversion.

The function correctly implements the inverse of mb_to_mib using the appropriate conversion factor and floor operation to ensure integer results.


308-310: Well-designed type annotations improve function clarity.

The use of Mapping instead of Dict correctly indicates that the function doesn't modify the parameters, and the return type properly reflects the filtered mapping.

src/snakemake/io/__init__.py (8)

1-1: Good addition of future annotations for modern Python compatibility.

The from __future__ import annotations enables postponed evaluation of annotations, improving import performance and allowing forward references.


32-33: Necessary typing imports for enhanced type safety.

The addition of Generic, Iterator, and Tuple supports the generic typing improvements in the Namedlist class.

Also applies to: 37-37


716-723: Excellent enhancement of Namedlist with generic typing.

The introduction of type variables _TNamedList and _TNamedKeys along with making Namedlist generic significantly improves type safety and IDE support. The docstrings clearly explain the purpose of each type variable.


732-732: Type annotation correctly uses generic parameters.

The fromdict parameter type now properly enforces type consistency with the generic Namedlist parameters.


836-836: Return type properly reflects the generic key-value pairs.

The items() method return type now correctly indicates it yields tuples of the generic key and value types.


1458-1458: Good practice using collections.abc.Iterable for abstract types.

Using the fully qualified name from collections.abc is the recommended approach for abstract base classes.


1618-1618: Clear type annotation improves code readability.

The explicit List[str] type annotation for names makes the code's intent clearer.


950-950:

Details

❓ Verification inconclusive

Important rename from Resources to ResourceList to avoid naming conflicts.

This rename is crucial for the resource management refactoring, as it distinguishes this class from the new Resources class in resources.py.


🏁 Script executed:

#!/bin/bash
# Verify that all references to the old Resources class have been updated
echo "Checking for remaining references to io.Resources that might need updating..."
rg -A 3 'from snakemake\.io import.*Resources' --type py
rg -A 3 'io\.Resources' --type py

Length of output: 259


Ensure complete rename from Resources to ResourceList

I ran the provided ripgrep check and didn’t find any remaining imports or usages of the old io.Resources class. However, please manually verify that:

  • All tests, examples, and documentation have been updated to use ResourceList.
  • There are no stray references (e.g., in comments, docstrings, or auxiliary scripts) to the old Resources name.
src/snakemake/workflow.py (7)

61-61: LGTM! Import changes align with resource system refactor.

The new imports for ResourceValidationError and Resources properly support the modernized resource handling architecture.

Also applies to: 119-119


167-169: LGTM! Direct assignment reflects the new typed resource architecture.

The direct assignment of self.resource_settings.resources eliminates the need for manual copying and leverages the new Resources container type.


587-587: LGTM! Proper use of the new Resource API.

Accessing the .value attribute correctly extracts the underlying value from the Resource objects while maintaining type safety and consistency with the new resource architecture.

Also applies to: 591-591


648-648: LGTM! Good addition of type annotation.

The return type annotation -> Rule improves type safety and developer experience.


777-779: LGTM! Cleaner resource copying with the new API.

Using Resources.copy() is more elegant and safer than manual dictionary copying.


812-813: LGTM! Clean resource updates with the new container.

The direct update from overwrite_resources leverages the Resources container's update capabilities effectively.


928-928: LGTM! Simplified resource updates with improved container handling.

The direct update with global_resources indicates that the new Resources container properly handles edge cases internally, making the code cleaner and more robust.

src/snakemake/rules.py (6)

28-28: Import changes look good.

The new imports properly support the resource system refactor by bringing in the modernized Resource, Resources, ResourceList classes and related exceptions. The type checking import for Workflow enables proper type annotations.

Also applies to: 49-54, 57-57, 76-77


92-92: Type annotations improve code clarity and type safety.

The type annotations for workflow and resources attributes are accurate and help with IDE support and static analysis.

Also applies to: 104-104


1065-1096: Well-designed resource evaluation function.

The new evaluate function provides comprehensive exception handling for resource evaluation, including proper handling of ResourceValidationError, NestedCoroutineError, and file-not-found scenarios. The logic for skipping evaluation and returning TBDString for async loop conflicts is sound.

🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 1089-1089:
inital ==> initial


1098-1105: Proper handling of the cores resource.

The explicit handling of the _cores resource first, with constraint by max_threads and type validation, ensures thread count is properly managed before evaluating other resources that may depend on it.


1107-1140: Comprehensive resource processing with good error handling.

The main resource evaluation loop correctly:

  • Skips non-evaluable resources with None values
  • Evaluates callable resources with thread context
  • Applies global resource constraints with proper type checking
  • Handles special mem and disk resources to set their _mb and _mib variants

The logic preserves the original functionality while leveraging the new resource system.

🧰 Tools
🪛 Ruff (0.11.9)

1125-1131: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1141-1141: Return type change aligns with modernized resource system.

Returning ResourceList instead of Resources is consistent with the broader refactor to separate resource containers from resource definitions, as evidenced by the usage in jobs.py.

src/snakemake/cli.py (6)

43-44: LGTM! Import changes align with resource standardization.

The replacement of DefaultResources and ParsedResource with the new Resource and Resources classes is consistent with the PR objectives to centralize resource handling.


108-118: Well-executed refactoring to use new Resource system.

The function now properly integrates with the new Resource class while maintaining the same external interface. The wrapper approach cleanly converts thread specifications to Resource objects.


163-189: Excellent refactoring that improves parsing logic and error handling.

The new implementation properly groups resource assignments by rule and leverages the standardized Resources.parse method. The error handling is improved with proper exception chaining, making debugging easier for users.


523-524: Proper integration with new Resources system.

The argument parser configuration correctly uses the new Resources class for both default values and parsing, maintaining consistency with the standardized resource management.


602-604: Excellent configuration for default resources.

The parser configuration appropriately enables full default presets and Python expression support, which aligns perfectly with the documented default resource examples like mem_mb=min(max(2*input.size_mb, 1000), 8000).


108-189: Comprehensive and well-executed resource system migration.

This refactoring successfully migrates CLI argument parsing to use the new unified Resource and Resources classes while maintaining backward compatibility and improving error handling. The changes are consistent throughout and align perfectly with the PR objectives to standardize resource handling in Snakemake.

Also applies to: 523-524, 602-604

🧰 Tools
🪛 Ruff (0.11.9)

149-149: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/jobs.py (4)

96-113: Excellent refactoring of scheduler resource filtering logic.

The changes improve resource handling in several ways:

  1. Better evaluation logic: Replacing the callable check with TBDString check is more explicit about which resources cannot be evaluated yet.

  2. Cleaner scope filtering: Using workflow.resource_scopes.is_local(k) is more maintainable than checking callable status.

  3. Type safety: Returning ResourceList(fromdict=res_dict) instead of a plain dict provides better type consistency.

These changes align well with the standardized resource handling goals of this PR.


466-466: Good type annotation improvement.

Adding the explicit return type annotation -> ResourceList improves type safety and documentation. This is consistent with the refactoring to standardize on ResourceList for resource representations.


1322-1322: Good type annotation for constructor parameter.

Adding the type annotation global_resources: Resources improves code documentation and type safety. This makes it clear what type of resource object is expected for the global resource constraints.


1501-1505: Excellent error handling improvements.

The changes enhance error handling in several ways:

  1. More specific exception handling: Catching ResourceInsufficiencyError in addition to WorkflowError provides better granularity for resource-related errors.

  2. Proper exception chaining: Using from err preserves the original exception context for better debugging.

  3. Informative error messages: Including the group ID in the error message helps with troubleshooting.

  4. Type consistency: Returning ResourceList(fromdict=self._resources) maintains consistent return types.

These improvements align well with the goal of standardizing resource handling across the codebase.

tests/tests.py (14)

14-18: LGTM! Import changes align with the new resource management system.

The addition of WorkflowError, ResourceDuplicationError, and Resources imports is appropriate for testing the new resource handling functionality.


670-671: Good addition: Test for thread overwriting via CLI.

This test validates that threads can be dynamically set using CLI expressions, which is an important feature of the new resource system.


864-865: Good simplification of the assert_resources function.

Direct dictionary comparison is cleaner and more straightforward than the previous implementation.


878-1293: Excellent comprehensive test coverage for the new resource management system!

The test modifications and additions thoroughly validate:

  • Resource normalization (mem_mb → mem, disk_mb → disk)
  • Human-readable resource parsing
  • Mebibyte unit support with proper conversion
  • Prevention of duplicate resource specifications using ResourceDuplicationError
  • Resource scope management (local/global)

The tests are well-structured and cover important edge cases.

🧰 Tools
🪛 Ruff (0.11.9)

1146-1146: Ambiguous variable name: l

(E741)


1146-1146: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)


14-18: LGTM! Import changes align with the resource refactoring.

The new imports for WorkflowError, ResourceDuplicationError, GroupResources, and Resources properly support the refactored resource handling system described in the PR objectives.


670-671: LGTM! Test function looks correct.

The test function call is properly structured and appears to test thread overwriting functionality correctly.


864-866: LGTM! Clean helper function for resource assertions.

The assert_resources function provides a clear and simple way to verify resource dictionaries in tests.


878-903: LGTM! Test properly validates resource normalization.

The updated test correctly demonstrates the new resource normalization feature where resources like mem_mb are expanded to include both the base form (mem) and suffixed variants (mem_mb, mem_mib). The migration from DefaultResources to Resources.parse with defaults="full" and allow_expressions=True is appropriate.


905-930: LGTM! Test validates mebibyte resource specification.

This new test properly validates that resources can be specified in mebibyte format (mem_mib) and are correctly converted to their megabyte equivalents. The assertion confirms that all resource variants are properly computed.


932-957: LGTM! Test validates human-readable resource format.

This test properly validates that global resources can be specified in human-readable format ("60 Gb") and are correctly converted to numeric values (60000 MB). This demonstrates an important usability improvement in the new resource system.


959-978: LGTM! Test validates resource duplication error handling.

This parametrized test properly validates that the new resource system detects and prevents conflicting resource specifications (e.g., providing both mem and mem_mb simultaneously). The use of ResourceDuplicationError as the expected failure type is appropriate.


981-1005: LGTM! Tests validate suffixed resource format constraints.

These tests properly validate that suffixed resources (like mem_mb) cannot use human-readable formats, while unsuffixed base resources (like mem) can. This enforces the design constraint that human-readable formats are only allowed for base resource names.


1018-1021: LGTM! Consistent migration to Resources.parse pattern.

The migration from DefaultResources to Resources.parse(["mem_mb=0"], defaults="full", allow_expressions=True) is consistently applied across multiple test functions. This aligns with the resource system refactoring described in the PR objectives.

Also applies to: 1048-1050, 1087-1089, 1113-1115, 1140-1142, 1166-1168, 1187-1189, 1209-1211, 1232-1234, 1259-1261, 1290-1292


1467-1484: LGTM! Tests validate new default resource format capabilities.

These new tests properly validate that default resources can be specified in both human-readable format ("1 Gb") and mebibyte format ("953.67 MiB"), extending the new resource capabilities to default resource specifications.

src/snakemake/resources.py (3)

1027-1166: Excellent implementation of ResourceScopes with proper validation!

The class properly:

  • Validates resource scopes (local/global/excluded)
  • Normalizes size resources to prevent duplicates
  • Provides convenient helper methods for scope checking
  • Handles suffixed resources correctly

602-608: ⚠️ Potential issue

Use isinstance() for type checking instead of direct type comparison.

Direct type comparison is fragile and doesn't work with subclasses. Use isinstance() for proper type checking.

self_val = self.value
-if not isinstance(self_val, type(other_val)):
+if type(self_val) != type(other_val):
    raise ResourceConstraintError(self_val, other_val)
if isinstance(self_val, int):
    assert isinstance(other_val, int)
    return Resource(self.name, min(other_val, self_val))
return self

Likely an incorrect or invalid review comment.


272-281: ⚠️ Potential issue

Fix mutable default arguments to prevent unexpected behavior.

Using mutable objects as default arguments can lead to unexpected behavior when the function is called multiple times.

def _merge_resource_dict(
    cls,
    resources: List[Dict[str, str | int]],
-    skip: Optional[List[str]] = None,
-    methods: Optional[Dict[str, Callable[[List[int]], int]]] = None,
+    skip: List[str] | None = None,
+    methods: Dict[str, Callable[[List[int]], int]] | None = None,
    default_method: Callable[[List[int]], int] = max,
):
    skip = skip or []
    methods = methods or {}

Likely an incorrect or invalid review comment.

@pvandyken pvandyken force-pushed the fix/human-friendly-units-groups branch from c13583a to 287e661 Compare July 29, 2025 20:27
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

♻️ Duplicate comments (6)
src/snakemake/rules.py (1)

1129-1135: Exception handling follows codebase conventions.

The exception handling here correctly avoids exception chaining, which is consistent with the codebase's backward compatibility requirements. The error message provides sufficient context for debugging without needing the original exception chain.

src/snakemake/resources.py (5)

4-4: Remove unused import.

The dataclass import is not used anywhere in this file.

-from dataclasses import dataclass

218-219: Exception chaining issue flagged by static analysis.

Based on the retrieved learnings, the Snakemake codebase avoids exception chaining for backward compatibility reasons. However, static analysis is flagging this pattern.


765-776: Exception chaining flagged by static analysis.

Based on the retrieved learnings, the Snakemake codebase avoids exception chaining for backward compatibility reasons. The current pattern is consistent with the codebase standards.


867-867: Exception chaining flagged by static analysis.

Based on the retrieved learnings, the Snakemake codebase avoids exception chaining for backward compatibility reasons.


881-881: Exception chaining flagged by static analysis.

Based on the retrieved learnings, the Snakemake codebase avoids exception chaining for backward compatibility reasons.

🧹 Nitpick comments (3)
src/snakemake/rules.py (1)

1093-1093: Fix spelling error in comment.

-                # Need to catch this because both input.size_mb and the inital
+                # Need to catch this because both input.size_mb and the initial
src/snakemake/jobs.py (1)

38-38: Remove unused import.

The is_callable import is not used in this file and should be removed to clean up the imports.

-    is_callable,
src/snakemake/cli.py (1)

218-237: Good generalization but address exception chaining.

The function has been well-generalized to use a wrapper function instead of a fallback parameter, providing more flexibility for value processing.

However, please address the exception chaining issue flagged by static analysis:

                 try:
                     assignments[key] = wrapper(value)
-                except Exception as e:
-                    raise ValueError(f"{errmsg} Cause: {e}")
+                except Exception as e:
+                    raise ValueError(f"{errmsg} Cause: {e}") from e

This preserves the original exception context for better debugging.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c13583a and 287e661.

📒 Files selected for processing (20)
  • src/snakemake/api.py (2 hunks)
  • src/snakemake/cli.py (7 hunks)
  • src/snakemake/common/__init__.py (5 hunks)
  • src/snakemake/dag.py (2 hunks)
  • src/snakemake/exceptions.py (3 hunks)
  • src/snakemake/io/__init__.py (8 hunks)
  • src/snakemake/jobs.py (5 hunks)
  • src/snakemake/logging.py (2 hunks)
  • src/snakemake/resources.py (14 hunks)
  • src/snakemake/rules.py (6 hunks)
  • src/snakemake/scheduler.py (5 hunks)
  • src/snakemake/script/__init__.py (1 hunks)
  • src/snakemake/settings/types.py (2 hunks)
  • src/snakemake/spawn_jobs.py (2 hunks)
  • src/snakemake/workflow.py (7 hunks)
  • tests/common.py (6 hunks)
  • tests/test_default_resources/Snakefile (1 hunks)
  • tests/test_github_issue3271/Snakefile_should_fail (1 hunks)
  • tests/test_group_jobs_resources/Snakefile (2 hunks)
  • tests/tests.py (19 hunks)
✅ Files skipped from review due to trivial changes (1)
  • src/snakemake/api.py
🚧 Files skipped from review as they are similar to previous changes (12)
  • tests/test_default_resources/Snakefile
  • tests/test_group_jobs_resources/Snakefile
  • src/snakemake/spawn_jobs.py
  • src/snakemake/logging.py
  • tests/test_github_issue3271/Snakefile_should_fail
  • src/snakemake/script/init.py
  • src/snakemake/scheduler.py
  • src/snakemake/dag.py
  • src/snakemake/settings/types.py
  • tests/common.py
  • src/snakemake/exceptions.py
  • src/snakemake/io/init.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py

⚙️ CodeRabbit Configuration File

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

Files:

  • tests/tests.py
  • src/snakemake/resources.py
  • src/snakemake/cli.py
  • src/snakemake/common/__init__.py
  • src/snakemake/jobs.py
  • src/snakemake/rules.py
  • src/snakemake/workflow.py
🧠 Learnings (8)
📓 Common learnings
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
tests/tests.py (4)

Learnt from: johanneskoester
PR: #3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in tests/test_wrapper/Snakefile, are for testing purposes and do not require updates to the project documentation.

Learnt from: johanneskoester
PR: #3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.

Learnt from: johanneskoester
PR: #3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the tests/test_cores_cluster/qsub script is a dummy script for testing, and input validation and error handling are not required in such scripts.

Learnt from: mbhall88
PR: #3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.

src/snakemake/resources.py (15)

Learnt from: johanneskoester
PR: #3026
File: snakemake/assets/init.py:0-0
Timestamp: 2024-08-27T20:15:58.699Z
Learning: Do not suggest type annotations for cls arguments in class methods when reviewing code for the Snakemake project.

Learnt from: johanneskoester
PR: #3026
File: snakemake/assets/init.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Do not suggest type annotations for cls arguments in class methods when reviewing code for the Snakemake project.

Learnt from: johanneskoester
PR: #3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In snakemake/dag.py, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.

Learnt from: johanneskoester
PR: #3014
File: snakemake/dag.py:1265-1266
Timestamp: 2024-10-11T07:09:54.544Z
Learning: For the Snakemake project, avoid suggesting trailing commas as they use Black as an autoformatter.

Learnt from: johanneskoester
PR: #3014
File: snakemake/dag.py:1265-1266
Timestamp: 2024-08-13T16:09:09.726Z
Learning: For the Snakemake project, avoid suggesting trailing commas as they use Black as an autoformatter.

Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-08-13T16:22:09.641Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.

Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.

Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the snakemake codebase, exception chaining (using raise ... from ...) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in snakemake codebase.

Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-10-08T17:41:54.542Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.

Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.

Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the snakemake codebase, especially when the cases handled don't make sense.

Learnt from: johanneskoester
PR: #3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.

Learnt from: johanneskoester
PR: #3107
File: snakemake/logging.py:510-515
Timestamp: 2025-01-14T14:04:30.554Z
Learning: In Snakemake's logging handlers, KeyboardInterrupt and SystemExit exceptions should be ignored (pass) rather than re-raised, as these are control flow signals that should not disrupt the logging process. The exception variable in the final except block should be kept for potential debugging purposes.

src/snakemake/cli.py (1)

Learnt from: johanneskoester
PR: #3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the snakemake/cli.py file, ExecutorPluginRegistry() is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.

src/snakemake/common/__init__.py (14)

Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.

Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-08-13T16:22:09.641Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.

Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-08-13T16:07:33.369Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.

Learnt from: johanneskoester
PR: #3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.

Learnt from: johanneskoester
PR: #2985
File: tests/tests.py:2051-2051
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.

Learnt from: johanneskoester
PR: #2985
File: tests/tests.py:2051-2051
Timestamp: 2024-08-13T09:25:24.046Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.

Learnt from: johanneskoester
PR: #3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in tests/test_wrapper/Snakefile, are for testing purposes and do not require updates to the project documentation.

Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Learnt from: johanneskoester
PR: #2927
File: snakemake/ioutils/branch.py:44-51
Timestamp: 2024-12-14T13:13:09.273Z
Learning: Snakemake needs to maintain compatibility with Python 3.7; avoid using syntax or features introduced after Python 3.7.

Learnt from: johanneskoester
PR: #2927
File: snakemake/ioutils/evaluate.py:13-17
Timestamp: 2024-12-14T13:05:08.032Z
Learning: The Snakemake project supports Python 3.7, so code contributions must remain compatible with Python 3.7 and avoid features introduced in later Python versions.

Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the snakemake codebase, exception chaining (using raise ... from ...) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in snakemake codebase.

Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-10-08T17:41:54.542Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.

Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.

src/snakemake/jobs.py (2)

Learnt from: lczech
PR: #3113
File: snakemake/scheduler.py:912-914
Timestamp: 2024-10-04T16:12:18.927Z
Learning: In snakemake/scheduler.py, avoid suggesting the use of asyncio.gather in the jobs_rewards method due to overhead concerns and the need for immediate results.

Learnt from: johanneskoester
PR: #3600
File: src/snakemake/jobs.py:960-964
Timestamp: 2025-05-23T09:40:24.474Z
Learning: In the cleanup method of the Job class in src/snakemake/jobs.py, files in the to_remove list should be formatted with fmt_iofile without specifying as_output=True or as_input=True parameters, as these files should be displayed as generic files rather than specifically as output files.

src/snakemake/rules.py (8)

Learnt from: johanneskoester
PR: #2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In snakemake/ioutils/evaluate.py, the use of eval() with globals() is acceptable because it's called by the user under consideration of the implications.

Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the snakemake codebase, exception chaining (using raise ... from ...) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in snakemake codebase.

Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-10-08T17:41:54.542Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.

Learnt from: johanneskoester
PR: #3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as raise ... from e.

Learnt from: johanneskoester
PR: #3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.

Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the snakemake codebase, especially when the cases handled don't make sense.

src/snakemake/workflow.py (6)

Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the snakemake codebase, exception chaining (using raise ... from ...) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in snakemake codebase.

Learnt from: johanneskoester
PR: #3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the snakemake codebase, especially when the cases handled don't make sense.

Learnt from: johanneskoester
PR: #3132
File: snakemake/rules.py:1105-1105
Timestamp: 2024-10-11T12:49:08.705Z
Learning: In snakemake/rules.py, within the Rule class, self.basedir is a custom class (not a pathlib.Path object) that has a join() method. Therefore, using self.basedir.join(conda_env) is appropriate in this context.

Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Learnt from: johanneskoester
PR: #3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In snakemake/deployment/conda.py, the exception handling in the _get_version method is intentional. The method raises a WorkflowError when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Learnt from: kdm9
PR: #3562
File: src/snakemake/checkpoints.py:90-90
Timestamp: 2025-05-06T01:37:23.382Z
Learning: In Snakemake checkpoints implementation, tracking only the first missing output for each checkpoint is sufficient, because if one output is missing, all outputs for that checkpoint are considered incomplete. This was the behavior before PR #3562 and maintained in the pluralized checkpoints.get() implementation.

🧬 Code Graph Analysis (6)
tests/tests.py (3)
src/snakemake/exceptions.py (1)
  • ResourceDuplicationError (640-647)
src/snakemake/resources.py (4)
  • Resources (790-1021)
  • from_mapping (979-985)
  • parse (910-976)
  • defaults (1068-1069)
tests/common.py (1)
  • run (152-511)
src/snakemake/resources.py (6)
src/snakemake/common/__init__.py (3)
  • get_input_function_aux_params (308-320)
  • mb_to_mib (64-65)
  • mib_to_mb (68-69)
src/snakemake/exceptions.py (7)
  • ResourceConstraintError (636-637)
  • ResourceDuplicationError (640-647)
  • ResourceError (599-600)
  • ResourceInsufficiencyError (603-622)
  • ResourceScopesException (625-629)
  • ResourceValidationError (632-633)
  • is_file_not_found_error (675-682)
src/snakemake/common/tbdstring.py (1)
  • TBDString (3-96)
src/snakemake/io/__init__.py (9)
  • AnnotatedString (933-953)
  • Wildcards (1941-1942)
  • items (1835-1837)
  • get (1880-1885)
  • is_callable (314-315)
  • is_callable (944-945)
  • is_callable (1151-1156)
  • keys (1868-1869)
  • update (1810-1816)
src/snakemake/jobs.py (20)
  • jobs (1346-1347)
  • jobs (1350-1351)
  • Job (177-1287)
  • resources (466-483)
  • resources (1491-1505)
  • name (1259-1260)
  • name (1642-1646)
  • group (446-449)
  • group (452-453)
  • wildcards (314-315)
  • wildcards (318-319)
  • input (322-323)
  • input (326-327)
  • input (1512-1517)
  • attempt (456-457)
  • attempt (460-463)
  • attempt (1713-1714)
  • attempt (1717-1722)
  • is_local (1014-1022)
  • is_local (1662-1663)
src/snakemake/workflow.py (7)
  • resources (2167-2172)
  • name (2251-2256)
  • group (2181-2186)
  • func (1646-1652)
  • input (1999-2004)
  • attempt (390-394)
  • is_local (687-691)
src/snakemake/cli.py (2)
src/snakemake/resources.py (10)
  • Resource (472-787)
  • Resources (790-1021)
  • from_cli_expression (682-727)
  • args (855-860)
  • parse (910-976)
  • items (999-1000)
  • value (550-560)
  • default (870-888)
  • parser_factory (891-907)
  • defaults (1068-1069)
src/snakemake/common/__init__.py (1)
  • parse_key_value_arg (72-79)
src/snakemake/common/__init__.py (1)
src/snakemake/exceptions.py (1)
  • NestedCoroutineError (550-551)
src/snakemake/rules.py (5)
src/snakemake/io/__init__.py (2)
  • ResourceList (1949-1950)
  • items (1835-1837)
src/snakemake/jobs.py (13)
  • resources (466-483)
  • resources (1491-1505)
  • name (1259-1260)
  • name (1642-1646)
  • wildcards (314-315)
  • wildcards (318-319)
  • input (322-323)
  • input (326-327)
  • input (1512-1517)
  • attempt (456-457)
  • attempt (460-463)
  • attempt (1713-1714)
  • attempt (1717-1722)
src/snakemake/resources.py (8)
  • Resource (472-787)
  • Resources (790-1021)
  • evaluate (566-583)
  • constrain (585-608)
  • value (550-560)
  • items (999-1000)
  • is_evaluable (562-564)
  • to_mib (610-635)
src/snakemake/exceptions.py (4)
  • ResourceConstraintError (636-637)
  • ResourceValidationError (632-633)
  • NestedCoroutineError (550-551)
  • InputFunctionException (250-262)
src/snakemake/common/tbdstring.py (1)
  • TBDString (3-96)
src/snakemake/workflow.py (4)
src/snakemake/exceptions.py (2)
  • ResourceValidationError (632-633)
  • RuleException (209-247)
src/snakemake/jobs.py (6)
  • resources (466-483)
  • resources (1491-1505)
  • name (1259-1260)
  • name (1642-1646)
  • threads (408-409)
  • threads (1706-1707)
src/snakemake/resources.py (8)
  • Resources (790-1021)
  • value (550-560)
  • copy (1013-1015)
  • get (1017-1021)
  • args (855-860)
  • from_mapping (979-985)
  • update (987-991)
  • update (1162-1165)
src/snakemake/rules.py (3)
  • name (138-139)
  • name (142-143)
  • Rule (84-1331)
🪛 Ruff (0.12.2)
tests/tests.py

1162-1162: Ambiguous variable name: l

(E741)


1162-1162: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

src/snakemake/resources.py

4-4: dataclasses.dataclass imported but unused

Remove unused import: dataclasses.dataclass

(F401)


219-219: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


536-539: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


765-776: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


867-867: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


881-881: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/cli.py

227-227: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


234-234: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/common/__init__.py

106-112: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/jobs.py

38-38: snakemake.io.is_callable imported but unused

Remove unused import: snakemake.io.is_callable

(F401)

src/snakemake/rules.py

1129-1135: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/workflow.py

1808-1811: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1820-1825: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py

[failure] 918-918:
assigments ==> assignments


[failure] 694-694:
evaulated ==> evaluated


[failure] 483-483:
intepreted ==> interpreted


[failure] 478-478:
retreived ==> retrieved

src/snakemake/rules.py

[failure] 1093-1093:
inital ==> initial

🪛 GitHub Actions: Codespell
src/snakemake/resources.py

[error] 478-478: codespell: 'retreived' is a misspelling; did you mean 'retrieved'?

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (56)
  • GitHub Check: tests (10, ubuntu-latest, py313)
  • GitHub Check: tests (8, ubuntu-latest, py313)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (8, windows-latest, py313)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py313)
  • GitHub Check: tests (10, windows-latest, py313)
  • GitHub Check: tests (9, windows-latest, py313)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (7, windows-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (4, windows-latest, py313)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (7, windows-latest, py313)
  • GitHub Check: tests (6, windows-latest, py313)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (5, windows-latest, py313)
  • GitHub Check: tests (7, ubuntu-latest, py313)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py313)
  • GitHub Check: tests (6, windows-latest, py312)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py313)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py313)
  • GitHub Check: tests (4, ubuntu-latest, py313)
  • GitHub Check: tests (2, windows-latest, py313)
  • GitHub Check: tests (3, windows-latest, py313)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (2, macos-latest, py312)
  • GitHub Check: tests (1, macos-latest, py313)
  • GitHub Check: tests (2, ubuntu-latest, py313)
  • GitHub Check: tests (2, macos-latest, py313)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (3, macos-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py313)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (1, windows-latest, py313)
  • GitHub Check: tests (1, ubuntu-latest, py311)
  • GitHub Check: tests (1, macos-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py312)
  • GitHub Check: tests (1, windows-latest, py312)
  • GitHub Check: apidocs
🔇 Additional comments (38)
src/snakemake/common/__init__.py (6)

15-15: LGTM!

The additional typing imports (Any and Mapping) are appropriately added to support the type annotations introduced in this PR.


24-24: LGTM!

The change to import NestedCoroutineError instead of WorkflowError is appropriate for the more specific exception handling in the async_run function and aligns with the new exception hierarchy.


64-64: LGTM!

The type annotation for the mb parameter is appropriate and improves code clarity by explicitly indicating the expected integer type.


68-70: LGTM!

The new mib_to_mb function appropriately complements the existing mb_to_mib function, using the correct mathematical inverse (floor division) and maintaining consistent type annotations.


106-112: LGTM!

The change to raise NestedCoroutineError instead of WorkflowError provides better exception specificity for async execution contexts while maintaining the same error message and behavior.


308-310: LGTM!

The type annotations accurately describe the function interface: a callable accepting any arguments, a mapping of string keys to any values for parameters, and returning a similar mapping. This improves code clarity and IDE support.

src/snakemake/workflow.py (9)

61-61: LGTM! Import changes align with new resource system.

The additions of ResourceValidationError and Resources imports are consistent with the centralized resource handling approach described in the PR objectives.

Also applies to: 119-119


167-167: LGTM! Correct integration with new Resources class.

The direct assignment reflects that resource_settings.resources is now a Resources object rather than a dictionary, which aligns with the new centralized resource handling system.


598-598: LGTM! Correct adaptation to Resource object structure.

The .value attribute access is consistent with the new Resource class design, where resource values are encapsulated within Resource objects rather than stored as raw values.

Also applies to: 602-602


659-659: LGTM! Good addition of type annotation.

The return type annotation improves code clarity and type safety without changing functionality.


793-795: LGTM! Proper initialization with new Resources class.

The changes correctly use Resources.copy() for copying default resources and Resources() constructor for empty resource initialization, aligning with the new resource management system.


799-811: LGTM! Proper resource validation and error handling.

The code correctly catches ResourceValidationError and converts it to RuleException with an appropriate error message. The exception handling follows snakemake's convention of not using exception chaining for backward compatibility.


813-826: LGTM! Correct resource parsing with improved error handling.

The use of Resources.from_mapping() is the proper way to convert resource mappings in the new system. The error handling correctly catches KeyError and provides informative error messages, following snakemake's exception handling patterns.


828-829: LGTM! Direct update with Resources object.

The direct update is correct since overwrite_resources now contains Resources objects rather than dictionaries, consistent with the new resource management approach.


944-944: LGTM! Correct handover resource handling.

The direct update with global_resources is appropriate since it's now a Resources object that handles resource management internally, eliminating the need for manual filtering of None values.

src/snakemake/rules.py (3)

28-28: LGTM! New imports support the resource refactoring.

The added imports for the new resource management classes (Resource, Resources, ResourceList) and related exceptions are appropriate for the modernized resource handling system.

Also applies to: 50-55, 58-58, 77-78


93-93: LGTM! Type annotations improve code clarity.

The type annotations for workflow and resources attributes enhance type safety and align with the new resource management system.

Also applies to: 105-105


1065-1145: Excellent refactoring of resource evaluation logic.

The rewritten expand_resources method significantly improves resource handling with:

  • Clear separation of concerns via the nested evaluate function
  • Proper exception handling for ResourceValidationError, NestedCoroutineError, and input function exceptions
  • Explicit constraint enforcement with meaningful error messages
  • Automatic conversion of memory/disk resources to _mb and _mib variants
  • Type-safe return of ResourceList instead of plain dictionary

The implementation correctly handles the core resource evaluation flow: evaluate _cores first with thread constraints, then process other resources with global constraint validation.

src/snakemake/jobs.py (4)

96-113: LGTM!

The scheduler resource filtering logic has been correctly updated to work with the new resource system. The iteration over resource items and direct TBDString checking on values is appropriate, and the use of ResourceList constructor aligns with the refactored architecture.


466-466: LGTM!

The explicit return type annotation for ResourceList improves type safety and aligns with the resource system refactor where ResourceList is used for job resource contexts.


1322-1322: LGTM!

The type annotation for global_resources parameter correctly specifies the Resources type, improving type safety and aligning with the refactored resource system.


1501-1505: LGTM!

The enhanced error handling correctly catches the new ResourceInsufficiencyError exception type and provides proper exception chaining with group context. The conversion to ResourceList aligns with the refactored resource architecture.

src/snakemake/cli.py (5)

12-12: LGTM! Import changes align with resource system refactoring.

The addition of Union to typing imports and the replacement of legacy resource classes (DefaultResources, ParsedResource, eval_resource_expression) with the new Resource and Resources classes is consistent with the PR's objective to standardize resource handling.

Also applies to: 43-44


109-118: Excellent integration with the new resource system.

The function now properly uses Resource.from_cli_expression to parse thread specifications, which aligns with the unified resource parsing approach. The wrapper function cleanly encapsulates the resource creation logic and the with_threads_arg=False parameter appropriately indicates this is for thread-specific parsing.


163-189: Excellent simplification and integration with Resources class.

The function has been significantly simplified by delegating the complex parsing logic to Resources.parse(). The new approach:

  • Groups resource assignments by rule name using defaultdict
  • Uses the standardized Resources.parse(exprs, allow_expressions=True) method
  • Maintains proper error handling with context preservation via from err

This change eliminates duplicate parsing logic and ensures consistent resource validation across the codebase.


525-526: Proper integration with Resources class.

The changes correctly update the --resources argument to:

  • Use Resources() as the default instead of an empty dict
  • Use Resources.parser_factory() for standardized parsing

This ensures consistent resource handling throughout the CLI interface.


604-606: Correct configuration of default resources parser.

The parser configuration properly uses:

  • maybe_base64() wrapper for encoding support
  • defaults="full" to enable the full default resource set
  • allow_expressions=True to support Python expressions in default resources

This aligns with the documented behavior of allowing expressions like 2*input.size_mb in default resource specifications.

tests/tests.py (8)

15-18: LGTM - Import changes support the new resource system.

The addition of ResourceDuplicationError and the updated Resources import align with the resource management refactor described in the PR objectives.


674-675: LGTM - Good test coverage for thread overwriting.

This test appropriately verifies the --set-threads functionality with expression evaluation.


880-918: LGTM - Resource normalization testing is comprehensive.

The changes properly test the new resource system where:

  • Resources are normalized from suffixed to base forms (e.g., mem_mbmem)
  • All variants are automatically calculated and available (mem, mem_mb, mem_mib)
  • Default resources use the simplified list format
  • Mathematical conversions are correct (60000 MB = 57221 MiB)

921-1021: Excellent test coverage for new resource system features.

These tests comprehensively verify the key capabilities introduced in the resource refactor:

  • Mebibyte support: Tests that mem_mib specifications work correctly
  • Human-readable formats: Validates that global resources can use formats like "60 Gb"
  • Duplicate detection: Properly tests ResourceDuplicationError for conflicting resource keys
  • Validation constraints: Ensures suffixed resources cannot use human-readable formats

The parameterized testing approach for duplicate detection is particularly well-designed.


1034-1036: LGTM - Clean migration to new Resources API.

The systematic replacement of complex DefaultResources construction with Resources.parse() provides a cleaner, more maintainable approach:

  • String-based resource definitions are more readable
  • Explicit defaults="full" and allow_expressions=True parameters clarify intent
  • Consistent pattern across all affected test functions

Also applies to: 1064-1066, 1103-1105, 1129-1131, 1156-1158, 1182-1184, 1203-1205, 1225-1227, 1248-1250, 1275-1277, 1306-1308


1211-1252: LGTM - Good integration testing for cluster resource submission.

These tests properly verify:

  • Resource normalization during cluster submission (mem_mb → mem)
  • Correct handling of resource scopes (excluded resources not submitted)
  • Integration with executor plugin interfaces

The use of mocking to test the submission interface is appropriate and follows good testing practices.


1141-1165: LGTM - Important edge case testing for resource scope handling.

This test validates that resource scope overrides work correctly when users specify base resource names (e.g., mem) to override scopes for prefixed variants (e.g., mem_mb). This addresses a potentially confusing user scenario and ensures consistent behavior across the normalization system.


1484-1499: LGTM - Good coverage for human-readable default resources.

These tests properly verify that default resources can be specified using human-readable formats like "1 Gb" and "0.001 TB", as well as mebibyte specifications. This complements the earlier tests for global resource limits.

src/snakemake/resources.py (3)

603-604: LGTM - Improved type checking.

Good use of isinstance() for type checking instead of direct type comparison. This is a more robust approach.


790-822: Excellent resource management design.

The Resources class provides a clean abstraction with proper validation, normalization of size resources, and clear separation of concerns. The handling of suffixed resources (MB/MiB) is particularly well-implemented.


1027-1165: Well-designed resource scopes management.

The ResourceScopes class provides a clean, consistent interface for managing resource scope assignments. The normalization of size resources matches the pattern used in the Resources class, and the property-based interface for querying scopes is intuitive and well-documented.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (7)
src/snakemake/resources.py (7)

4-4: Remove unused import.

The dataclass import is not used anywhere in this file.


479-479: Fix spelling error.

The word "retreived" should be "retrieved".


484-484: Fix spelling error.

The word "intepreted" should be "interpreted".


695-695: Fix spelling error.

The word "evaulated" should be "evaluated".


771-771: Fix spelling error.

The word "intepreted" should be "interpreted".


776-776: Fix spelling error.

The word "interpetation" should be "interpretation".


925-925: Fix spelling error.

The word "assigments" should be "assignments".

🧹 Nitpick comments (1)
src/snakemake/resources.py (1)

604-604: Simplify type comparison logic.

The current type comparison logic can be simplified and made more readable.

-        if not isinstance(self_val, type(other_val)):
+        if type(self_val) != type(other_val):

This is more direct and follows the pattern used elsewhere in the codebase for type comparisons.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 287e661 and c296c16.

📒 Files selected for processing (3)
  • src/snakemake/resources.py (14 hunks)
  • src/snakemake/scheduler.py (4 hunks)
  • tests/test_ioutils/Snakefile (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/snakemake/scheduler.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py

⚙️ CodeRabbit Configuration File

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

Files:

  • src/snakemake/resources.py
🧠 Learnings (19)
📓 Common learnings
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
📚 Learning: changes made within test cases, such as in `tests/test_wrapper/snakefile`, are for testing purposes ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.

Applied to files:

  • tests/test_ioutils/Snakefile
📚 Learning: new test dependencies for snakemake should be introduced in separate prs rather than being added as ...
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.

Applied to files:

  • tests/test_ioutils/Snakefile
📚 Learning: in `snakemake/assets/__init__.py`, the `version` placeholders in asset urls refer to the versions of...
Learnt from: johanneskoester
PR: snakemake/snakemake#3203
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-11-21T10:15:16.961Z
Learning: In `snakemake/assets/__init__.py`, the `version` placeholders in asset URLs refer to the versions of external assets, not the Snakemake version. Therefore, these placeholders should not be flagged for version inconsistencies related to Snakemake.

Applied to files:

  • tests/test_ioutils/Snakefile
📚 Learning: the file "test.jsonl" in tests/test_nonstr_params is automatically created by snakemake, rather than...
Learnt from: johanneskoester
PR: snakemake/snakemake#2925
File: tests/test_nonstr_params/Snakefile:14-15
Timestamp: 2024-12-21T15:10:31.992Z
Learning: The file "test.jsonl" in tests/test_nonstr_params is automatically created by Snakemake, rather than manually generated in the Snakefile.

Applied to files:

  • tests/test_ioutils/Snakefile
📚 Learning: in the snakemake repository, avoid suggesting return type annotations for test functions....
Learnt from: johanneskoester
PR: snakemake/snakemake#2985
File: tests/tests.py:2051-2051
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.

Applied to files:

  • tests/test_ioutils/Snakefile
📚 Learning: in `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoi...
Learnt from: johanneskoester
PR: snakemake/snakemake#3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: do not suggest type annotations for `cls` arguments in class methods when reviewing code for the sna...
Learnt from: johanneskoester
PR: snakemake/snakemake#3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: for the snakemake project, avoid suggesting trailing commas as they use black as an autoformatter....
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/dag.py:1265-1266
Timestamp: 2024-10-11T07:09:54.544Z
Learning: For the Snakemake project, avoid suggesting trailing commas as they use Black as an autoformatter.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: in `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syn...
Learnt from: johanneskoester
PR: snakemake/snakemake#3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: avoid suggesting type annotations for functions that are inside methods in the snakemake codebase....
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: avoid adding input validation or error handling that unnecessarily complicates the code in the `snak...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: snakemake needs to maintain compatibility with python 3.7; avoid using syntax or features introduced...
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/branch.py:44-51
Timestamp: 2024-12-14T13:13:09.273Z
Learning: Snakemake needs to maintain compatibility with Python 3.7; avoid using syntax or features introduced after Python 3.7.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: in the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backw...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: in `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intention...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: the codebase must remain compatible with older python versions. avoid suggesting syntax not supporte...
Learnt from: johanneskoester
PR: snakemake/snakemake#3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: the user does not want to use exception chaining with “raise ... from e” for backward compatibility ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: in snakemake's logging handlers, keyboardinterrupt and systemexit exceptions should be ignored (pass...
Learnt from: johanneskoester
PR: snakemake/snakemake#3107
File: snakemake/logging.py:510-515
Timestamp: 2025-01-14T14:04:30.554Z
Learning: In Snakemake's logging handlers, KeyboardInterrupt and SystemExit exceptions should be ignored (pass) rather than re-raised, as these are control flow signals that should not disrupt the logging process. The exception variable in the final except block should be kept for potential debugging purposes.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: in `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's ...
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.

Applied to files:

  • src/snakemake/resources.py
🧬 Code Graph Analysis (1)
src/snakemake/resources.py (6)
src/snakemake/common/__init__.py (3)
  • get_input_function_aux_params (308-320)
  • mb_to_mib (64-65)
  • mib_to_mb (68-69)
src/snakemake/exceptions.py (8)
  • NestedCoroutineError (550-551)
  • ResourceConstraintError (636-637)
  • ResourceDuplicationError (640-647)
  • ResourceError (599-600)
  • ResourceInsufficiencyError (603-622)
  • ResourceScopesException (625-629)
  • ResourceValidationError (632-633)
  • is_file_not_found_error (675-682)
src/snakemake/common/tbdstring.py (1)
  • TBDString (3-96)
src/snakemake/io/__init__.py (9)
  • AnnotatedString (933-953)
  • Wildcards (1941-1942)
  • items (1835-1837)
  • get (1880-1885)
  • is_callable (314-315)
  • is_callable (944-945)
  • is_callable (1151-1156)
  • keys (1868-1869)
  • update (1810-1816)
src/snakemake/jobs.py (21)
  • jobs (1346-1347)
  • jobs (1350-1351)
  • resources (466-483)
  • resources (1491-1505)
  • name (1259-1260)
  • name (1642-1646)
  • group (446-449)
  • group (452-453)
  • wildcards (314-315)
  • wildcards (318-319)
  • input (322-323)
  • input (326-327)
  • input (1512-1517)
  • attempt (456-457)
  • attempt (460-463)
  • attempt (1713-1714)
  • attempt (1717-1722)
  • threads (408-409)
  • threads (1706-1707)
  • is_local (1014-1022)
  • is_local (1662-1663)
src/snakemake/workflow.py (8)
  • resources (2167-2172)
  • name (2251-2256)
  • group (2181-2186)
  • func (1646-1652)
  • input (1999-2004)
  • attempt (390-394)
  • threads (2146-2151)
  • is_local (687-691)
🪛 Ruff (0.12.2)
src/snakemake/resources.py

4-4: dataclasses.dataclass imported but unused

Remove unused import: dataclasses.dataclass

(F401)


220-220: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


537-540: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


768-783: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


874-874: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


888-888: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py

[failure] 925-925:
assigments ==> assignments


[failure] 776-776:
interpetation ==> interpretation


[failure] 771-771:
intepreted ==> interpreted


[failure] 695-695:
evaulated ==> evaluated


[failure] 484-484:
intepreted ==> interpreted


[failure] 479-479:
retreived ==> retrieved

🪛 GitHub Actions: Codespell
src/snakemake/resources.py

[error] 479-479: codespell: 'retreived' is a misspelling; did you mean 'retrieved'?

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (56)
  • GitHub Check: tests (8, windows-latest, py313)
  • GitHub Check: tests (10, windows-latest, py313)
  • GitHub Check: tests (10, ubuntu-latest, py313)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (9, windows-latest, py313)
  • GitHub Check: tests (7, windows-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (9, ubuntu-latest, py313)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py313)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (6, windows-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (6, windows-latest, py313)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (7, windows-latest, py313)
  • GitHub Check: tests (7, ubuntu-latest, py313)
  • GitHub Check: tests (5, windows-latest, py313)
  • GitHub Check: tests (6, ubuntu-latest, py313)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (4, windows-latest, py313)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (5, ubuntu-latest, py313)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (2, ubuntu-latest, py313)
  • GitHub Check: tests (3, ubuntu-latest, py313)
  • GitHub Check: tests (2, windows-latest, py313)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (3, windows-latest, py313)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py313)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (2, macos-latest, py313)
  • GitHub Check: tests (3, macos-latest, py313)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (1, macos-latest, py312)
  • GitHub Check: tests (2, macos-latest, py312)
  • GitHub Check: tests (1, macos-latest, py313)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (1, windows-latest, py313)
  • GitHub Check: tests (1, ubuntu-latest, py312)
  • GitHub Check: tests (1, windows-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py313)
  • GitHub Check: tests (1, ubuntu-latest, py311)
  • GitHub Check: apidocs
🔇 Additional comments (8)
src/snakemake/resources.py (7)

219-220: Exception chaining not used per project guidelines.

Based on the retrieved learnings, this project avoids exception chaining for backward compatibility reasons, so the current implementation is correct.


276-281: LGTM: Mutable default arguments fixed.

The change from mutable default arguments to None with conditional assignment inside the function is correct and addresses the static analysis warning.


823-823: LGTM: Correct None comparison.

Using is None is the proper way to check for None values in Python.


874-874: Exception handling follows project guidelines.

The exception is re-raised without chaining, which is correct per the project's backward compatibility requirements.


888-888: Exception handling follows project guidelines.

The exception is re-raised without chaining, which is correct per the project's backward compatibility requirements.


1034-1173: LGTM: Well-designed ResourceScopes class.

The ResourceScopes class provides a clean interface for managing resource scopes with proper validation, normalization of suffixed resources, and comprehensive scope querying methods. The inheritance from Dict[str, ValidScope] is appropriate and the validation logic is robust.


1-1173: Excellent refactoring of resource management system.

This comprehensive refactor successfully modernizes Snakemake's resource handling with several key improvements:

  1. Type Safety: Introduction of Resource and Resources classes with proper type annotations
  2. Validation: Robust validation of resource values and prevention of duplicate equivalent resources
  3. Unit Conversion: Seamless handling of MB/MiB conversions and human-friendly resource parsing
  4. Error Handling: Clear error messages and appropriate exception types
  5. Extensibility: Clean separation of concerns and modular design

The new system addresses the stated objectives of standardizing resource handling while maintaining backward compatibility.

tests/test_ioutils/Snakefile (1)

89-89: LGTM: Test updated for new resource handling.

The assertion correctly reflects the new resource system where resources.mem now returns the parsed numeric value (1000 MB) instead of the original string representation ("1GB"). This change is consistent with the resource refactoring in src/snakemake/resources.py.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (3)
src/snakemake/rules.py (1)

1130-1136: Exception chaining intentionally omitted per project standards.

The static analysis suggests adding exception chaining, but based on retrieved learnings, the Snakemake codebase intentionally avoids exception chaining for backward compatibility reasons.

tests/tests.py (2)

1141-1143: Fix variable naming in list comprehension.

The static analysis correctly identified an ambiguous variable name that should be fixed for better readability.

Apply this diff to fix the variable naming:

-        lines = [line for line in f.readlines() if line != "\n"]
+        lines = [line for line in f.readlines() if line != "\n"]

1148-1173: New test validates scope inheritance for resource variants.

Good test ensuring that scope overrides using base resource names properly apply to all variants. However, there's a code quality issue to address.

Fix the variable naming and comparison style at line 1169:

-        lines = [l for l in f.readlines() if not l == "\n"]
+        lines = [line for line in f.readlines() if line != "\n"]
🧹 Nitpick comments (1)
src/snakemake/rules.py (1)

1067-1098: LGTM: Well-structured resource evaluation with comprehensive error handling.

The evaluate function properly handles resource evaluation with appropriate exception handling for different error types. The logic correctly delegates to apply_input_function and handles edge cases like nested coroutines.

Minor spelling fix needed:

-                # Need to catch this because both input.size_mb and the inital
+                # Need to catch this because both input.size_mb and the initial
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c296c16 and 195d7bb.

📒 Files selected for processing (2)
  • src/snakemake/rules.py (7 hunks)
  • tests/tests.py (19 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py

⚙️ CodeRabbit Configuration File

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

Files:

  • src/snakemake/rules.py
  • tests/tests.py
🧠 Learnings (13)
📓 Common learnings
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
📚 Learning: in `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's ...
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: in the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backw...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: in `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intention...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: the codebase must remain compatible with older python versions. avoid suggesting syntax not supporte...
Learnt from: johanneskoester
PR: snakemake/snakemake#3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: the user does not want to use exception chaining with “raise ... from e” for backward compatibility ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: avoid adding input validation or error handling that unnecessarily complicates the code in the `snak...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: in the snakemake project, type annotations are not necessary for internal helper functions....
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: changes made within test cases, such as in `tests/test_wrapper/snakefile`, are for testing purposes ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.

Applied to files:

  • tests/tests.py
📚 Learning: in the snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing,...
Learnt from: johanneskoester
PR: snakemake/snakemake#3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.

Applied to files:

  • tests/tests.py
📚 Learning: in `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syn...
Learnt from: johanneskoester
PR: snakemake/snakemake#3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.

Applied to files:

  • tests/tests.py
📚 Learning: in the snakemake test suite, tests for configuration keys like 'group-components' may already be cov...
Learnt from: johanneskoester
PR: snakemake/snakemake#3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.

Applied to files:

  • tests/tests.py
📚 Learning: in test scripts within the snakemake project, concise code is preferred over verbose error handling ...
Learnt from: mbhall88
PR: snakemake/snakemake#3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.

Applied to files:

  • tests/tests.py
🧬 Code Graph Analysis (1)
src/snakemake/rules.py (4)
src/snakemake/io/__init__.py (3)
  • ResourceList (1949-1950)
  • get (1880-1885)
  • items (1835-1837)
src/snakemake/jobs.py (13)
  • resources (466-483)
  • resources (1491-1505)
  • name (1259-1260)
  • name (1642-1646)
  • wildcards (314-315)
  • wildcards (318-319)
  • input (322-323)
  • input (326-327)
  • input (1512-1517)
  • attempt (456-457)
  • attempt (460-463)
  • attempt (1713-1714)
  • attempt (1717-1722)
src/snakemake/resources.py (9)
  • Resource (473-794)
  • Resources (797-1028)
  • evaluate (567-584)
  • constrain (586-609)
  • get (1024-1028)
  • value (551-561)
  • items (1006-1007)
  • is_evaluable (563-565)
  • to_mib (611-636)
src/snakemake/exceptions.py (4)
  • ResourceConstraintError (636-637)
  • ResourceValidationError (632-633)
  • NestedCoroutineError (550-551)
  • InputFunctionException (250-262)
🪛 Ruff (0.12.2)
src/snakemake/rules.py

1130-1136: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

tests/tests.py

1169-1169: Ambiguous variable name: l

(E741)


1169-1169: Use l != "\n" instead of not l == "\n"

Replace with != operator

(SIM201)

🪛 GitHub Check: Check for spelling errors
src/snakemake/rules.py

[failure] 1091-1091:
inital ==> initial

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (56)
  • GitHub Check: apidocs
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py313)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (10, windows-latest, py313)
  • GitHub Check: tests (7, windows-latest, py313)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py313)
  • GitHub Check: tests (9, windows-latest, py313)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (8, windows-latest, py313)
  • GitHub Check: tests (8, ubuntu-latest, py313)
  • GitHub Check: tests (6, windows-latest, py313)
  • GitHub Check: tests (7, ubuntu-latest, py313)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (7, windows-latest, py312)
  • GitHub Check: tests (6, windows-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py313)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (5, windows-latest, py313)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (4, macos-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (2, macos-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py313)
  • GitHub Check: tests (3, windows-latest, py313)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (4, macos-latest, py313)
  • GitHub Check: tests (4, windows-latest, py313)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py313)
  • GitHub Check: tests (3, ubuntu-latest, py313)
  • GitHub Check: tests (3, macos-latest, py312)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (2, macos-latest, py313)
  • GitHub Check: tests (2, windows-latest, py313)
  • GitHub Check: tests (1, windows-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py313)
  • GitHub Check: tests (1, ubuntu-latest, py313)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (1, ubuntu-latest, py311)
  • GitHub Check: tests (1, windows-latest, py313)
  • GitHub Check: tests (1, ubuntu-latest, py312)
🔇 Additional comments (28)
src/snakemake/rules.py (2)

28-28: LGTM: Clean integration with new resource system.

The import additions and type annotations properly integrate with the new resource management system. The use of TYPE_CHECKING block correctly avoids circular imports, and the type annotations improve code clarity without violating the coding guidelines.

Also applies to: 50-55, 58-58, 77-79, 93-93, 105-105


1100-1146: LGTM: Excellent resource processing implementation.

The main resource processing logic is well-implemented with:

  • Proper handling of _cores resource first with thread constraints
  • Type consistency enforcement between rule and global resources
  • Correct evaluation flow with skip_evaluation support
  • Special handling for memory and disk resources to set _mb and _mib variants
  • Appropriate return of ResourceList instead of Resources

The integration with the new resource system is seamless and maintains backward compatibility.

tests/tests.py (26)

15-18: Import changes align with resource management refactoring.

The addition of ResourceDuplicationError and the import of the new Resources class properly support the PR's objectives for centralized resource handling and duplicate resource validation.


674-676: New test validates expression support in thread settings.

Good addition of test coverage for setting threads using mathematical expressions, which aligns with the enhanced resource handling capabilities.


880-881: Assertion tightened to enforce exact resource equality.

The change to exact equality checking aligns with the stricter resource normalization and validation introduced by the new resource management system.


902-919: Test updated to verify resource normalization and variant handling.

The changes correctly validate that the new resource system properly normalizes memory and disk resources to their base forms while maintaining all variants (mem/mem_mb/mem_mib and disk/disk_mb/disk_mib).


921-946: New test validates mebibyte resource specification support.

Excellent test coverage for the PR objective allowing mem_mib specifications to correctly map to their _mb variants. The conversion from 57221 MiB to 60000 MB is accurate.


947-954: Test validates resource handling with limited cores.

Good addition to test resource management behavior when global core supply is constrained.


956-980: Test validates human-readable resource specification support.

Excellent coverage of the PR objective supporting human-readable resources. The conversion from "60 Gb" to numeric values is correctly tested.


982-1002: Comprehensive test for preventing duplicate resource specifications.

Excellent parameterized test coverage ensuring that the system correctly raises ResourceDuplicationError when the same effective resource is specified in multiple forms, directly addressing one of the PR's key objectives.


1004-1015: Test validates format constraints on suffixed resources.

Good test ensuring that suffixed resource variants (mem_mb, mem_mib) cannot use human-readable formats, preventing potential unit conversion ambiguities.


1017-1028: Test ensures consistent format constraints across resource contexts.

Good coverage ensuring that the human-readable format restriction for suffixed resources applies consistently to both global and rule-level resource declarations.


1041-1058: Test updated to use new Resources.parse API.

Properly migrated from the legacy DefaultResources to the new Resources.parse method, maintaining test functionality while using the updated API.


1071-1088: Consistent migration to Resources.parse API.

Test properly updated to use the new resource parsing API while maintaining coverage for resource-constrained scenarios.


1095-1118: Improved test documentation and consistent API usage.

Good addition of explanatory comments clarifying the test's behavior. The migration to Resources.parse is consistent with other tests.


1189-1191: Consistent API migration.

Test properly updated to use Resources.parse.


1210-1212: API update for cluster scope submission test.

Properly migrated to Resources.parse.


1218-1240: Excellent test for resource normalization in cluster submission.

This test validates a critical aspect of the PR - ensuring that resource variants (mem_mb, disk_mb) are properly normalized to their base forms (mem, disk) when submitted to cluster executors.


1242-1260: Test validates resource exclusion from cluster submission.

Good coverage of the feature allowing certain resources to be excluded from cluster submission by setting their scope to "excluded".


1282-1301: Updated for new Resources API with pipe groups.

Test properly migrated to use Resources.parse while maintaining coverage for pipe-based job groups.


1313-1329: New test for insufficient cores with pipe groups.

Good addition of test coverage for the edge case where pipe groups cannot be scheduled due to insufficient cores. All Resources API migrations are consistent.


1497-1500: Migrated to list-based default resources.

Test updated to use the new list-based format for default resources instead of the removed DefaultResources class.


1503-1511: Test validates human-readable format for default resources.

Good coverage ensuring that human-readable formats work not just for global resources but also for default resource specifications.


1513-1520: Test validates mebibyte units in default resources.

Good test coverage for mebibyte support in default resources, including fractional values to verify proper rounding behavior.


1693-1693: Consistent default resources format update.

Updated to use list-based default resources format.


2051-2051: Added cleanup preservation for debugging.

Good addition of cleanup=False to help with debugging incomplete parameter scenarios.


2082-2084: Updated default resources format.

Consistent migration to list-based default resources.


2575-2575: Consistent default resources migration.

Updated to use list-based default resources format.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
src/snakemake/cli.py (1)

232-238: Improve exception handling pattern.

The exception handling should use proper exception chaining for better error traceability.

Apply this diff to improve exception chaining:

-                    raise ValueError(f"{errmsg} Cause: {e}")
+                    raise ValueError(f"{errmsg} Cause: {e}") from e
tests/tests.py (1)

1172-1172: Fix comparison style for better readability.

Use the != operator instead of not == for clearer comparison logic.

-        lines = [line for line in f.readlines() if not line == "\n"]
+        lines = [line for line in f.readlines() if line != "\n"]
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 195d7bb and 2a803fc.

📒 Files selected for processing (6)
  • src/snakemake/cli.py (7 hunks)
  • src/snakemake/exceptions.py (3 hunks)
  • src/snakemake/io/__init__.py (8 hunks)
  • src/snakemake/resources.py (14 hunks)
  • src/snakemake/scheduler.py (4 hunks)
  • tests/tests.py (19 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
  • src/snakemake/scheduler.py
  • src/snakemake/exceptions.py
  • src/snakemake/io/init.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py

⚙️ CodeRabbit Configuration File

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

Files:

  • tests/tests.py
  • src/snakemake/resources.py
  • src/snakemake/cli.py
🧠 Learnings (19)
📓 Common learnings
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
📚 Learning: changes made within test cases, such as in `tests/test_wrapper/snakefile`, are for testing purposes ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.

Applied to files:

  • tests/tests.py
📚 Learning: in the snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing,...
Learnt from: johanneskoester
PR: snakemake/snakemake#3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.

Applied to files:

  • tests/tests.py
📚 Learning: in `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syn...
Learnt from: johanneskoester
PR: snakemake/snakemake#3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.

Applied to files:

  • tests/tests.py
  • src/snakemake/resources.py
📚 Learning: in the snakemake test suite, tests for configuration keys like 'group-components' may already be cov...
Learnt from: johanneskoester
PR: snakemake/snakemake#3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.

Applied to files:

  • tests/tests.py
📚 Learning: in test scripts within the snakemake project, concise code is preferred over verbose error handling ...
Learnt from: mbhall88
PR: snakemake/snakemake#3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.

Applied to files:

  • tests/tests.py
📚 Learning: in `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoi...
Learnt from: johanneskoester
PR: snakemake/snakemake#3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: do not suggest type annotations for `cls` arguments in class methods when reviewing code for the sna...
Learnt from: johanneskoester
PR: snakemake/snakemake#3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: for the snakemake project, avoid suggesting trailing commas as they use black as an autoformatter....
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/dag.py:1265-1266
Timestamp: 2024-10-11T07:09:54.544Z
Learning: For the Snakemake project, avoid suggesting trailing commas as they use Black as an autoformatter.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: avoid suggesting type annotations for functions that are inside methods in the snakemake codebase....
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: snakemake needs to maintain compatibility with python 3.7; avoid using syntax or features introduced...
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/branch.py:44-51
Timestamp: 2024-12-14T13:13:09.273Z
Learning: Snakemake needs to maintain compatibility with Python 3.7; avoid using syntax or features introduced after Python 3.7.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: avoid adding input validation or error handling that unnecessarily complicates the code in the `snak...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: in the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backw...
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: in `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intention...
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: the codebase must remain compatible with older python versions. avoid suggesting syntax not supporte...
Learnt from: johanneskoester
PR: snakemake/snakemake#3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: the user does not want to use exception chaining with “raise ... from e” for backward compatibility ...
Learnt from: johanneskoester
PR: snakemake/snakemake#3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: in snakemake's logging handlers, keyboardinterrupt and systemexit exceptions should be ignored (pass...
Learnt from: johanneskoester
PR: snakemake/snakemake#3107
File: snakemake/logging.py:510-515
Timestamp: 2025-01-14T14:04:30.554Z
Learning: In Snakemake's logging handlers, KeyboardInterrupt and SystemExit exceptions should be ignored (pass) rather than re-raised, as these are control flow signals that should not disrupt the logging process. The exception variable in the final except block should be kept for potential debugging purposes.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: in `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's ...
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: in the `snakemake/cli.py` file, `executorpluginregistry()` is a singleton class, so multiple instant...
Learnt from: johanneskoester
PR: snakemake/snakemake#3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the `snakemake/cli.py` file, `ExecutorPluginRegistry()` is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.

Applied to files:

  • src/snakemake/cli.py
🪛 Ruff (0.12.2)
tests/tests.py

1172-1172: Use line != "\n" instead of not line == "\n"

Replace with != operator

(SIM201)


1262-1262: Do not assert False (python -O removes these calls), raise AssertionError()

Replace assert False

(B011)

src/snakemake/resources.py

4-4: dataclasses.dataclass imported but unused

Remove unused import: dataclasses.dataclass

(F401)


539-542: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


770-784: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


875-875: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


889-889: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/cli.py

229-229: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


236-236: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors
src/snakemake/resources.py

[failure] 777-777:
interpetation ==> interpretation

🪛 GitHub Actions: Codespell
src/snakemake/resources.py

[error] 777-777: codespell: 'interpetation' should be 'interpretation'

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (56)
  • GitHub Check: tests (10, ubuntu-latest, py313)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (10, windows-latest, py313)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (9, windows-latest, py313)
  • GitHub Check: tests (9, ubuntu-latest, py313)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (8, ubuntu-latest, py313)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (8, windows-latest, py313)
  • GitHub Check: tests (7, macos-latest, py312)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (7, windows-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py313)
  • GitHub Check: tests (7, windows-latest, py313)
  • GitHub Check: tests (6, windows-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py313)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (5, ubuntu-latest, py313)
  • GitHub Check: tests (5, windows-latest, py313)
  • GitHub Check: tests (6, windows-latest, py313)
  • GitHub Check: tests (4, windows-latest, py313)
  • GitHub Check: tests (3, ubuntu-latest, py313)
  • GitHub Check: tests (4, ubuntu-latest, py313)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (3, windows-latest, py313)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (2, macos-latest, py313)
  • GitHub Check: tests (2, macos-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (2, windows-latest, py313)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py313)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (1, macos-latest, py313)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (1, windows-latest, py313)
  • GitHub Check: tests (1, ubuntu-latest, py313)
  • GitHub Check: tests (1, ubuntu-latest, py312)
  • GitHub Check: tests (1, macos-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py311)
  • GitHub Check: tests (1, windows-latest, py312)
  • GitHub Check: apidocs
🔇 Additional comments (19)
src/snakemake/cli.py (4)

12-12: LGTM!

The import updates correctly replace legacy resource classes with the new unified Resource and Resources classes, aligning with the centralized resource handling objectives.

Also applies to: 43-44


109-120: LGTM!

The integration with Resource.from_cli_expression properly unifies thread parsing with the new resource system, providing consistent expression evaluation and type safety.


165-191: LGTM!

The function rewrite properly leverages Resources.parse for unified resource handling. The logic correctly groups assignments by rule and maintains good exception handling practices with proper exception chaining.


527-528: LGTM!

The argument parser integration with Resources() defaults and Resources.parser_factory() properly unifies CLI resource parsing with the new resource system, supporting both basic resource constraints and expression-enabled default resources.

Also applies to: 606-608

tests/tests.py (9)

15-18: LGTM - Import updates align with resource system refactoring.

The new imports for ResourceDuplicationError and updated resource classes (GroupResources, Resources) are necessary to support the enhanced resource management system introduced in this PR.


882-883: LGTM - Enhanced precision in resource testing.

Changing from partial resource matching to exact equality testing reflects the improved consistency and deterministic behavior of the new Resources class. This makes the tests more precise and reliable.


924-948: LGTM - Comprehensive test for mebibyte resource support.

This test properly validates that resources can be specified in mebibytes (mem_mib) and verifies the correct conversion to megabytes, supporting one of the key features mentioned in the PR objectives.


959-983: LGTM - Human-readable global resources test.

This test validates the new capability to specify global resource limits using human-readable formats (e.g., "60 Gb"), which addresses a key improvement mentioned in the PR objectives.


986-1005: LGTM - Resource duplication validation test.

This parameterized test effectively validates that the system properly prevents specifying the same effective resource multiple times (e.g., mem and mem_mb together), which was one of the bugs this PR aimed to fix.


1008-1018: LGTM - Validation for suffixed resource constraints.

This test ensures that suffixed resources like mem_mb cannot accept human-readable strings, maintaining the intended behavior where only unsuffixed resources support human-readable formats.


1021-1031: LGTM - Resource declaration validation test.

This test validates that suffixed resource declarations in rule overrides cannot use human-readable formats, ensuring consistent behavior across the resource system.


1044-1046: LGTM - Proper migration to new resource system.

The systematic replacement of the legacy DefaultResources class with Resources.parse() calls properly migrates to the new resource management system. The parameters defaults="full" and allow_expressions=True align with the enhanced capabilities of the new system.


675-677: LGTM - Test for resource expression evaluation.

This new test validates that resource expressions can be properly evaluated when setting threads via command line, supporting the enhanced expression capabilities in the new resource system.

src/snakemake/resources.py (6)

1-52: LGTM! Imports are well-organized and type annotations are appropriate.

The imports properly support the new resource management system with appropriate type annotations and forward references.


54-430: Excellent type safety improvements and error handling.

The GroupResources class has been significantly improved with:

  • Comprehensive type annotations for all parameters and return values
  • Better error handling with specific ResourceError and ResourceInsufficiencyError exceptions
  • Cleaner separation of concerns in resource processing

The refactoring maintains backward compatibility while improving code clarity.


432-466: Well-implemented unit conversion helpers.

The helper functions properly handle conversion between MB and MiB for evaluable resources with appropriate type validation and error messages.


475-796: Excellent Resource class implementation with comprehensive functionality.

The Resource class provides:

  • Proper validation and type conversion
  • Support for evaluable resources with deferred evaluation
  • Unit conversion between MB and MiB
  • Constraint enforcement
  • Human-friendly resource parsing
  • CLI expression evaluation

The design is robust and handles edge cases well, including TBDString handling and proper error messages.


798-1030: Well-designed Resources container class.

The Resources class provides excellent functionality:

  • Proper Mapping interface implementation
  • Automatic normalization of size resources (mem, disk) and their suffixed variants
  • Comprehensive CLI-style parsing with expression support
  • Good default resource management
  • Proper validation and error handling

The class effectively replaces the old DefaultResources with better type safety and more features.


1032-1174: Solid ResourceScopes implementation.

The ResourceScopes class provides:

  • Proper validation of scope values (local, global, excluded)
  • Automatic normalization of size resource suffixes
  • Convenient property methods and test methods for different scopes
  • Good integration with the overall resource system

The class design is clean and follows the established patterns from the Resources class.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (3)
tests/tests.py (1)

1172-1174: Remove debug print statement and fix comparison style.

Debug print should be removed and the comparison style should be fixed (though the latter is already flagged).

-        lines = [line for line in f.readlines() if not line == "\n"]
-        print(lines)
+        lines = [line for line in f.readlines() if line != "\n"]
src/snakemake/resources.py (2)

4-4: Remove unused import.


478-478: Fix spelling errors in docstring.

Also applies to: 483-483

🧹 Nitpick comments (2)
tests/tests.py (2)

1145-1147: Remove debug print statement.

Debug print statement should be removed from the test.

-        lines = [line for line in f.readlines() if line != "\n"]
-        print(lines)
+        lines = [line for line in f.readlines() if line != "\n"]

1221-1275: Address commented-out test implementations.

These tests for cluster resource submission appear incomplete. They should either be:

  1. Completed if the functionality is ready
  2. Marked with @pytest.mark.skip with a reason if temporarily disabled
  3. Removed if no longer needed

The tests seem important for validating resource normalization and exclusion in cluster submissions.

Would you like help implementing these tests or should we create an issue to track their completion?

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 85564b8 and e74152d.

📒 Files selected for processing (21)
  • src/snakemake/api.py (2 hunks)
  • src/snakemake/cli.py (8 hunks)
  • src/snakemake/common/__init__.py (5 hunks)
  • src/snakemake/dag.py (2 hunks)
  • src/snakemake/exceptions.py (3 hunks)
  • src/snakemake/io/__init__.py (8 hunks)
  • src/snakemake/jobs.py (5 hunks)
  • src/snakemake/logging.py (2 hunks)
  • src/snakemake/resources.py (14 hunks)
  • src/snakemake/rules.py (7 hunks)
  • src/snakemake/scheduler.py (4 hunks)
  • src/snakemake/script/__init__.py (1 hunks)
  • src/snakemake/settings/types.py (2 hunks)
  • src/snakemake/spawn_jobs.py (2 hunks)
  • src/snakemake/workflow.py (7 hunks)
  • tests/common.py (6 hunks)
  • tests/test_default_resources/Snakefile (1 hunks)
  • tests/test_github_issue3271/Snakefile_should_fail (1 hunks)
  • tests/test_group_jobs_resources/Snakefile (2 hunks)
  • tests/test_ioutils/Snakefile (1 hunks)
  • tests/tests.py (19 hunks)
🚧 Files skipped from review as they are similar to previous changes (14)
  • tests/test_ioutils/Snakefile
  • tests/test_github_issue3271/Snakefile_should_fail
  • src/snakemake/logging.py
  • tests/test_group_jobs_resources/Snakefile
  • tests/test_default_resources/Snakefile
  • src/snakemake/scheduler.py
  • src/snakemake/api.py
  • src/snakemake/io/init.py
  • src/snakemake/spawn_jobs.py
  • tests/common.py
  • src/snakemake/script/init.py
  • src/snakemake/settings/types.py
  • src/snakemake/dag.py
  • src/snakemake/exceptions.py
👮 Files not reviewed due to content moderation or server errors (5)
  • src/snakemake/common/init.py
  • src/snakemake/workflow.py
  • src/snakemake/rules.py
  • src/snakemake/jobs.py
  • src/snakemake/cli.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py

⚙️ CodeRabbit Configuration File

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

Files:

  • tests/tests.py
  • src/snakemake/resources.py
  • src/snakemake/cli.py
  • src/snakemake/common/__init__.py
  • src/snakemake/jobs.py
  • src/snakemake/rules.py
  • src/snakemake/workflow.py
🧠 Learnings (28)
📓 Common learnings
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
📚 Learning: 2024-10-06T14:09:54.370Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.

Applied to files:

  • tests/tests.py
  • src/snakemake/common/__init__.py
📚 Learning: 2024-11-12T20:22:54.184Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.

Applied to files:

  • tests/tests.py
📚 Learning: 2024-10-14T09:42:11.571Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.

Applied to files:

  • tests/tests.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-13T14:29:38.902Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.

Applied to files:

  • tests/tests.py
📚 Learning: 2024-08-13T09:25:24.046Z
Learnt from: johanneskoester
PR: snakemake/snakemake#2985
File: tests/tests.py:2051-2051
Timestamp: 2024-08-13T09:25:24.046Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.

Applied to files:

  • tests/tests.py
  • src/snakemake/common/__init__.py
📚 Learning: 2025-01-17T12:00:09.368Z
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: pytest-benchmark is not currently part of the Snakemake project's test dependencies. Suggestions involving pytest-benchmark should be deferred until it's added to the project.

Applied to files:

  • tests/tests.py
📚 Learning: 2025-01-17T12:00:09.368Z
Learnt from: leoschwarz
PR: snakemake/snakemake#3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.

Applied to files:

  • tests/tests.py
📚 Learning: 2024-11-12T12:08:20.342Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.

Applied to files:

  • tests/tests.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-04T16:12:18.927Z
Learnt from: lczech
PR: snakemake/snakemake#3113
File: snakemake/scheduler.py:912-914
Timestamp: 2024-10-04T16:12:18.927Z
Learning: In `snakemake/scheduler.py`, avoid suggesting the use of `asyncio.gather` in the `jobs_rewards` method due to overhead concerns and the need for immediate results.

Applied to files:

  • tests/tests.py
  • src/snakemake/jobs.py
📚 Learning: 2024-11-07T00:32:44.137Z
Learnt from: mbhall88
PR: snakemake/snakemake#3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.

Applied to files:

  • tests/tests.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-11T07:09:54.544Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/dag.py:1265-1266
Timestamp: 2024-10-11T07:09:54.544Z
Learning: For the Snakemake project, avoid suggesting trailing commas as they use Black as an autoformatter.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: 2024-08-27T20:15:58.699Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-08-27T20:15:58.699Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: 2024-08-13T16:22:09.641Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-08-13T16:22:09.641Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.

Applied to files:

  • src/snakemake/resources.py
  • src/snakemake/common/__init__.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.

Applied to files:

  • src/snakemake/resources.py
  • src/snakemake/rules.py
  • src/snakemake/workflow.py
📚 Learning: 2024-12-14T13:13:09.273Z
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/branch.py:44-51
Timestamp: 2024-12-14T13:13:09.273Z
Learning: Snakemake needs to maintain compatibility with Python 3.7; avoid using syntax or features introduced after Python 3.7.

Applied to files:

  • src/snakemake/resources.py
  • src/snakemake/common/__init__.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.

Applied to files:

  • src/snakemake/resources.py
  • src/snakemake/common/__init__.py
  • src/snakemake/rules.py
  • src/snakemake/workflow.py
📚 Learning: 2024-10-06T14:09:26.494Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Applied to files:

  • src/snakemake/resources.py
  • src/snakemake/common/__init__.py
  • src/snakemake/rules.py
  • src/snakemake/workflow.py
📚 Learning: 2024-09-25T10:55:53.963Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.

Applied to files:

  • src/snakemake/resources.py
  • src/snakemake/common/__init__.py
  • src/snakemake/rules.py
📚 Learning: 2024-12-21T15:04:25.569Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3237
File: snakemake/jobs.py:1192-1199
Timestamp: 2024-12-21T15:04:25.569Z
Learning: The user does not want to use exception chaining with “raise ... from e” for backward compatibility reasons.

Applied to files:

  • src/snakemake/resources.py
  • src/snakemake/rules.py
📚 Learning: 2025-01-14T14:04:30.554Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3107
File: snakemake/logging.py:510-515
Timestamp: 2025-01-14T14:04:30.554Z
Learning: In Snakemake's logging handlers, KeyboardInterrupt and SystemExit exceptions should be ignored (pass) rather than re-raised, as these are control flow signals that should not disrupt the logging process. The exception variable in the final except block should be kept for potential debugging purposes.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: 2024-12-14T13:11:32.988Z
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.

Applied to files:

  • src/snakemake/resources.py
  • src/snakemake/rules.py
📚 Learning: 2024-10-13T14:10:37.796Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the `snakemake/cli.py` file, `ExecutorPluginRegistry()` is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.

Applied to files:

  • src/snakemake/cli.py
📚 Learning: 2024-08-13T16:07:33.369Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-08-13T16:07:33.369Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.

Applied to files:

  • src/snakemake/common/__init__.py
📚 Learning: 2024-12-14T13:05:08.032Z
Learnt from: johanneskoester
PR: snakemake/snakemake#2927
File: snakemake/ioutils/evaluate.py:13-17
Timestamp: 2024-12-14T13:05:08.032Z
Learning: The Snakemake project supports Python 3.7, so code contributions must remain compatible with Python 3.7 and avoid features introduced in later Python versions.

Applied to files:

  • src/snakemake/common/__init__.py
📚 Learning: 2025-05-23T09:40:24.474Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3600
File: src/snakemake/jobs.py:960-964
Timestamp: 2025-05-23T09:40:24.474Z
Learning: In the `cleanup` method of the `Job` class in `src/snakemake/jobs.py`, files in the `to_remove` list should be formatted with `fmt_iofile` without specifying `as_output=True` or `as_input=True` parameters, as these files should be displayed as generic files rather than specifically as output files.

Applied to files:

  • src/snakemake/jobs.py
📚 Learning: 2024-10-11T12:49:08.705Z
Learnt from: johanneskoester
PR: snakemake/snakemake#3132
File: snakemake/rules.py:1105-1105
Timestamp: 2024-10-11T12:49:08.705Z
Learning: In `snakemake/rules.py`, within the `Rule` class, `self.basedir` is a custom class (not a `pathlib.Path` object) that has a `join()` method. Therefore, using `self.basedir.join(conda_env)` is appropriate in this context.

Applied to files:

  • src/snakemake/workflow.py
📚 Learning: 2025-05-06T01:37:23.382Z
Learnt from: kdm9
PR: snakemake/snakemake#3562
File: src/snakemake/checkpoints.py:90-90
Timestamp: 2025-05-06T01:37:23.382Z
Learning: In Snakemake checkpoints implementation, tracking only the first missing output for each checkpoint is sufficient, because if one output is missing, all outputs for that checkpoint are considered incomplete. This was the behavior before PR #3562 and maintained in the pluralized `checkpoints.get()` implementation.

Applied to files:

  • src/snakemake/workflow.py
🧬 Code Graph Analysis (3)
src/snakemake/resources.py (5)
src/snakemake/common/__init__.py (3)
  • get_input_function_aux_params (308-320)
  • mb_to_mib (64-65)
  • mib_to_mb (68-69)
src/snakemake/exceptions.py (8)
  • NestedCoroutineError (551-552)
  • ResourceConstraintError (637-638)
  • ResourceDuplicationError (641-648)
  • ResourceError (600-601)
  • ResourceInsufficiencyError (604-623)
  • ResourceScopesException (626-630)
  • ResourceValidationError (633-634)
  • is_file_not_found_error (676-683)
src/snakemake/common/tbdstring.py (1)
  • TBDString (3-96)
src/snakemake/io/__init__.py (9)
  • AnnotatedString (934-954)
  • Wildcards (1942-1943)
  • items (1836-1838)
  • get (1881-1886)
  • is_callable (315-316)
  • is_callable (945-946)
  • is_callable (1152-1157)
  • keys (1869-1870)
  • update (1811-1817)
src/snakemake/jobs.py (20)
  • jobs (1346-1347)
  • jobs (1350-1351)
  • Job (177-1287)
  • resources (466-483)
  • resources (1491-1505)
  • name (1259-1260)
  • name (1642-1646)
  • group (446-449)
  • group (452-453)
  • wildcards (314-315)
  • wildcards (318-319)
  • input (322-323)
  • input (326-327)
  • input (1512-1517)
  • attempt (456-457)
  • attempt (460-463)
  • attempt (1713-1714)
  • attempt (1717-1722)
  • is_local (1014-1022)
  • is_local (1662-1663)
src/snakemake/common/__init__.py (1)
src/snakemake/exceptions.py (1)
  • NestedCoroutineError (551-552)
src/snakemake/workflow.py (4)
src/snakemake/exceptions.py (1)
  • ResourceValidationError (633-634)
src/snakemake/jobs.py (6)
  • resources (466-483)
  • resources (1491-1505)
  • name (1259-1260)
  • name (1642-1646)
  • threads (408-409)
  • threads (1706-1707)
src/snakemake/resources.py (8)
  • Resources (798-1029)
  • value (553-563)
  • copy (1021-1023)
  • get (1025-1029)
  • args (863-868)
  • from_mapping (987-993)
  • update (995-999)
  • update (1170-1173)
src/snakemake/rules.py (3)
  • name (138-139)
  • name (142-143)
  • Rule (84-1332)
🪛 Ruff (0.12.2)
tests/tests.py

1172-1172: Use line != "\n" instead of not line == "\n"

Replace with != operator

(SIM201)

src/snakemake/resources.py

4-4: dataclasses.dataclass imported but unused

Remove unused import: dataclasses.dataclass

(F401)


539-542: Use ternary operator value = "function(...)" if self.is_evaluable() else self._value instead of if-else-block

Replace if-else-block with value = "function(...)" if self.is_evaluable() else self._value

(SIM108)


770-784: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


875-875: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


889-889: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/cli.py

229-229: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


236-236: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/common/__init__.py

106-112: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/jobs.py

38-38: snakemake.io.is_callable imported but unused

Remove unused import: snakemake.io.is_callable

(F401)

src/snakemake/rules.py

1130-1136: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/workflow.py

1808-1811: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1820-1825: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

🪛 GitHub Check: Check for spelling errors
src/snakemake/cli.py

[failure] 317-317:
intepretation ==> interpretation

🪛 GitHub Actions: Codespell
src/snakemake/cli.py

[error] 317-317: codespell: Spelling mistake found - 'intepretation' should be 'interpretation'.

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (49)
  • GitHub Check: tests (2, windows-latest, py313)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (10, ubuntu-latest, py313)
  • GitHub Check: tests (10, windows-latest, py313)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (8, windows-latest, py313)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (9, windows-latest, py313)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (8, ubuntu-latest, py313)
  • GitHub Check: tests (7, ubuntu-latest, py313)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (6, ubuntu-latest, py313)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (7, windows-latest, py312)
  • GitHub Check: tests (3, windows-latest, py313)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (7, windows-latest, py313)
  • GitHub Check: tests (5, windows-latest, py313)
  • GitHub Check: tests (6, windows-latest, py313)
  • GitHub Check: tests (4, windows-latest, py313)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (6, windows-latest, py312)
  • GitHub Check: tests (4, macos-latest, py313)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py312)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py313)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py313)
  • GitHub Check: tests (5, ubuntu-latest, py313)
  • GitHub Check: tests (4, macos-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (1, windows-latest, py313)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (2, macos-latest, py313)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (1, macos-latest, py313)
  • GitHub Check: tests (2, ubuntu-latest, py313)
  • GitHub Check: tests (1, macos-latest, py312)
🔇 Additional comments (21)
tests/tests.py (12)

15-19: LGTM! Import updates align with the new resource system.

The imports correctly reflect the migration from DefaultResources to the new Resources class and add necessary exception types for enhanced resource validation.


675-677: Good test coverage for resource expression evaluation.

This test validates the new capability to set threads via expressions, which is a key feature of the enhanced resource management system.


882-883: Improved assertion strictness for resource validation.

The change to exact equality checking provides more rigorous validation, ensuring all resources match exactly rather than just checking a subset.


904-921: Comprehensive validation of resource unit conversions.

The updated test properly validates the new resource system's ability to handle different unit representations (MB vs MiB) and ensures correct conversions between them.


923-948: Excellent test for mebibyte resource specification.

This test validates the new capability to specify resources using mebibyte units, with correct unit conversion logic.


950-956: Valid smoke test for global resource constraints.

Simple but effective test to ensure global resource constraints properly limit core allocation.


958-983: Great test for human-readable resource specifications.

This validates the important new feature of supporting human-readable strings like "60 Gb" for global resource limits.


985-1005: Comprehensive validation of resource duplication prevention.

Excellent parameterized test ensuring the system correctly rejects simultaneous specification of equivalent resources with different suffixes.


1007-1031: Good validation of suffixed resource constraints.

These tests correctly ensure that resources with explicit unit suffixes (e.g., mem_mb) cannot use human-readable strings, preventing ambiguity in unit interpretation.


1334-1345: Good test for pipe group resource validation.

This test properly validates that insufficient cores for pipe groups are detected, using the new Resources.parse method correctly.


1519-1536: Excellent coverage of default resource specification formats.

These tests validate important new capabilities:

  • Human-readable default resources ('1 Gb', '0.001 TB')
  • Mebibyte unit specifications with fractional values

Both are key features of the enhanced resource system.


1709-1709: Consistent migration to new resource system.

All test updates correctly migrate from DefaultResources to either plain lists or Resources.parse() with appropriate parameters, maintaining test functionality while using the new resource system.

Also applies to: 2596-2596

src/snakemake/resources.py (9)

219-222: Add exception chaining for better error context.

When re-raising exceptions, preserve the original exception context for better debugging.

 except ResourceError as err:
     raise ResourceInsufficiencyError(
         additive_resources, err.args[0]
-    ) from err
+    ) from err

Wait, the code already has from err. Let me re-check... Actually, the code correctly includes exception chaining. This is good practice.


275-310: Well-implemented resource merging with proper default handling.

Good fix for mutable default arguments by initializing them within the function. The type annotations and logic are solid.


432-459: Clean implementation of unit conversion decorators.

These decorators properly handle unit conversion for evaluable resources with appropriate type checking and error messages.


588-612: Well-implemented resource constraint logic.

The constrain method properly handles type checking and edge cases, ensuring resources are appropriately limited by constraints.


613-677: Excellent unit conversion implementation.

The conversion methods properly handle all resource types including evaluable resources and TBDStrings, with clear error messages for invalid operations.


787-796: Clean human-friendly parsing implementation.

Properly handles parsing of human-readable size and time strings with appropriate unit conversions.


798-841: Robust resource container with proper normalization.

The Resources class correctly enforces that suffixed and unsuffixed versions of size resources cannot coexist, preventing ambiguity in resource specifications.


986-1030: Well-designed utility methods for resource management.

The utility methods provide clean interfaces for resource manipulation with proper type handling and value unwrapping capabilities.


1032-1174: Comprehensive resource scope management.

The ResourceScopes class properly validates scope values, normalizes size resource suffixes, and provides clean interfaces for querying resources by scope. The special handling of suffixed resources in the excluded property is particularly well thought out.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tests/common.py (1)

305-308: Capture the subprocess failure before re-raising.

With the new shouldfail branch we now re-raise exception, but in the shellcmd path we never assign the caught subprocess.CalledProcessError to exception. When a CLI invocation is expected to fail and shouldfail is set to an exception type, this code will attempt to raise exception where exception is None, triggering a TypeError instead of validating the original failure. Please assign the caught exception so that the later pytest.raises block can re-raise it.

         except subprocess.CalledProcessError as e:
             success = False
+            exception = e
             print(e.stdout.decode(), file=sys.stderr)
♻️ Duplicate comments (2)
src/snakemake/dag.py (1)

18-18: Fix duplicate typing import.

Dict is still listed twice, which keeps the lint/docs job failing (Ruff F811). Please drop the trailing , Dict.

-from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Union, Dict
+from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Union
src/snakemake/scheduling/job_scheduler.py (1)

103-111: Global resource expansion still drops human-readable limits.

We’re still calling expand_items(..., expand_sized=False) and discarding every string value, so a global --resources mem="1 GB" no longer yields mem_mb/mem_mib and the scheduler ends up with no numeric constraint at all. This recreates the bug flagged earlier: human-friendly limits are silently ignored for DAG/job scheduling. Please expand sized resources and keep only the numeric (or None) values so the scheduler enforces the correct totals.

-        self.global_resources = {
-            name: (sys.maxsize if res is None else res)
-            for name, res in workflow.global_resources.expand_items(
-                constraints={},
-                evaluate=None,
-                expand_sized=False,
-            )
-            if not isinstance(res, str)
-        }
+        self.global_resources = {
+            name: (sys.maxsize if res is None else res)
+            for name, res in workflow.global_resources.expand_items(
+                constraints={},
+                evaluate=None,
+                expand_sized=True,
+            )
+            if res is None or isinstance(res, int)
+        }
🧹 Nitpick comments (7)
tests/tests.py (2)

1168-1171: Drop leftover debug print.

The print can make CI logs noisy. Safe to remove.

-        lines = [line for line in f.readlines() if line != "\n"]
-        print(lines)
+        lines = [line for line in f.readlines() if line != "\n"]

1175-1197: Simplify comparison style.

Prefer line != "\n" over not line == "\n" for clarity and consistency with nearby code.

-        lines = [line for line in f.readlines() if not line == "\n"]
+        lines = [line for line in f.readlines() if line != "\n"]
src/snakemake/workflow.py (1)

1891-1906: Avoid exception chaining in rule resource mapping.

Remove “from err” to follow Snakemake’s style and keep errors clean.

-                except ResourceDuplicationError as err:
-                    raise RuleException(err, rule=rule) from err
+                except ResourceDuplicationError as err:
+                    raise RuleException(err, rule=rule)
@@
-                except ResourceConversionError as err:
+                except ResourceConversionError as err:
                     msg = "Standard resource specified with invalid type, got error:\n"
-                    raise RuleException(msg + str(err), rule=rule) from err
+                    raise RuleException(msg + str(err), rule=rule)
@@
-                except ResourceValidationError as err:
-                    raise RuleException(err, rule=rule) from err
+                except ResourceValidationError as err:
+                    raise RuleException(err, rule=rule)

Based on learnings

src/snakemake/resources.py (4)

34-34: Remove redundant import.

humanfriendly is already imported on line 6 with specific functions. This bare import on line 34 is redundant.

Apply this diff:

-import humanfriendly

282-285: Consider using f-string for consistency.

The error message uses .format() instead of an f-string, which is inconsistent with the rest of the file.

Apply this diff:

             if len(unique) > 1:
                 raise WorkflowError(
-                    "Resource {name} is a string but not all jobs in group require the "
-                    "same value. Observed values: {values}.".format(
-                        name=name, values=unique
-                    )
+                    f"Resource {name} is a string but not all jobs in group require the "
+                    f"same value. Observed values: {unique}."
                 )

399-399: Unusual type annotation syntax.

The type hint zip[Tuple[int, ...]] is non-standard Python. The zip built-in doesn't accept type parameters in this way. This should be Iterator[Tuple[int, ...]] or left without a type hint.

Apply this diff:

-        resource_groups: zip[Tuple[int, ...]] = zip(*resources.values())
+        resource_groups: Iterator[Tuple[int, ...]] = zip(*resources.values())

768-816: Exception handling uses is_ordinary_string before definition.

Line 798 calls is_ordinary_string(val) but this function is defined later at line 1303. While Python allows this (the function isn't called until runtime when is_ordinary_string will be defined), consider moving the helper function earlier in the file for better code organization.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 960a3a4 and 80e19ad.

📒 Files selected for processing (22)
  • src/snakemake/api.py (2 hunks)
  • src/snakemake/cli.py (7 hunks)
  • src/snakemake/common/__init__.py (5 hunks)
  • src/snakemake/dag.py (2 hunks)
  • src/snakemake/exceptions.py (3 hunks)
  • src/snakemake/io/__init__.py (8 hunks)
  • src/snakemake/jobs.py (5 hunks)
  • src/snakemake/logging.py (2 hunks)
  • src/snakemake/resources.py (13 hunks)
  • src/snakemake/rules.py (7 hunks)
  • src/snakemake/scheduling/job_scheduler.py (4 hunks)
  • src/snakemake/script/__init__.py (1 hunks)
  • src/snakemake/settings/types.py (2 hunks)
  • src/snakemake/spawn_jobs.py (2 hunks)
  • src/snakemake/workflow.py (7 hunks)
  • test.txt (1 hunks)
  • tests/common.py (7 hunks)
  • tests/test_default_resources/Snakefile (1 hunks)
  • tests/test_github_issue3271/Snakefile_should_fail (1 hunks)
  • tests/test_group_jobs_resources/Snakefile (2 hunks)
  • tests/test_ioutils/Snakefile (1 hunks)
  • tests/tests.py (19 hunks)
🚧 Files skipped from review as they are similar to previous changes (5)
  • src/snakemake/spawn_jobs.py
  • test.txt
  • src/snakemake/script/init.py
  • tests/test_ioutils/Snakefile
  • tests/test_group_jobs_resources/Snakefile
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py

⚙️ CodeRabbit configuration file

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

Files:

  • src/snakemake/settings/types.py
  • src/snakemake/scheduling/job_scheduler.py
  • src/snakemake/common/__init__.py
  • tests/common.py
  • src/snakemake/workflow.py
  • tests/tests.py
  • src/snakemake/exceptions.py
  • src/snakemake/jobs.py
  • src/snakemake/api.py
  • src/snakemake/cli.py
  • src/snakemake/io/__init__.py
  • src/snakemake/dag.py
  • src/snakemake/rules.py
  • src/snakemake/resources.py
  • src/snakemake/logging.py
🧠 Learnings (22)
📚 Learning: 2025-10-30T15:39:38.224Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3813
File: tests/test_pathvars/Snakefile:22-23
Timestamp: 2025-10-30T15:39:38.224Z
Learning: In Snakemake, `<resources>` is a default pathvar that does not need to be explicitly defined in the pathvars section. It is available for use in rules and functions without requiring declaration.

Applied to files:

  • src/snakemake/settings/types.py
  • tests/common.py
  • src/snakemake/workflow.py
  • tests/tests.py
  • src/snakemake/jobs.py
  • src/snakemake/api.py
  • src/snakemake/cli.py
  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-04T16:12:18.927Z
Learnt from: lczech
Repo: snakemake/snakemake PR: 3113
File: snakemake/scheduler.py:912-914
Timestamp: 2024-10-04T16:12:18.927Z
Learning: In `snakemake/scheduler.py`, avoid suggesting the use of `asyncio.gather` in the `jobs_rewards` method due to overhead concerns and the need for immediate results.

Applied to files:

  • src/snakemake/scheduling/job_scheduler.py
  • src/snakemake/jobs.py
📚 Learning: 2024-10-13T14:10:37.796Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the `snakemake/cli.py` file, `ExecutorPluginRegistry()` is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.

Applied to files:

  • src/snakemake/scheduling/job_scheduler.py
📚 Learning: 2024-10-06T14:09:26.494Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Applied to files:

  • src/snakemake/common/__init__.py
  • src/snakemake/rules.py
📚 Learning: 2024-10-06T14:09:54.370Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.

Applied to files:

  • tests/common.py
  • tests/tests.py
  • tests/test_github_issue3271/Snakefile_should_fail
  • tests/test_default_resources/Snakefile
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.

Applied to files:

  • src/snakemake/workflow.py
  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-11-12T20:22:54.184Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.

Applied to files:

  • tests/tests.py
  • src/snakemake/rules.py
📚 Learning: 2024-10-13T14:29:38.902Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.

Applied to files:

  • tests/tests.py
📚 Learning: 2025-01-17T12:00:09.368Z
Learnt from: leoschwarz
Repo: snakemake/snakemake PR: 3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.

Applied to files:

  • tests/test_github_issue3271/Snakefile_should_fail
  • tests/test_default_resources/Snakefile
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2985
File: tests/tests.py:2051-2051
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.

Applied to files:

  • src/snakemake/io/__init__.py
  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.

Applied to files:

  • src/snakemake/io/__init__.py
  • src/snakemake/rules.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.

Applied to files:

  • src/snakemake/io/__init__.py
  • src/snakemake/rules.py
📚 Learning: 2024-10-14T09:42:11.571Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.

Applied to files:

  • src/snakemake/dag.py
📚 Learning: 2024-11-12T12:08:20.342Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.

Applied to files:

  • src/snakemake/dag.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2024-11-07T00:32:44.137Z
Learnt from: mbhall88
Repo: snakemake/snakemake PR: 3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2025-08-28T13:15:55.232Z
Learnt from: Hocnonsense
Repo: snakemake/snakemake PR: 3714
File: src/snakemake/modules.py:195-195
Timestamp: 2025-08-28T13:15:55.232Z
Learning: In src/snakemake/modules.py, the line `self.globals["__name__"] = namespace` is intentionally designed to set __name__ to None when namespace is None (i.e., when not inside a module context, for top-level workflows). This is correct behavior and should not be guarded.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2025-08-07T21:51:08.043Z
Learnt from: pvandyken
Repo: snakemake/snakemake PR: 3421
File: src/snakemake/resources.py:606-607
Timestamp: 2025-08-07T21:51:08.043Z
Learning: In Snakemake's Resource.constrain method, the type comparison uses isinstance(self_val, type(other_val)) rather than direct type equality to allow strings to be compared with AnnotatedString objects. This polymorphic behavior is intentional and necessary for proper resource constraint functionality.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1250-1251
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid commenting on trailing commas in code reviews for the Snakemake repository as per the user's preference.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2024-09-25T10:55:53.963Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: 2024-12-14T13:11:32.988Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.

Applied to files:

  • src/snakemake/resources.py
🔇 Additional comments (27)
tests/test_github_issue3271/Snakefile_should_fail (1)

1-13: Test structure looks correct.

The resource lambda correctly uses input.size_mb (which returns a numeric value, not a string), and the params lambda properly accesses resources.mem_mb. The division by 1000 on line 9 is appropriate for decimal unit conversion (MB → GB in decimal units). The test file structure appears sound for its intended purpose of validating error handling in issue #3271.

tests/tests.py (5)

15-19: Imports align with new Resources API.

Looks good; correct modules used for Resource* exceptions and Resources utilities.


688-690: New overwrite-threads test is appropriate.

Covers expression handling for --set-threads; no issues.


906-908: Assert helper tightened.

Strict dict equality here is fine for the expected resource dicts.


946-964: Resource normalization and validation tests look solid.

  • Human-readable/global vs local and duplication checks are well covered.
  • Error types match the new exceptions.

Also applies to: 965-990, 1001-1025, 1028-1046, 1050-1055


1067-1070: Fix mem_mib expected value (off-by-one).

60000 MB converts to ceil(60000×0.95367431640625) = 57221 MiB, not 57222. Update assertion.

-        mem_mib=57222,
+        mem_mib=57221,

Also applies to: 1075-1084

src/snakemake/rules.py (3)

30-31: API/type updates look correct.

Imports, TYPE_CHECKING, and attributes align with the new Resources model.

Also applies to: 52-58, 81-83, 97-97, 110-110


1132-1164: Resource expansion and constraint flow LGTM.

  • _cores evaluation constrained by max_threads and global _cores.
  • skip={"_cores"} avoids re-evaluating threads.
  • Error messaging for constraint/conversion is clear.

1085-1164: Tighten exception handling in expand_resources.

  • Avoid exception chaining (“from err/e”) to match Snakemake style.
  • Catch Exception instead of BaseException to avoid swallowing KeyboardInterrupt/SystemExit.

Apply this diff:

-            except ResourceValidationError as err:
-                raise WorkflowError(err, rule=self) from err
+            except ResourceValidationError as err:
+                raise WorkflowError(err, rule=self)
@@
-            except BaseException as e:
-                raise InputFunctionException(e, rule=self, wildcards=wildcards) from e
+            except Exception as e:
+                raise InputFunctionException(e, rule=self, wildcards=wildcards)

To check consistency repo-wide:

Based on learnings

src/snakemake/workflow.py (3)

132-133: Resources integration looks consistent.

  • Using ResourceScopes/Resources and .value access on _cores/_nodes aligns with the new model.
  • Directly reusing resource_settings.resources for global_resources matches PR goals.

Also applies to: 182-185, 631-631, 635-635


1871-1876: Default resources and threads initialization LGTM.

Copies defaults when provided, ensures _nodes=1, and sets _cores with overwrite precedence.

Also applies to: 1878-1885


2009-2011: Handover behavior confirmed.

Updating rule.resources from global_resources when unspecified matches the documented intent.

src/snakemake/resources.py (15)

169-170: Verify SizedResources filtering doesn't hide TBDString issues.

The condition filters out both TBDString instances and SizedResources, but the order matters. If a resource value is a TBDString and also has a name in SizedResources, both conditions should be checked. The current logic is correct but could be clearer.


447-459: Helper types and constants look good.

The type aliases, constants, and helper function are well-defined and appropriately typed.


493-517: Resource initialization handles validation and conversion well.

The __init__ method properly validates input types, converts floats to integers, parses human-friendly values, and sets up evaluators. The logic is sound.


571-602: Constrain method logic is correct.

Based on retrieved learnings, the isinstance(self_val, type(other_val)) check on line 592 is intentionally designed to allow polymorphic behavior between strings and AnnotatedString objects. The implementation correctly handles None values, TBDString, and type mismatches.

Based on learnings


838-847: Human-friendly parsing ensures minimum value of 1.

The use of max(..., 1) on lines 840 and 845 ensures that parsed sizes and timespans are at least 1, preventing zero or negative values from human-readable inputs. This is a good defensive practice.


875-882: Default resource expressions are well-designed.

The DEFAULTS dictionary provides sensible defaults:

  • mem scales with input size (2x) with min 1GB and max 8GB
  • disk scales with input size (2x) with min 1GB, or 50GB if no input
  • tmpdir uses system temp directory

These expressions will be evaluated in the context of each rule.


892-903: Size normalization prevents resource duplication.

The _normalize_sizes method correctly detects when multiple variants of a size resource (e.g., mem, mem_mb, mem_mib) are provided and raises ResourceDuplicationError. The standardization ensures only one canonical form is stored.


1006-1015: Good validation of reserved resource name.

Explicitly checking for and rejecting _cores prevents conflicts with Snakemake's internal resource management. The error message is clear and actionable.


1025-1037: Integer validation correctly allows zero.

Based on the past review comment discussion, the check val < 0 is correct because zero is a valid value for some contexts (e.g., --set-resources). The validation only rejects negative integers.

Based on learnings


1108-1156: expand_items provides comprehensive resource expansion.

This method handles:

  • Evaluation of callable resources
  • Constraint application
  • Expansion of sized resources into both human-friendly and mebibyte forms

The logic correctly yields tuples and handles the skip set and expand_sized flag.


1169-1189: ResourceScopes initialization validates scope values.

The __init__ method:

  • Normalizes size resource variants (mem/mem_mb/mem_mib)
  • Validates that all scopes are either 'local', 'global', or 'excluded'
  • Raises ResourceScopesException with clear error message if invalid

The implementation is correct.


1202-1204: Sensible default scopes.

The defaults() classmethod sets:

  • mem_mb and disk_mb as local (per-job pools)
  • runtime as excluded (not submitted to cluster)

These defaults align with typical Snakemake resource management patterns.


1287-1295: Excluded property correctly expands sized resources.

The excluded property adds suffixed versions (_mb, _mib) of any sized resources that have a defined scope. This ensures that when a base resource (e.g., mem) is excluded, all its variants are also excluded.


1303-1336: is_ordinary_string helper has comprehensive validation.

The function uses a well-designed regex pattern to detect:

  • Function calls: func(...)
  • Dictionary literals: {...}
  • Lambda expressions: lambda x: ...
  • Math operators: +, -, *, /, %
  • Attribute access: .attribute

This helps distinguish ordinary strings from Python expressions that should be evaluated. Consider adding test coverage for edge cases like escaped quotes or nested structures.


212-213: Potential KeyError when constraint not found.

Line 212 calls constraints.get(name).value which will raise an AttributeError if name is not in constraints (since .get() returns a Resource("", None) per line 1105, and calling .value on that should work). However, this assumes the Resources.get() method returns a default Resource object.

Run this script to verify the Resources.get() behavior:

@pvandyken
Copy link
Copy Markdown
Contributor Author

@pvandyken is this test here fixed along with your changes? #3367

Yes!

@pvandyken pvandyken force-pushed the fix/human-friendly-units-groups branch from 80e19ad to e017870 Compare November 3, 2025 22:02
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (3)
src/snakemake/exceptions.py (1)

658-665: Fix the ValueError message to avoid tuple representation.

This issue was previously flagged. Passing duplicate_resources as a second positional argument to super().__init__() causes str(ResourceDuplicationError) to display as a tuple ('message', [...]). Remove it from the super call and keep only the message string.

Apply this diff:

 class ResourceDuplicationError(ValueError):
     def __init__(self, duplicate_resources: Sequence[str]):
         self.duplicate_resources = duplicate_resources
         super().__init__(
             "The following resources are equivalent and cannot be simultaneously "
-            "provided to a rule:",
-            duplicate_resources,
+            "provided to a rule: " + ", ".join(duplicate_resources)
         )
src/snakemake/dag.py (1)

18-18: Duplicate Dict in typing import (ruff F811).

Remove the second Dict to prevent redefinition warnings/errors.

-from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Union, Dict
+from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Union
src/snakemake/cli.py (1)

168-193: Past review concerns remain unresolved.

Three issues previously flagged in past reviews are still present:

  1. Grammar error (Line 171): Missing comma in error message—"a positive integer a quoted string" should be "a positive integer, a quoted string".
  2. Missing validation (Line 183): No check that ":" exists in entry before split(":", maxsplit=1), which could raise ValueError with an unclear message if malformed.
  3. Incomplete exception handling (Line 192): Only catches ValueError, but Resources.parse can also raise WorkflowError per the code in src/snakemake/resources.py. Should catch both and chain them.

Apply this consolidated fix:

     errmsg = (
         "Invalid resource definition: entries have to be defined as "
-        "RULE:RESOURCE=VALUE, with VALUE being a positive integer a quoted string, or "
+        "RULE:RESOURCE=VALUE, with VALUE being a positive integer, a quoted string, or "
         "a Python expression (e.g. min(max(2*input.size_mb, 1000), 8000))."
     )

     if args is None:
         return {}

     assignments: Dict[str, List[str]] = defaultdict(list)

     for entry in args:
+        if ":" not in entry:
+            raise ValueError(f"{errmsg} (Unparsable value: {repr(entry)})")
         rule, assign = entry.split(":", maxsplit=1)
         assignments[rule].append(assign)

     try:
         return {
             rule: Resources.parse(assigns, allow_expressions=True)
             for rule, assigns in assignments.items()
         }

-    except ValueError as err:
+    except (ValueError, WorkflowError) as err:
         raise ValueError(errmsg) from err

Also add the missing import:

from snakemake.exceptions import (
    CliException,
    ResourceScopesException,
+   WorkflowError,
    print_exception,
)
🧹 Nitpick comments (5)
src/snakemake/resources.py (2)

835-851: Remove redundant int() casts.

The parse_size() and parse_timespan() functions already return int values, making the int() casts on lines 843 and 848 redundant.

Apply this diff:

         if name in SizedResources:
             try:
-                return max(int(math.ceil(parse_size(stripped) / 1e6)), 1)
+                return max(math.ceil(parse_size(stripped) / 1e6), 1)
             except InvalidSize as err:
                 raise WorkflowError(err_msg.format(unit="size in MB")) from err
         elif name in TimeResources:
             try:
-                return max(int(round(parse_timespan(stripped) / 60)), 1)
+                return max(round(parse_timespan(stripped) / 60), 1)
             except InvalidTimespan as err:
                 raise WorkflowError(err_msg.format(unit="minutes")) from err

878-885: Consider annotating DEFAULTS with ClassVar.

Static analysis suggests that mutable class attributes should be annotated with typing.ClassVar to indicate they are class-level rather than instance-level.

+from typing import ClassVar
+
 class Resources(Mapping[str, Resource]):
-    DEFAULTS = {
+    DEFAULTS: ClassVar[dict[str, dict[str, str]]] = {
         "full": {
             "mem": "min(max(2*input.size_mb, 1000), 8000)",
             "disk": "max(2*input.size_mb, 1000) if input else 50000",
             "tmpdir": "system_tmpdir",
         },
         "bare": {"tmpdir": "system_tmpdir"},
     }
src/snakemake/common/__init__.py (2)

107-113: Use exception chaining for context (raise ... from e).

Preserve the original cause instead of passing e as an argument.

-        raise NestedCoroutineError(
-            "Error running coroutine in event loop. Snakemake currently does not "
-            "support being executed from an already running event loop. "
-            "If you run Snakemake e.g. from a Jupyter notebook, make sure to spawn a "
-            "separate process for Snakemake.",
-            e,
-        )
+        raise NestedCoroutineError(
+            "Error running coroutine in event loop. Snakemake currently does not "
+            "support being executed from an already running event loop. "
+            "If you run Snakemake e.g. from a Jupyter notebook, make sure to spawn a "
+            "separate process for Snakemake."
+        ) from e

65-71: Nit: remove redundant int() around math.ceil/math.floor.

Both return int in modern Python.

-def mb_to_mib(mb: int):
-    return int(math.ceil(mb * 0.95367431640625))
+def mb_to_mib(mb: int):
+    return math.ceil(mb * 0.95367431640625)

-def mib_to_mb(mib: int):
-    return int(math.floor(mib / 0.95367431640625))
+def mib_to_mb(mib: int):
+    return math.floor(mib / 0.95367431640625)
src/snakemake/scheduling/job_scheduler.py (1)

103-111: Add a short comment clarifying why sized/human-friendly resources are filtered.

Using expand_sized=False and filtering str excludes human-friendly sized resources (e.g., mem/disk) from global limits, aligning with ResourceScopes (mem/disk default to local). A brief comment will prevent future “fixes” that reintroduce them into global constraints.

         self.global_resources = {
             name: (sys.maxsize if res is None else res)
             for name, res in workflow.global_resources.expand_items(
                 constraints={},
                 evaluate=None,
                 expand_sized=False,
             )
             if not isinstance(res, str)
         }
+        # Note: sized resources (mem/disk) are intentionally excluded here:
+        # they normalize to human‑readable strings and default to local scope,
+        # hence they must not constrain global scheduling.
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 80e19ad and e017870.

📒 Files selected for processing (22)
  • src/snakemake/api.py (2 hunks)
  • src/snakemake/cli.py (7 hunks)
  • src/snakemake/common/__init__.py (5 hunks)
  • src/snakemake/dag.py (2 hunks)
  • src/snakemake/exceptions.py (3 hunks)
  • src/snakemake/io/__init__.py (8 hunks)
  • src/snakemake/jobs.py (5 hunks)
  • src/snakemake/logging.py (2 hunks)
  • src/snakemake/resources.py (13 hunks)
  • src/snakemake/rules.py (7 hunks)
  • src/snakemake/scheduling/job_scheduler.py (4 hunks)
  • src/snakemake/script/__init__.py (1 hunks)
  • src/snakemake/settings/types.py (2 hunks)
  • src/snakemake/spawn_jobs.py (2 hunks)
  • src/snakemake/workflow.py (7 hunks)
  • test.txt (1 hunks)
  • tests/common.py (7 hunks)
  • tests/test_default_resources/Snakefile (1 hunks)
  • tests/test_github_issue3271/Snakefile_should_fail (1 hunks)
  • tests/test_group_jobs_resources/Snakefile (2 hunks)
  • tests/test_ioutils/Snakefile (1 hunks)
  • tests/tests.py (19 hunks)
🚧 Files skipped from review as they are similar to previous changes (5)
  • tests/test_ioutils/Snakefile
  • src/snakemake/settings/types.py
  • test.txt
  • src/snakemake/logging.py
  • src/snakemake/api.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py

⚙️ CodeRabbit configuration file

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

Files:

  • src/snakemake/script/__init__.py
  • src/snakemake/rules.py
  • src/snakemake/spawn_jobs.py
  • src/snakemake/exceptions.py
  • src/snakemake/scheduling/job_scheduler.py
  • tests/common.py
  • src/snakemake/dag.py
  • tests/tests.py
  • src/snakemake/cli.py
  • src/snakemake/workflow.py
  • src/snakemake/jobs.py
  • src/snakemake/resources.py
  • src/snakemake/common/__init__.py
  • src/snakemake/io/__init__.py
🧠 Learnings (23)
📚 Learning: 2025-10-30T15:39:38.224Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3813
File: tests/test_pathvars/Snakefile:22-23
Timestamp: 2025-10-30T15:39:38.224Z
Learning: In Snakemake, `<resources>` is a default pathvar that does not need to be explicitly defined in the pathvars section. It is available for use in rules and functions without requiring declaration.

Applied to files:

  • src/snakemake/script/__init__.py
  • src/snakemake/rules.py
  • tests/common.py
  • tests/tests.py
  • src/snakemake/cli.py
  • src/snakemake/workflow.py
  • src/snakemake/jobs.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/workflow.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-06T14:09:26.494Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/common/__init__.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2024-11-07T00:32:44.137Z
Learnt from: mbhall88
Repo: snakemake/snakemake PR: 3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.

Applied to files:

  • src/snakemake/rules.py
  • tests/test_default_resources/Snakefile
📚 Learning: 2024-08-13T16:22:09.641Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-08-13T16:22:09.641Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/io/__init__.py
📚 Learning: 2025-08-28T13:15:55.232Z
Learnt from: Hocnonsense
Repo: snakemake/snakemake PR: 3714
File: src/snakemake/modules.py:195-195
Timestamp: 2025-08-28T13:15:55.232Z
Learning: In src/snakemake/modules.py, the line `self.globals["__name__"] = namespace` is intentionally designed to set __name__ to None when namespace is None (i.e., when not inside a module context, for top-level workflows). This is correct behavior and should not be guarded.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/io/__init__.py
📚 Learning: 2024-11-12T20:22:54.184Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.

Applied to files:

  • src/snakemake/rules.py
  • tests/tests.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2985
File: tests/tests.py:2051-2051
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2025-08-07T21:51:08.043Z
Learnt from: pvandyken
Repo: snakemake/snakemake PR: 3421
File: src/snakemake/resources.py:606-607
Timestamp: 2025-08-07T21:51:08.043Z
Learning: In Snakemake's Resource.constrain method, the type comparison uses isinstance(self_val, type(other_val)) rather than direct type equality to allow strings to be compared with AnnotatedString objects. This polymorphic behavior is intentional and necessary for proper resource constraint functionality.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1250-1251
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid commenting on trailing commas in code reviews for the Snakemake repository as per the user's preference.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2024-10-04T16:12:18.927Z
Learnt from: lczech
Repo: snakemake/snakemake PR: 3113
File: snakemake/scheduler.py:912-914
Timestamp: 2024-10-04T16:12:18.927Z
Learning: In `snakemake/scheduler.py`, avoid suggesting the use of `asyncio.gather` in the `jobs_rewards` method due to overhead concerns and the need for immediate results.

Applied to files:

  • src/snakemake/scheduling/job_scheduler.py
  • src/snakemake/jobs.py
📚 Learning: 2025-07-29T14:53:04.598Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3676
File: src/snakemake/cli.py:0-0
Timestamp: 2025-07-29T14:53:04.598Z
Learning: In the Snakemake repository, deprecated scheduler interfaces (like scheduler_ilp_solver, --scheduler-solver-path, --scheduler-greediness) should be kept for backward compatibility during the migration to the plugin-based scheduler system, rather than being removed immediately.

Applied to files:

  • src/snakemake/scheduling/job_scheduler.py
📚 Learning: 2024-10-13T14:10:37.796Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the `snakemake/cli.py` file, `ExecutorPluginRegistry()` is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.

Applied to files:

  • src/snakemake/scheduling/job_scheduler.py
📚 Learning: 2024-10-06T14:09:54.370Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.

Applied to files:

  • tests/test_group_jobs_resources/Snakefile
  • tests/test_github_issue3271/Snakefile_should_fail
  • tests/tests.py
  • tests/test_default_resources/Snakefile
📚 Learning: 2024-10-14T09:42:11.571Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.

Applied to files:

  • src/snakemake/dag.py
📚 Learning: 2024-11-12T12:08:20.342Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.

Applied to files:

  • src/snakemake/dag.py
📚 Learning: 2024-10-13T14:29:38.902Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3135
File: snakemake/profiles.py:54-55
Timestamp: 2024-10-13T14:29:38.902Z
Learning: In the Snakemake test suite, tests for configuration keys like 'group-components' may already be covered within existing tests for 'groups'. Ensure to verify this before suggesting additional tests.

Applied to files:

  • tests/tests.py
📚 Learning: 2025-01-17T12:00:09.368Z
Learnt from: leoschwarz
Repo: snakemake/snakemake PR: 3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.

Applied to files:

  • tests/test_default_resources/Snakefile
📚 Learning: 2024-09-25T10:55:53.963Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: 2024-12-14T13:11:32.988Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.

Applied to files:

  • src/snakemake/resources.py
🧬 Code graph analysis (13)
src/snakemake/script/__init__.py (3)
src/snakemake/jobs.py (2)
  • resources (481-498)
  • resources (1519-1533)
src/snakemake/workflow.py (1)
  • resources (2230-2235)
src/snakemake/io/__init__.py (1)
  • ResourceList (1995-1996)
src/snakemake/rules.py (4)
src/snakemake/io/__init__.py (2)
  • ResourceList (1995-1996)
  • get (1915-1920)
src/snakemake/jobs.py (13)
  • resources (481-498)
  • resources (1519-1533)
  • wildcards (329-330)
  • wildcards (333-334)
  • input (337-338)
  • input (341-342)
  • input (1540-1545)
  • attempt (471-472)
  • attempt (475-478)
  • attempt (1732-1733)
  • attempt (1736-1741)
  • name (1292-1293)
  • name (1670-1674)
src/snakemake/resources.py (7)
  • Resource (461-851)
  • Resources (854-1158)
  • evaluate (552-569)
  • constrain (571-602)
  • get (1105-1109)
  • value (536-546)
  • expand_items (1111-1158)
src/snakemake/exceptions.py (5)
  • ResourceConstraintError (637-638)
  • ResourceValidationError (633-634)
  • NestedCoroutineError (551-552)
  • ResourceConversionError (641-655)
  • InputFunctionException (250-262)
src/snakemake/exceptions.py (1)
src/snakemake/resources.py (1)
  • value (536-546)
src/snakemake/scheduling/job_scheduler.py (2)
src/snakemake/resources.py (3)
  • expand_items (1111-1158)
  • evaluate (552-569)
  • value (536-546)
src/snakemake/workflow.py (2)
  • name (2314-2319)
  • resources (2230-2235)
tests/common.py (1)
src/snakemake/resources.py (5)
  • ResourceScopes (1164-1302)
  • Resources (854-1158)
  • from_mapping (1060-1080)
  • parse (982-1057)
  • defaults (1205-1206)
src/snakemake/dag.py (1)
src/snakemake/scheduling/job_scheduler.py (1)
  • stats (187-188)
tests/tests.py (3)
src/snakemake/exceptions.py (2)
  • ResourceConversionError (641-655)
  • ResourceDuplicationError (658-665)
src/snakemake/resources.py (6)
  • GroupResources (62-444)
  • is_ordinary_string (1305-1338)
  • Resources (854-1158)
  • from_mapping (1060-1080)
  • parse (982-1057)
  • defaults (1205-1206)
tests/common.py (1)
  • run (155-521)
src/snakemake/cli.py (3)
src/snakemake/resources.py (10)
  • Resource (461-851)
  • Resources (854-1158)
  • from_cli_expression (724-769)
  • args (929-934)
  • parse (982-1057)
  • items (1094-1095)
  • value (536-546)
  • default (944-962)
  • parser_factory (965-979)
  • defaults (1205-1206)
src/snakemake/workflow.py (1)
  • rule (1806-2046)
src/snakemake/common/__init__.py (1)
  • parse_key_value_arg (73-80)
src/snakemake/workflow.py (2)
src/snakemake/exceptions.py (3)
  • ResourceDuplicationError (658-665)
  • ResourceConversionError (641-655)
  • ResourceValidationError (633-634)
src/snakemake/resources.py (8)
  • Resources (854-1158)
  • value (536-546)
  • copy (1101-1103)
  • get (1105-1109)
  • args (929-934)
  • from_mapping (1060-1080)
  • update (1082-1086)
  • update (1299-1302)
src/snakemake/jobs.py (4)
src/snakemake/io/__init__.py (2)
  • ResourceList (1995-1996)
  • items (1870-1872)
src/snakemake/workflow.py (2)
  • resources (2230-2235)
  • resource_scopes (535-536)
src/snakemake/resources.py (4)
  • GroupResources (62-444)
  • Resources (854-1158)
  • items (1094-1095)
  • args (929-934)
src/snakemake/exceptions.py (1)
  • ResourceInsufficiencyError (604-623)
src/snakemake/resources.py (6)
src/snakemake/common/__init__.py (3)
  • get_input_function_aux_params (311-323)
  • mb_to_mib (65-66)
  • mib_to_mb (69-70)
src/snakemake/exceptions.py (11)
  • NestedCoroutineError (551-552)
  • ResourceConstraintError (637-638)
  • ResourceDuplicationError (658-665)
  • ResourceError (600-601)
  • ResourceInsufficiencyError (604-623)
  • ResourceScopesException (626-630)
  • ResourceConversionError (641-655)
  • ResourceValidationError (633-634)
  • is_file_not_found_error (693-700)
  • format (651-655)
  • format_evaluated (643-648)
src/snakemake/common/tbdstring.py (1)
  • TBDString (3-96)
src/snakemake/io/__init__.py (11)
  • AnnotatedString (957-977)
  • items (1870-1872)
  • get (1915-1920)
  • size (176-177)
  • size (651-661)
  • size (1963-1964)
  • is_callable (324-325)
  • is_callable (968-969)
  • is_callable (1186-1191)
  • keys (1903-1904)
  • update (1845-1851)
src/snakemake/jobs.py (22)
  • jobs (1377-1378)
  • jobs (1381-1382)
  • Job (183-1320)
  • resources (481-498)
  • resources (1519-1533)
  • name (1292-1293)
  • name (1670-1674)
  • group (461-464)
  • group (467-468)
  • wildcards (329-330)
  • wildcards (333-334)
  • input (337-338)
  • input (341-342)
  • input (1540-1545)
  • attempt (471-472)
  • attempt (475-478)
  • attempt (1732-1733)
  • attempt (1736-1741)
  • threads (423-424)
  • threads (1725-1726)
  • is_local (1020-1028)
  • is_local (1681-1682)
src/snakemake/rules.py (8)
  • name (152-153)
  • name (156-157)
  • group (182-191)
  • group (194-195)
  • evaluate (1093-1120)
  • input (291-292)
  • input (1403-1425)
  • add (1356-1360)
src/snakemake/common/__init__.py (1)
src/snakemake/exceptions.py (1)
  • NestedCoroutineError (551-552)
src/snakemake/io/__init__.py (2)
src/snakemake/gui.py (1)
  • index (84-95)
src/snakemake/resources.py (1)
  • items (1094-1095)
🪛 Ruff (0.14.3)
src/snakemake/rules.py

1106-1106: Unused lambda argument: e

(ARG005)


1133-1133: Avoid specifying long messages outside the exception class

(TRY003)


1146-1150: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1146-1150: Avoid specifying long messages outside the exception class

(TRY003)


1159-1159: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/dag.py

18-18: Redefinition of unused Dict from line 18

Remove definition: Dict

(F811)

src/snakemake/cli.py

232-232: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


244-244: Avoid specifying long messages outside the exception class

(TRY003)

src/snakemake/workflow.py

1886-1889: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1886-1889: Avoid specifying long messages outside the exception class

(TRY003)


1894-1894: Avoid specifying long messages outside the exception class

(TRY003)

src/snakemake/jobs.py

1530-1532: Avoid specifying long messages outside the exception class

(TRY003)

src/snakemake/resources.py

386-386: zip() without an explicit strict= parameter

Add explicit value for parameter strict=

(B905)


399-399: zip() without an explicit strict= parameter

Add explicit value for parameter strict=

(B905)


542-545: Avoid specifying long messages outside the exception class

(TRY003)


569-569: Use raise without specifying exception name

Remove exception name

(TRY201)


648-650: Avoid specifying long messages outside the exception class

(TRY003)


742-742: Unused function argument: wildcards

(ARG001)


746-746: Unused function argument: rulename

(ARG001)


757-757: Unused function argument: wildcards

(ARG001)


757-757: Unused function argument: rulename

(ARG001)


773-773: Unused static method argument: name

(ARG004)


780-789: Use of possibly insecure function; consider using ast.literal_eval

(S307)


795-795: Do not catch blind exception: Exception

(BLE001)


804-818: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


804-818: Avoid specifying long messages outside the exception class

(TRY003)


843-843: Value being cast to int is already an integer

Remove unnecessary int call

(RUF046)


848-848: Value being cast to int is already an integer

Remove unnecessary int call

(RUF046)


878-885: Mutable class attributes should be annotated with typing.ClassVar

(RUF012)


919-919: Avoid specifying long messages outside the exception class

(TRY003)


941-941: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


941-941: Avoid specifying long messages outside the exception class

(TRY003)


955-955: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


955-955: Avoid specifying long messages outside the exception class

(TRY003)


1010-1013: Avoid specifying long messages outside the exception class

(TRY003)


1048-1050: Avoid specifying long messages outside the exception class

(TRY003)


1187-1191: Avoid specifying long messages outside the exception class

(TRY003)

src/snakemake/common/__init__.py

66-66: Value being cast to int is already an integer

Remove unnecessary int call

(RUF046)


70-70: Value being cast to int is already an integer

Remove unnecessary int call

(RUF046)


107-113: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


107-113: Avoid specifying long messages outside the exception class

(TRY003)

src/snakemake/io/__init__.py

1-1: The file is executable but no shebang is present

(EXE002)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (46)
  • GitHub Check: tests (10, ubuntu-latest, py313)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (10, windows-2022, py313)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (9, windows-2022, py313)
  • GitHub Check: tests (8, windows-2022, py313)
  • GitHub Check: tests (9, ubuntu-latest, py313)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (7, windows-2022, py313)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (7, ubuntu-latest, py313)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (6, windows-2022, py313)
  • GitHub Check: tests (8, ubuntu-latest, py313)
  • GitHub Check: tests (6, macos-latest, py313)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (6, ubuntu-latest, py313)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py313)
  • GitHub Check: tests (5, ubuntu-latest, py313)
  • GitHub Check: tests (5, windows-2022, py313)
  • GitHub Check: tests (4, windows-2022, py313)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (4, macos-latest, py313)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py313)
  • GitHub Check: tests (3, windows-2022, py313)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (3, macos-latest, py313)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py313)
  • GitHub Check: tests (1, windows-2022, py313)
  • GitHub Check: tests (2, macos-latest, py313)
  • GitHub Check: tests (2, windows-2022, py313)
  • GitHub Check: tests (1, macos-latest, py313)
  • GitHub Check: tests (1, ubuntu-latest, py313)
  • GitHub Check: tests (1, ubuntu-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py311)
  • GitHub Check: apidocs
🔇 Additional comments (33)
src/snakemake/resources.py (4)

64-269: LGTM! Well-typed refactor of resource grouping logic.

The addition of comprehensive type annotations and the migration to the typed Resources API improves type safety. The error handling with ResourceInsufficiencyError is appropriate.


461-716: LGTM! Well-designed Resource class with proper validation.

The Resource class provides a clean abstraction for resource values with appropriate validation, evaluation, and conversion methods. The design handles evaluable resources, human-friendly parsing, and unit conversions elegantly.


854-1159: LGTM! Robust Mapping-based Resources container.

The Resources class provides a clean, type-safe container for resource management with proper normalization of sized resources and good support for default resources and parsing.


1164-1303: LGTM! Clean ResourceScopes implementation.

The ResourceScopes class properly handles resource scope management with appropriate normalization of size variants and clear testing methods.

tests/test_group_jobs_resources/Snakefile (2)

18-18: LGTM! Tests human-friendly resource format.

The change from mem_mb=20000 to mem="20 GB" appropriately tests the new human-friendly resource parsing feature.


49-49: LGTM! Tests mebibyte resource handling.

The addition of mem_mib=10000 tests the new mebibyte resource suffix functionality.

src/snakemake/spawn_jobs.py (1)

97-118: LGTM! Simplified resource argument formatting.

The direct use of value.raw aligns with the new Resource API and removes unnecessary indirection from the previous get_orig_arg helper. The change maintains the same behavior while being more straightforward.

src/snakemake/script/__init__.py (1)

113-113: LGTM! Type annotation updated for compatibility.

The change from io_.Resources to io_.ResourceList aligns with the PR's renaming strategy to preserve plugin compatibility while introducing the new typed Resources API.

tests/test_github_issue3271/Snakefile_should_fail (1)

7-7: LGTM! Tests input.size_mb in resource function.

The change from attempt * 1000 to input.size_mb appropriately tests the evaluation of resources that depend on input file sizes, which is a key feature of the new resource system.

src/snakemake/dag.py (1)

3189-3214: LGTM: stats() implementation and return typing are consistent.

Counts and message formatting look correct; sorting by rule.name is clear.

tests/test_default_resources/Snakefile (1)

9-14: LGTM: exercising mem/disk normalization across mem, mem_mb, mem_mib.

The guards correctly validate human-friendly and sized variants before touching outputs.

Also applies to: 21-26

tests/common.py (1)

342-356: LGTM: resources/default_resources normalization to Resources instances.

Resources.from_mapping(...) and Resources.parse(..., defaults="full", allow_expressions=True) match the new API and keep tests aligned with CLI semantics.

src/snakemake/io/__init__.py (1)

1995-1996: Rename verified—no stale imports of old Resources from snakemake.io detected.

All usages of Resources correctly import from snakemake.resources: cli.py, workflow.py, and tests/common.py. The old Resources class has been removed from snakemake.io, replaced by the ResourceList placeholder shown in the code snippet. Refactoring is complete with no import issues.

src/snakemake/rules.py (1)

1084-1163: LGTM! Well-structured resource evaluation with proper error handling.

The refactored expand_resources method correctly:

  • Evaluates resources through the new Resource API
  • Applies thread constraints from max_threads and global _cores
  • Uses Resources.expand_items for standardized expansion
  • Handles NestedCoroutineError for async evaluation conflicts
  • Catches and wraps ResourceConversionError with helpful context
  • Returns ResourceList as documented in the AI summary

The error handling appropriately catches specific exceptions without chaining (per codebase conventions).

src/snakemake/jobs.py (2)

108-116: LGTM! Proper filtering of TBDString resources and scope-aware scheduling.

The updated _get_scheduler_resources correctly:

  • Filters out TBDString resources (unevaluated resources)
  • Applies scope filtering for non-local execution via resource_scopes.is_local()
  • Maintains consistent resource dictionary construction

1529-1533: LGTM! Proper exception handling for group job resources.

The exception handling correctly:

  • Catches both WorkflowError and ResourceInsufficiencyError
  • Wraps with contextual group ID information
  • Uses from err for proper exception chaining (note: the comment about avoiding chaining applies to hiding business logic; here we're preserving the error context appropriately)
tests/tests.py (7)

688-690: LGTM! Test verifies expression evaluation in thread overwriting.

The test correctly exercises the --set-threads CLI option with an expression ('10*2'), validating that thread resource expressions are properly evaluated.


906-907: LGTM! Simplified assertion aligns with new resource API.

The simplified assert_resources directly compares resource dictionaries, which is appropriate now that resources are returned as ResourceList (a dictionary-like structure).


938-963: LGTM! Comprehensive test of human-readable resource handling in groups.

The test correctly:

  • Exercises group job resource aggregation via spy on GroupResources.basic_layered
  • Verifies that mem_mb=60000 is properly expanded to include mem (human-readable) and mem_mib variants
  • Validates all resource values including the correctly computed mem_mib=57221
  • Uses default_resources=["mem_mb=0"] to set baseline defaults

966-989: LGTM! Validates resources can be provided in mebibytes.

This test correctly verifies that users can specify resources using mem_mib instead of mem_mb, and that the system properly converts and expands all variants (mem, mem_mb, mem_mib).


1028-1047: LGTM! Validates duplicate resource detection.

The parametrized test correctly verifies that providing multiple variants of the same sized resource (e.g., mem and mem_mb together) raises ResourceDuplicationError, preventing ambiguous resource specifications.


1050-1057: LGTM! Validates human-readable resources cannot use suffixed names.

The test correctly verifies that suffixed resources like mem_mb must be integers and cannot accept human-readable strings like "40 Gb", raising ResourceConversionError as expected.


1542-1567: LGTM! Comprehensive tests for human-readable default resources.

The tests correctly validate:

  1. test_default_resources_humanreadable: Human-readable defaults (e.g., mem='1 Gb') work correctly
  2. test_default_resources_humanreadable_disallowed_for_suffixed: Suffixed resources (mem_mb, disk_mib) cannot use human-readable strings (should fail)
  3. test_default_resources_mebibytes: Mebibyte decimal values are handled correctly

These align with the PR's goal of standardizing human-readable resource handling.

src/snakemake/cli.py (4)

110-123: LGTM: Threads wrapper correctly handles integers and expressions.

The wrapper properly distinguishes between integer values (creating Resource("_cores", orig_value)) and expression strings (using Resource.from_cli_expression), addressing the concern from previous reviews.


223-245: LGTM: Integer parsing and wrapper application is correct.

The function properly:

  • Attempts integer conversion and validates non-negativity
  • Falls back to wrapper for non-integer values when wrapper is provided
  • Includes exception chaining at line 244 where appropriate

The static analysis warning (B904) at line 232 about missing exception chaining is intentional per the codebase's backward compatibility requirements.

Based on learnings


533-534: LGTM: Resources initialization for --resources is correct.

Properly uses Resources() as the default and delegates parsing to Resources.parser_factory(allow_expressions=False), which correctly prevents expressions in global resource constraints.


612-614: LGTM: Default resources parser correctly configured.

Uses Resources.parser_factory(defaults="full", allow_expressions=True), which appropriately:

  • Enables Python expressions for default resources (e.g., 2*input.size_mb)
  • Includes full default values (mem, disk, tmpdir)
  • Aligns with the PR's goal to standardize resource handling
src/snakemake/workflow.py (6)

72-74: LGTM: Resource imports correctly updated.

The new exception types (ResourceDuplicationError, ResourceConversionError, ResourceValidationError) and Resources container class are properly imported to support the centralized resource handling.

Also applies to: 132-132


182-184: LGTM: Global resources initialization is correct.

Properly initializes global_resources from resource_settings.resources (now a Resources object) and adds the required _cores and _nodes entries.


630-631: LGTM: Property access on Resource objects is correct.

The .value property correctly retrieves concrete resource values. Note that per the Resource class implementation (lines ~535-545 in src/snakemake/resources.py), .value raises ValueError if the resource is still evaluable (unevaluated). Since _cores and _nodes are set as concrete values in __post_init__, this is safe.

Also applies to: 634-635


687-687: LGTM: Return type annotation improves type safety.

The added -> Rule return type annotation enhances code clarity and enables better static analysis.


1870-1909: LGTM: Resource handling is comprehensive and correct.

The implementation properly:

  • Initializes rule resources from defaults or creates empty Resources()
  • Sets the required _nodes resource
  • Handles thread overrides and validation (lines 1877-1889)
  • Processes rule-specific resources via Resources.from_mapping (line 1896)
  • Catches and converts all resource-specific exceptions (ResourceDuplicationError, ResourceConversionError, ResourceValidationError) into RuleException with appropriate context (lines 1897-1903)
  • Applies resource overwrites from CLI/API settings (lines 1907-1908)

The static analysis warnings about exception chaining (B904) and message formatting (TRY003) are expected—exception chaining is intentionally avoided per backward compatibility requirements.

Based on learnings


2010-2010: LGTM: Handover rule resource allocation is appropriate.

Correctly provides handover rules with all available global resources via rule.resources.update(self.global_resources), enabling them to manage resource allocation for their spawned jobs.

@johanneskoester
Copy link
Copy Markdown
Contributor

@pvandyken, see my message on discord.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (3)
src/snakemake/dag.py (1)

18-18: Remove duplicate Dict import to satisfy Ruff and avoid F811.

The typing import currently includes Dict twice:

from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Union, Dict

This triggers Ruff F811 and is unnecessary. Please drop the trailing , Dict:

-from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Union, Dict
+from typing import Dict, Iterable, List, Mapping, Optional, Set, Tuple, Union
tests/common.py (1)

507-511: Still need to guard against exception is None when expecting a specific failure.

shouldfail can be an exception type, but exception may remain None (e.g. shell-based runs only set success = False), so raise exception will itself raise a TypeError instead of the expected exception, making the assertion misleading. This is the same issue noted in a prior review.

Apply this minimal guard to fail clearly if no exception was captured:

-    if shouldfail is not False:
-        assert not success, "expected error on execution"
-        if shouldfail is not True:
-            with pytest.raises(shouldfail):
-                raise exception
+    if shouldfail is not False:
+        assert not success, "expected error on execution"
+        if shouldfail is not True:
+            if exception is None:
+                pytest.fail("Expected failure, but no exception was captured.")
+            with pytest.raises(shouldfail):
+                raise exception
src/snakemake/cli.py (1)

168-194: parse_set_resources needs more robust validation and error wrapping.

Right now:

  • Entries without a ":" will raise a bare ValueError during unpacking (rule, assign = entry.split(...)), bypassing the try/except and yielding an unfriendly traceback.
  • Resources.parse(...) can raise WorkflowError (e.g. invalid names or negative values), which currently escapes instead of being converted into the CLI-friendly ValueError(errmsg).
  • The error message text is slightly off grammatically (“positive integer a quoted string”).

To harden this, I suggest:

-from snakemake.exceptions import (
-    CliException,
-    ResourceScopesException,
-    print_exception,
-)
+from snakemake.exceptions import (
+    CliException,
+    ResourceScopesException,
+    WorkflowError,
+    print_exception,
+)
@@
-def parse_set_resources(args: List[str] | None) -> Dict[str, Resources]:
-    errmsg = (
-        "Invalid resource definition: entries have to be defined as "
-        "RULE:RESOURCE=VALUE, with VALUE being a positive integer a quoted string, or "
-        "a Python expression (e.g. min(max(2*input.size_mb, 1000), 8000))."
-    )
+def parse_set_resources(args: List[str] | None) -> Dict[str, Resources]:
+    errmsg = (
+        "Invalid resource definition: entries have to be defined as "
+        "RULE:RESOURCE=VALUE, with VALUE being a positive integer, a quoted string, "
+        "or a Python expression (e.g. min(max(2*input.size_mb, 1000), 8000))."
+    )
@@
-    if args is None:
-        return {}
-
-    assignments: Dict[str, List[str]] = defaultdict(list)
-
-    for entry in args:
-        rule, assign = entry.split(":", maxsplit=1)
-        assignments[rule].append(assign)
-
-    try:
-        return {
-            rule: Resources.parse(assigns, allow_expressions=True)
-            for rule, assigns in assignments.items()
-        }
-
-    except ValueError as err:
-        raise ValueError(errmsg) from err
+    if args is None:
+        return {}
+
+    assignments: Dict[str, List[str]] = defaultdict(list)
+    try:
+        for entry in args:
+            if ":" not in entry:
+                raise ValueError(f"Unparsable value: {repr(entry)}")
+            rule, assign = entry.split(":", maxsplit=1)
+            assignments[rule].append(assign)
+
+        return {
+            rule: Resources.parse(assigns, allow_expressions=True)
+            for rule, assigns in assignments.items()
+        }
+    except (ValueError, WorkflowError) as err:
+        raise ValueError(errmsg) from err

This keeps CLI behavior predictable and ensures all parsing failures are reported via a uniform ValueError message.

🧹 Nitpick comments (8)
src/snakemake/resources.py (5)

223-232: Align ResourceError payload with ResourceInsufficiencyError.excess_resources expectations.

Here _get_layers raises ResourceError with a joined string, and you pass err.args[0] as excess_resources into ResourceInsufficiencyError, whose signature/documentation describes excess_resources as a sequence of resources.

If you intend excess_resources to be structured (e.g. one entry per offending resource), consider either:

  • Raising ResourceError with a list of offending specs and passing that list through, or
  • Relaxing the ResourceInsufficiencyError type hint/docstring to explicitly allow a single formatted string.

Functionality is fine; this is mainly about keeping the error API and types self-consistent.


448-455: SizedResources/TimeResources sets are clear; consider documenting them as shared constants.

The SizedResources and TimeResources sets are central to parsing and normalization across Resource, Resources, and ResourceScopes. It may be worth a brief module‑level comment (or docstring mention) that these define the canonical sized/time resource families so future additions stay consistent.

No functional issue; just a small maintainability aid.


879-886: Resources.DEFAULTS + default() interaction aligns with the new Resource API.

  • DEFAULTS["full"] and "bare" are expressed as CLI‑style expressions, which default() wraps via Resource.from_cli_expression.
  • Resources.__init__ then normalizes sized resources so that externally you get _mb-suffixed entries for sizes, as expected.
  • The docstring for default() still correctly describes that the resulting Resources includes pre‑set memory/disk/tmpdir defaults.

The mutable DEFAULTS dict is effectively constant here; if you want to make that explicit for type checkers, annotating it as ClassVar[dict[str, dict[str, str]]] would be a nice‑to‑have, but not required.


945-963: Resources.default error message still mentions DefaultResources.

The logic is fine, but the error message still refers to the old DefaultResources name:

raise ValueError(f"Unexpected mode for DefaultResources: {mode}")

Given the class is now Resources, consider updating the wording to avoid confusion for API users.


1086-1114: Resources.update and get semantics are slightly non-standard but intentional.

  • update accepts either another Resources or a plain mapping, in the latter case normalizing via from_mapping, which is convenient.
  • get returning Resource("", None) instead of None keeps callers from having to special-case None before calling .value or .constrain().

Since this is a new public API, it may be worth documenting the get() behavior explicitly to avoid surprises for consumers expecting a dict-like None default.

src/snakemake/jobs.py (2)

103-120: Scheduler resource filtering correctly respects scopes and excludes placeholders.

The updated _get_scheduler_resources() now (a) drops TBDString entries, and (b) for non-local execution, omits resources whose scope is local via resource_scopes.is_local(k). This ensures only applicable, fully-evaluated resources are exposed to the scheduler.

If you want to make debugging easier in future, consider documenting (in a short comment) that _scheduler_resources intentionally excludes local-scoped and unevaluated values.


1374-1378: Grouped resource aggregation and exposure via GroupJob.resources look sound.

GroupJob.__init__ clearly treats global_resources as a Resources constraint set, and resources now:

  • Computes grouped resources via GroupResources.basic_layered(...) with appropriate flags for local execution and additive/sort-by keys.
  • Wraps WorkflowError and ResourceInsufficiencyError with group-id context while chaining the original exception.
  • Returns a ResourceList(fromdict=self._resources), giving executors a uniform, Namedlist-like view.

You might later consider caching the constructed ResourceList alongside _resources if repeated access to GroupJob.resources becomes performance-sensitive.

Also applies to: 1539-1557

src/snakemake/io/__init__.py (1)

1747-1756: Namedlist generics and ResourceList definition align with new resource surface.

Making Namedlist generic over key and item types, adding self-typed helper methods, and typing items() as Iterator[Tuple[_TNamedKeys, _TNamedList]] strengthens static checking. Defining ResourceList(Namedlist) here cleanly exposes the list-like {resources} object consumed by jobs and format contexts.

If you ever specialize ResourceList further (e.g., to expose convenience accessors), consider parameterizing it explicitly (e.g., ResourceList(Namedlist[str, Any])) for even clearer intent, though this is not required for correctness.

Also applies to: 1763-1769, 1869-1871, 2006-2007

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e017870 and d23bb09.

📒 Files selected for processing (22)
  • src/snakemake/api.py (2 hunks)
  • src/snakemake/cli.py (7 hunks)
  • src/snakemake/common/__init__.py (5 hunks)
  • src/snakemake/dag.py (2 hunks)
  • src/snakemake/exceptions.py (3 hunks)
  • src/snakemake/io/__init__.py (8 hunks)
  • src/snakemake/jobs.py (5 hunks)
  • src/snakemake/logging.py (2 hunks)
  • src/snakemake/resources.py (13 hunks)
  • src/snakemake/rules.py (7 hunks)
  • src/snakemake/scheduling/job_scheduler.py (4 hunks)
  • src/snakemake/script/__init__.py (1 hunks)
  • src/snakemake/settings/types.py (3 hunks)
  • src/snakemake/spawn_jobs.py (2 hunks)
  • src/snakemake/workflow.py (7 hunks)
  • test.txt (1 hunks)
  • tests/common.py (7 hunks)
  • tests/test_default_resources/Snakefile (1 hunks)
  • tests/test_github_issue3271/Snakefile_should_fail (1 hunks)
  • tests/test_group_jobs_resources/Snakefile (2 hunks)
  • tests/test_ioutils/Snakefile (1 hunks)
  • tests/tests.py (19 hunks)
🚧 Files skipped from review as they are similar to previous changes (6)
  • src/snakemake/logging.py
  • tests/test_ioutils/Snakefile
  • tests/test_github_issue3271/Snakefile_should_fail
  • tests/tests.py
  • src/snakemake/exceptions.py
  • tests/test_group_jobs_resources/Snakefile
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py

⚙️ CodeRabbit configuration file

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

Files:

  • src/snakemake/scheduling/job_scheduler.py
  • tests/common.py
  • src/snakemake/script/__init__.py
  • src/snakemake/dag.py
  • src/snakemake/spawn_jobs.py
  • src/snakemake/settings/types.py
  • src/snakemake/cli.py
  • src/snakemake/jobs.py
  • src/snakemake/workflow.py
  • src/snakemake/common/__init__.py
  • src/snakemake/api.py
  • src/snakemake/rules.py
  • src/snakemake/io/__init__.py
  • src/snakemake/resources.py
🧠 Learnings (21)
📚 Learning: 2024-10-04T16:12:18.927Z
Learnt from: lczech
Repo: snakemake/snakemake PR: 3113
File: snakemake/scheduler.py:912-914
Timestamp: 2024-10-04T16:12:18.927Z
Learning: In `snakemake/scheduler.py`, avoid suggesting the use of `asyncio.gather` in the `jobs_rewards` method due to overhead concerns and the need for immediate results.

Applied to files:

  • src/snakemake/scheduling/job_scheduler.py
  • src/snakemake/jobs.py
📚 Learning: 2024-10-13T14:10:37.796Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3114
File: snakemake/cli.py:708-708
Timestamp: 2024-10-13T14:10:37.796Z
Learning: In the `snakemake/cli.py` file, `ExecutorPluginRegistry()` is a singleton class, so multiple instantiations are acceptable and do not cause unnecessary overhead.

Applied to files:

  • src/snakemake/scheduling/job_scheduler.py
📚 Learning: 2025-10-30T15:39:38.224Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3813
File: tests/test_pathvars/Snakefile:22-23
Timestamp: 2025-10-30T15:39:38.224Z
Learning: In Snakemake, `<resources>` is a default pathvar that does not need to be explicitly defined in the pathvars section. It is available for use in rules and functions without requiring declaration.

Applied to files:

  • tests/common.py
  • src/snakemake/script/__init__.py
  • src/snakemake/spawn_jobs.py
  • src/snakemake/cli.py
  • src/snakemake/jobs.py
  • src/snakemake/workflow.py
  • src/snakemake/api.py
  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-14T09:42:11.571Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3140
File: snakemake/dag.py:1308-1308
Timestamp: 2024-10-14T09:42:11.571Z
Learning: In `snakemake/dag.py`, avoid flagging missing lines or indentation issues when there is no clear syntax or logical error to prevent false positives.

Applied to files:

  • src/snakemake/dag.py
📚 Learning: 2024-11-12T12:08:20.342Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3148
File: snakemake/dag.py:1332-1336
Timestamp: 2024-11-12T12:08:20.342Z
Learning: In `snakemake/dag.py`, when code is outdated and will disappear upon resolving merge conflicts, avoid making code review suggestions on that code.

Applied to files:

  • src/snakemake/dag.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: In the `snakemake` codebase, exception chaining (using `raise ... from ...`) is avoided due to backward compatibility reasons. Do not suggest using exception chaining in `snakemake` codebase.

Applied to files:

  • src/snakemake/workflow.py
  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-06T14:09:54.370Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.

Applied to files:

  • tests/test_default_resources/Snakefile
📚 Learning: 2025-01-17T12:00:09.368Z
Learnt from: leoschwarz
Repo: snakemake/snakemake PR: 3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.

Applied to files:

  • tests/test_default_resources/Snakefile
📚 Learning: 2024-10-06T14:09:26.494Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: snakemake/deployment/conda.py:0-0
Timestamp: 2024-10-06T14:09:26.494Z
Learning: In `snakemake/deployment/conda.py`, the exception handling in the `_get_version` method is intentional. The method raises a `WorkflowError` when the version cannot be determined, and this behavior is desired. Do not suggest modifying this exception handling.

Applied to files:

  • src/snakemake/common/__init__.py
  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-29T09:26:26.636Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3171
File: snakemake/cli.py:106-145
Timestamp: 2024-10-29T09:26:26.636Z
Learning: Avoid adding input validation or error handling that unnecessarily complicates the code in the `snakemake` codebase, especially when the cases handled don't make sense.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2024-11-07T00:32:44.137Z
Learnt from: mbhall88
Repo: snakemake/snakemake PR: 3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2024-08-13T16:22:09.641Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1147-1147
Timestamp: 2024-08-13T16:22:09.641Z
Learning: Avoid suggesting type annotations for functions that are inside methods in the Snakemake codebase.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/io/__init__.py
📚 Learning: 2025-08-28T13:15:55.232Z
Learnt from: Hocnonsense
Repo: snakemake/snakemake PR: 3714
File: src/snakemake/modules.py:195-195
Timestamp: 2025-08-28T13:15:55.232Z
Learning: In src/snakemake/modules.py, the line `self.globals["__name__"] = namespace` is intentionally designed to set __name__ to None when namespace is None (i.e., when not inside a module context, for top-level workflows). This is correct behavior and should not be guarded.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:0-0
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake project, type annotations are not necessary for internal helper functions.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/io/__init__.py
📚 Learning: 2024-11-12T20:22:54.184Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3204
File: tests/test_cores_cluster/qsub:1-6
Timestamp: 2024-11-12T20:22:54.184Z
Learning: In the Snakemake codebase, the `tests/test_cores_cluster/qsub` script is a dummy script for testing, and input validation and error handling are not required in such scripts.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2985
File: tests/tests.py:2051-2051
Timestamp: 2024-10-08T17:41:54.542Z
Learning: In the Snakemake repository, avoid suggesting return type annotations for test functions.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/io/__init__.py
  • src/snakemake/resources.py
📚 Learning: 2025-08-07T21:51:08.043Z
Learnt from: pvandyken
Repo: snakemake/snakemake PR: 3421
File: src/snakemake/resources.py:606-607
Timestamp: 2025-08-07T21:51:08.043Z
Learning: In Snakemake's Resource.constrain method, the type comparison uses isinstance(self_val, type(other_val)) rather than direct type equality to allow strings to be compared with AnnotatedString objects. This polymorphic behavior is intentional and necessary for proper resource constraint functionality.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-08-27T20:15:58.699Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3026
File: snakemake/assets/__init__.py:0-0
Timestamp: 2024-08-27T20:15:58.699Z
Learning: Do not suggest type annotations for `cls` arguments in class methods when reviewing code for the Snakemake project.

Applied to files:

  • src/snakemake/rules.py
  • src/snakemake/resources.py
📚 Learning: 2024-10-08T17:41:54.542Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3014
File: snakemake/workflow.py:1250-1251
Timestamp: 2024-10-08T17:41:54.542Z
Learning: Avoid commenting on trailing commas in code reviews for the Snakemake repository as per the user's preference.

Applied to files:

  • src/snakemake/rules.py
📚 Learning: 2024-09-25T10:55:53.963Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3103
File: snakemake/persistence.py:263-275
Timestamp: 2024-09-25T10:55:53.963Z
Learning: The codebase must remain compatible with older Python versions. Avoid suggesting syntax not supported in these versions, such as `raise ... from e`.

Applied to files:

  • src/snakemake/resources.py
📚 Learning: 2024-12-14T13:11:32.988Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2927
File: snakemake/ioutils/evaluate.py:11-11
Timestamp: 2024-12-14T13:11:32.988Z
Learning: In `snakemake/ioutils/evaluate.py`, the use of `eval()` with `globals()` is acceptable because it's called by the user under consideration of the implications.

Applied to files:

  • src/snakemake/resources.py
🧬 Code graph analysis (11)
tests/common.py (1)
src/snakemake/resources.py (3)
  • from_mapping (1061-1084)
  • parse (983-1058)
  • defaults (1209-1210)
src/snakemake/script/__init__.py (3)
src/snakemake/workflow.py (1)
  • resources (2245-2250)
src/snakemake/jobs.py (2)
  • resources (482-499)
  • resources (1539-1553)
src/snakemake/io/__init__.py (1)
  • ResourceList (2006-2007)
src/snakemake/dag.py (1)
src/snakemake/scheduling/job_scheduler.py (1)
  • stats (187-188)
src/snakemake/spawn_jobs.py (1)
src/snakemake/settings/types.py (2)
  • _parsed_overwrite_resources (372-376)
  • _parsed_overwrite_threads (386-392)
src/snakemake/settings/types.py (1)
src/snakemake/resources.py (5)
  • Resource (460-852)
  • Resources (855-1162)
  • from_mapping (1061-1084)
  • items (1098-1099)
  • default (945-963)
src/snakemake/workflow.py (3)
src/snakemake/exceptions.py (3)
  • ResourceDuplicationError (665-672)
  • ResourceConversionError (648-662)
  • ResourceValidationError (640-641)
src/snakemake/resources.py (8)
  • ResourceScopes (1168-1306)
  • Resources (855-1162)
  • value (535-545)
  • copy (1105-1107)
  • get (1109-1113)
  • from_mapping (1061-1084)
  • update (1086-1090)
  • update (1303-1306)
src/snakemake/settings/types.py (4)
  • _parsed_resources (368-369)
  • _parsed_default_resources (379-383)
  • _parsed_overwrite_threads (386-392)
  • _parsed_overwrite_resources (372-376)
src/snakemake/common/__init__.py (1)
src/snakemake/exceptions.py (1)
  • NestedCoroutineError (558-559)
src/snakemake/api.py (1)
src/snakemake/resources.py (2)
  • Resources (855-1162)
  • default (945-963)
src/snakemake/rules.py (4)
src/snakemake/io/__init__.py (1)
  • ResourceList (2006-2007)
src/snakemake/resources.py (6)
  • Resource (460-852)
  • Resources (855-1162)
  • evaluate (551-568)
  • constrain (570-601)
  • value (535-545)
  • expand_items (1115-1162)
src/snakemake/exceptions.py (5)
  • ResourceConstraintError (644-645)
  • ResourceValidationError (640-641)
  • InputOpenException (583-586)
  • NestedCoroutineError (558-559)
  • ResourceConversionError (648-662)
src/snakemake/persistence.py (1)
  • rule (474-475)
src/snakemake/io/__init__.py (1)
src/snakemake/resources.py (1)
  • items (1098-1099)
src/snakemake/resources.py (3)
src/snakemake/common/__init__.py (3)
  • get_input_function_aux_params (315-327)
  • mb_to_mib (69-70)
  • mib_to_mb (73-74)
src/snakemake/exceptions.py (7)
  • NestedCoroutineError (558-559)
  • ResourceError (607-608)
  • ResourceInsufficiencyError (611-630)
  • ResourceScopesException (633-637)
  • ResourceConversionError (648-662)
  • format (658-662)
  • format_evaluated (650-655)
src/snakemake/io/__init__.py (6)
  • AnnotatedString (956-976)
  • items (1869-1871)
  • get (1914-1919)
  • is_callable (326-327)
  • is_callable (967-968)
  • is_callable (1185-1190)
🪛 Ruff (0.14.7)
src/snakemake/dag.py

18-18: Redefinition of unused Dict from line 18

Remove definition: Dict

(F811)

src/snakemake/settings/types.py

358-358: Do not perform function call immutables.Map in dataclass defaults

(RUF009)

src/snakemake/cli.py

232-232: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


244-244: Avoid specifying long messages outside the exception class

(TRY003)

src/snakemake/jobs.py

1550-1552: Avoid specifying long messages outside the exception class

(TRY003)

src/snakemake/workflow.py

1889-1892: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1889-1892: Avoid specifying long messages outside the exception class

(TRY003)


1897-1897: Avoid specifying long messages outside the exception class

(TRY003)

src/snakemake/common/__init__.py

70-70: Value being cast to int is already an integer

Remove unnecessary int call

(RUF046)


74-74: Value being cast to int is already an integer

Remove unnecessary int call

(RUF046)


111-117: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


111-117: Avoid specifying long messages outside the exception class

(TRY003)

src/snakemake/rules.py

1114-1114: Unused lambda argument: e

(ARG005)


1141-1141: Avoid specifying long messages outside the exception class

(TRY003)


1154-1158: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


1154-1158: Avoid specifying long messages outside the exception class

(TRY003)


1167-1167: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)

src/snakemake/io/__init__.py

1-1: The file is executable but no shebang is present

(EXE002)

src/snakemake/resources.py

387-387: zip() without an explicit strict= parameter

Add explicit value for parameter strict=

(B905)


400-400: zip() without an explicit strict= parameter

Add explicit value for parameter strict=

(B905)


541-544: Avoid specifying long messages outside the exception class

(TRY003)


568-568: Use raise without specifying exception name

Remove exception name

(TRY201)


647-649: Avoid specifying long messages outside the exception class

(TRY003)


741-741: Unused function argument: wildcards

(ARG001)


745-745: Unused function argument: rulename

(ARG001)


756-756: Unused function argument: wildcards

(ARG001)


756-756: Unused function argument: rulename

(ARG001)


772-772: Unused static method argument: name

(ARG004)


781-790: Use of possibly insecure function; consider using ast.literal_eval

(S307)


796-796: Do not catch blind exception: Exception

(BLE001)


805-819: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


805-819: Avoid specifying long messages outside the exception class

(TRY003)


844-844: Value being cast to int is already an integer

Remove unnecessary int call

(RUF046)


849-849: Value being cast to int is already an integer

Remove unnecessary int call

(RUF046)


879-886: Mutable class attributes should be annotated with typing.ClassVar

(RUF012)


920-920: Avoid specifying long messages outside the exception class

(TRY003)


942-942: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


942-942: Avoid specifying long messages outside the exception class

(TRY003)


956-956: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling

(B904)


956-956: Avoid specifying long messages outside the exception class

(TRY003)


1011-1014: Avoid specifying long messages outside the exception class

(TRY003)


1049-1051: Avoid specifying long messages outside the exception class

(TRY003)


1191-1195: Avoid specifying long messages outside the exception class

(TRY003)

🔇 Additional comments (53)
src/snakemake/dag.py (1)

3178-3203: DAG.stats typing change looks consistent with implementation.

The updated return annotation -> Tuple[str, Dict[str, int]] matches the actual return value (message, stats_dict) and aligns with how stats are consumed elsewhere (e.g. schedulers). No behavior change introduced here.

src/snakemake/resources.py (18)

492-517: Resource.__init__ validation and evaluator detection look sound.

The constructor correctly:

  • Restricts values to str | int | float | callable | None | TBDString and normalizes floats to rounded ints.
  • Parses human‑friendly strings only for known sized/time resources.
  • Detects both plain callables and AnnotatedString callables as evaluators, leaving others as concrete.

This matches the intended behavior described in the docstring and should integrate cleanly with evaluate() and standardize_size().


570-601: Resource.constrain correctly enforces type compatibility, including AnnotatedString.

The method:

  • Bails out for TBDString and None values as intended.
  • Uses isinstance(self_val, type(other_val)) rather than direct type(...) equality, which preserves compatibility between plain str and AnnotatedString (per prior discussion).
  • Only applies a min() constraint when both sides are int, returning self otherwise.

This is a good balance between safety and the polymorphic behavior needed for annotated strings.


603-625: from_mb_to_mib handles both concrete and evaluable resources correctly.

Using _convert_units(mb_to_mib) combined with renaming to *_mib ensures:

  • Concrete integer MB values are converted once.
  • Evaluable resources are wrapped so the converter is applied post‑evaluation.
  • Non‑numeric/evaluable types raise ResourceConversionError as expected.

This matches the documented behavior and works with later normalization in Resources.


626-651: format_human_friendly behavior is reasonable; just be aware it assumes MB units.

The method:

  • Rejects unevaluated resources with a TypeError, which is safer than silently formatting.
  • Delegates to _convert_units(mb_to_str), so it will also correctly preserve TBDString.
  • Assumes the stored unit is MB and only strips _mb, which is consistent with how standardize_size() normalizes sized resources.

No changes needed; just worth noting the MB assumption when adding new sized resource types.


678-715: standardize_size’s normalization flow is coherent with Resources._normalize_sizes.

The method:

  • Validates _mb resources using an identity converter.
  • Converts _mib_mb via mib_to_mb.
  • For unsuffixed sized resources, wraps the evaluator (if any) to ensure evaluation and human‑friendly parsing happen before attaching _mb.

This aligns with Resources._normalize_sizes, which converts any of {name, name_mb, name_mib} into a single _mb entry. Behavior looks correct.


722-769: CLI expression helpers (from_cli_expression / cli_evaluator) are consistent with workflow semantics.

  • from_cli_expression builds evaluators with the expected signature used by the job machinery (optionally including threads), and preserves the raw expression string.
  • cli_evaluator limits the eval environment to input, attempt, system_tmpdir, shutil, and optionally threads, and falls back to treating values as plain strings on NameError/SyntaxError or when is_ordinary_string returns True.
  • File‑not‑found conditions are mapped to TBDString in line with the rest of the resource evaluation logic.

Given Snakemake’s explicit support for Python expressions in resource specifications, this is an appropriate design.


823-852: parse_human_friendly behavior and error messages look good.

  • Only string values (excluding TBDString) for known human‑friendly resources are parsed.
  • Size strings are converted to at least 1 MB; time strings to at least 1 minute, with clear unit‑specific error messages.
  • Non‑human‑friendly resources and non‑string values are passed through untouched.

Note that callers like Resources.parse intentionally swallow WorkflowError here to re‑interpret inputs as plain ints/expressions, which matches the PR’s design notes.


888-908: Resources.__init__ and _normalize_sizes implement sized-resource canonicalization correctly.

  • Passing a Dict[str, Resource] preserves existing instances, while None initializes an empty container.
  • _normalize_sizes enforces that only one of {name, name_mb, name_mib} is present and then normalizes to a single _mb entry via standardize_size(), raising ResourceDuplicationError otherwise.

This achieves the “one effective sized resource per name” invariant described in the PR.


937-943: Resources.decode_arg provides minimal but sufficient validation.

Splitting on the first = and raising ValueError when it’s missing is consistent with the CLI contract that resources must be specified as name=value. Callers higher up already wrap this in Snakemake-specific error handling.


965-981: parser_factory cleanly wraps parse for argparse-style usage.

The returned lambda closes over allow_expressions and defaults and simply forwards the list of CLI strings to Resources.parse, which matches the intended usage in the CLI layer. No issues here.


983-1057: Resources.parse behavior matches documented CLI semantics.

Key points:

  • Enforces valid identifiers and reserves _cores.
  • Attempts human‑friendly parsing first, then integer parsing, then (optionally) expression parsing; otherwise it raises WorkflowError with a clear message.
  • Accepts zero and positive integers but rejects negatives, in line with earlier discussion about zero being valid for some flags.
  • Properly merges defaults when defaults is provided, with explicit override semantics.

Overall this looks correct and matches the PR’s stated behavior.


1060-1084: Resources.from_mapping covers both plain mappings and existing Resources.

  • Short‑circuiting when mapping is already a Resources instance avoids unnecessary wrapping.
  • For plain mappings, values are eagerly wrapped in Resource, so all downstream normalization and validation logic applies.

This is a straightforward and useful constructor.


1115-1163: Resources.expand_items correctly evaluates, constrains, and expands sized resources.

  • Respects an optional evaluate callback and skips names in skip.
  • Applies per-resource constraints via Resource.constrain.
  • When expand_sized is True, emits three entries for each *_mb sized resource: _mb (int), unsuffixed (human‑friendly string), and _mib (int), which matches the PR’s design that executors can choose the most convenient representation.

This should give executors and plugins a coherent, backwards‑compatible view of resources.


1165-1207: ResourceScopes normalization and validation logic look correct.

  • Constructor normalizes sized resource names to their unsuffixed form and raises ResourceDuplicationError for conflicting variants, mirroring Resources.
  • It allows the internal "excluded" scope while keeping the error message user‑facing (“local”/“global” only), as intended.
  • _normalize_sizes correctly promotes a single suffixed variant to the base name.

No functional issues spotted here.


1208-1219: ResourceScopes.defaults and _test give a clear, centralized scoping policy.

Defaults of mem_mb="local", disk_mb="local", and runtime="excluded" align with the intended scheduler behavior, and _test properly collapses _mb/_mib to the base sized resource for scope checks.


1220-1255: Scope query helpers on ResourceScopes are straightforward and consistent.

is_local, is_global, and is_excluded all delegate to _test, and the corresponding locals, globals, and excluded properties expose these as sets, with excluded also including all suffixed sized resources when the base is excluded. This matches the intended semantics for scheduler vs. executor handling.


1303-1307: ResourceScopes.update preserves normalization on incoming mappings.

By wrapping non‑ResourceScopes inputs in a new ResourceScopes instance, you ensure sized resources in overrides are normalized before merging, preventing conflicting suffix variants from slipping through. Good defensive design.


1309-1342: is_ordinary_string heuristic is reasonable for distinguishing expressions from “plain” values.

The regex specifically filters out function calls, dict literals, lambdas, arithmetic, and attribute access. Combined with the higher-level handling in cli_evaluator (only used on non‑NameError/SyntaxError exceptions), this should avoid misclassifying most “resource-like” strings while still surfacing genuine expression errors.

test.txt (1)

1-1: Fixture file looks fine.

The test.txt file containing 3.0 is a simple fixture and does not impact runtime behavior.

src/snakemake/api.py (1)

66-66: Switch to Resources.default("full") maintains previous default-resources behavior.

Importing Resources and using:

self.workflow_api.resource_settings.default_resources = Resources.default("full")

for non-local execution cleanly aligns the API with the new Resources abstraction while preserving the prior semantics of “full” default resources in cluster/cloud mode.

Also applies to: 571-575

src/snakemake/script/__init__.py (1)

114-114: LGTM!

The type annotation change from io_.Resources to io_.ResourceList aligns with the PR's rename of the old Resource class to ResourceList for executor plugin compatibility. This is consistent with the class definition in src/snakemake/io/__init__.py and usage in src/snakemake/jobs.py.

src/snakemake/spawn_jobs.py (2)

98-119: LGTM!

The refactored resource handling correctly uses the new _parsed_overwrite_resources and _parsed_overwrite_threads properties, which return Resources and Resource objects respectively. Accessing .raw on these Resource objects to serialize the original CLI argument is consistent with the new Resource API design.


364-371: LGTM!

The change to use _parsed_default_resources with attr="args" correctly serializes default resources for spawned jobs using the new Resources API. The args attribute provides the CLI-compatible representation needed for subprocess invocation.

src/snakemake/common/__init__.py (3)

69-75: Consider using raise ... from e for proper exception chaining.

The static analysis correctly identifies that in the async_run function (lines 111-117), the exception should be raised with explicit chaining. However, looking at the actual code, the RuntimeError is already passed as the second argument to NestedCoroutineError. If this is intentional (storing the original exception), this is acceptable. Otherwise, consider using raise ... from e for standard exception chaining.

The mib_to_mb function is a useful addition that complements mb_to_mib. The int() calls on lines 70 and 74 are technically redundant since math.ceil and math.floor already return integers in Python 3, but they serve as explicit type documentation.


111-117: Exception handling looks intentional but verify chaining behavior.

The NestedCoroutineError is raised with the original RuntimeError passed as a second argument. This appears intentional, as WorkflowError (the parent class) likely handles this pattern. The static analysis hint (B904) about raise ... from err is a false positive in this context since the original error is explicitly passed to preserve the chain.


315-327: LGTM!

The updated type annotations for get_input_function_aux_params improve code clarity. Using Callable[..., Any] and Mapping[str, Any] appropriately describes the function's interface.

tests/test_default_resources/Snakefile (2)

9-14: LGTM!

The expanded test correctly validates that sized resources now expose all three representations (mem_mb, mem, mem_mib). The expected values are mathematically consistent: 1 GB = 1000 MB, and 1000 MB ≈ 954 MiB (using the mb_to_mib conversion factor of ~0.9537).


21-26: LGTM!

Consistent with rule a, this test validates the disk resource representations. The shell syntax with backslash line continuations is correct for bash.

src/snakemake/scheduling/job_scheduler.py (2)

1-2: LGTM!

The import reorganization and TYPE_CHECKING pattern for the Workflow type hint follow Python best practices for avoiding circular imports while maintaining type safety.

Also applies to: 10-13, 19-19, 39-41


103-119: Resource expansion logic appears intentional per prior discussion.

The expand_sized=False setting and string filtering has been previously discussed, with the author clarifying that docstrings were updated to explain the intended behavior. The change to access workflow.global_resources["_nodes"].value correctly uses the new Resource API to check if nodes are explicitly set.

Using copy.copy(self.global_resources) instead of dict(self.global_resources) is a good practice for explicit intent.

tests/common.py (2)

11-20: Imports and is_connected URLError handling look correct.

The additional typing imports, urllib.error usage, and ResourceScopes/Resources imports are consistent with later code; catching urllib.error.URLError is the right exception type here.

Also applies to: 32-32, 57-65


163-209: Resource normalization in run() aligns with new Resources API.

Wrapping resources via Resources.from_mapping(...) and parsing list-style default_resources with Resources.parse(..., defaults="full", allow_expressions=True) ensures tests exercise the same code paths as the CLI and always hand ResourceSettings proper Resources instances.

Also applies to: 373-399

src/snakemake/jobs.py (2)

41-45: Resource-related imports are consistent with new API usage.

Using ResourceList from snakemake.io alongside GroupResources, Resources, and ResourceInsufficiencyError matches how resources and grouped resource errors are handled later in this module.

Also applies to: 52-53, 56-62


482-499: Job.resources now returning ResourceList is compatible with existing usage.

The property still lazily expands rule resources, respects skip_evaluation for tmpdir, merges _aux_resources, and now exposes them as a ResourceList, preserving attribute/item access semantics needed by {resources} in formatting and by Job.threads.

src/snakemake/io/__init__.py (2)

1-1: Future import and extended typing imports are appropriate.

Enabling postponed evaluation of annotations and pulling in Generic, Iterator, and Tuple matches the new generic/iterator typing later in this module without affecting runtime behavior.

Also applies to: 27-41


1489-1497: Type refinements for do_expand and glob_wildcards improve clarity without semantic change.

Using Dict[str, Union[str, collections.abc.Iterable[str]]] and List[str] for wildcard name collections better reflects actual usage and keeps compatibility with older Python versions while leaving the expansion/globbing logic untouched.

Also applies to: 1650-1652

src/snakemake/cli.py (4)

13-13: Typing and resource imports are consistent with later usage.

Adding Dict and importing Resource, Resources, and ResourceScopes matches how parsing, resource scopes, and ResourceSettings are wired up below.

Also applies to: 42-46


110-123: parse_set_threads and generalized parse_set_ints correctly support ints and expressions.

The new wrapper converts positive integers into Resource("_cores", n) and defers non-ints to Resource.from_cli_expression, while parse_set_ints:

  • Enforces non-negativity for integer values.
  • Propagates conversion errors as ValueError(errmsg) when no wrapper is used.
  • Wraps any wrapper failures with raise ValueError(... ) from e.

This preserves existing integer semantics and adds robust expression handling for --set-threads.

Also applies to: 223-245


529-547: Switching --resources to Resources.parser_factory is a good centralization.

Using default=Resources() plus parse_func=Resources.parser_factory(allow_expressions=False) standardizes global resource parsing and validation, and ensures CLI-provided --resources values flow through the same code paths as the rest of the resource system.


608-614: --default-resources now correctly leverages Resources.parser_factory with defaults.

Wrapping Resources.parser_factory(defaults="full", allow_expressions=True) in maybe_base64(...) lets profile/CLI values populate args.default_resources as a fully initialized Resources instance with the built-in defaults, including expression support, matching the behavior used elsewhere in the workflow API.

src/snakemake/workflow.py (5)

182-184: LGTM!

The refactoring to use _parsed_resources centralizes resource handling correctly. The Resources object supports item assignment for _cores and _nodes.


630-635: LGTM!

The property accessors correctly extract .value from the Resource objects now stored in global_resources.


687-698: LGTM!

Adding the return type annotation improves API clarity.


1873-1912: LGTM!

The refactored resource handling correctly:

  1. Initializes rule resources from parsed default resources with a copy
  2. Handles thread overwrites with proper validation
  3. Parses rule-defined resources via Resources.from_mapping
  4. Translates resource-specific exceptions to RuleException with context

Based on learnings, the exception handling pattern without chaining in some places is intentional for backward compatibility.


2011-2018: LGTM!

The handover handling correctly uses Resources.update() to merge global resources wholesale when no explicit resources are provided.

src/snakemake/settings/types.py (3)

349-350: LGTM!

The ValidResource type alias correctly captures the full range of valid resource types, including the recursive callable case for dynamic resource evaluation.


352-365: LGTM!

The field type unions provide flexibility by accepting both raw mappings (for CLI/API compatibility) and pre-parsed Resources objects. The use of immutables.Map() as default is a safe, pre-existing pattern in this codebase.


367-392: LGTM!

The cached property accessors provide clean lazy parsing:

  • _parsed_resources and _parsed_overwrite_resources correctly use Resources.from_mapping
  • _parsed_default_resources defaults to "bare" mode when None
  • _parsed_overwrite_threads normalizes both raw integers and Resource objects to Resource instances
src/snakemake/rules.py (4)

52-58: LGTM!

The new resource-related imports align with the refactored resource handling architecture and are all used appropriately in the file.


97-97: LGTM!

The explicit type annotations for self.workflow and self.resources improve code clarity and enable better IDE support without introducing circular import issues.

Also applies to: 109-109


1101-1128: LGTM!

The evaluate() helper function correctly:

  1. Skips evaluation for resources in skip_evaluation, returning TBDString
  2. Handles ResourceValidationError by wrapping in WorkflowError
  3. Handles NestedCoroutineError gracefully for async nesting scenarios (well-documented)
  4. Propagates other exceptions as InputFunctionException

The unused lambda argument e at line 1114 is required by the incomplete_checkpoint_func callback signature.


1130-1171: LGTM!

The refactored expand_resources correctly:

  1. Evaluates and constrains threads via chained .constrain() calls
  2. Uses Resources.expand_items() for centralized resource evaluation and expansion
  3. Translates ResourceConstraintError and ResourceConversionError to contextual WorkflowError
  4. Returns a ResourceList for executor plugin compatibility

Based on learnings, the exception handling pattern without explicit chaining is intentional for this codebase.

@pvandyken pvandyken force-pushed the fix/human-friendly-units-groups branch from 01d165b to bf37c8c Compare February 4, 2026 17:19
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
src/snakemake/common/__init__.py (1)

16-26: ⚠️ Potential issue | 🟠 Major

Fix NameError by aligning exception type and docstring with imports.

Line 132 raises WorkflowError, but this exception is not imported (only NestedCoroutineError is imported on line 25). Switch the raised exception to NestedCoroutineError and update the docstrings at lines 119 and 174 to reflect this change.

🛠️ Proposed fixes
-    WorkflowError
-        If called from within an already running event loop.
+    NestedCoroutineError
+        If called from within an already running event loop.
-        raise WorkflowError(
+        raise NestedCoroutineError(
             "Error running coroutine in event loop. Snakemake currently does not "
             "support being executed from an already running event loop. "
             "If you run Snakemake e.g. from a Jupyter notebook, make sure to spawn a "
             "separate process for Snakemake.",
         )
-    WorkflowError
-        If called from within an already running event loop.
+    NestedCoroutineError
+        If called from within an already running event loop.
tests/tests.py (1)

1057-1084: ⚠️ Potential issue | 🟠 Major

Fix expected mem_mib value for 60000 MB (Line 1077).

Other tests in this file already assert mem_mib=57221 for 60000 MB. Keeping 57222 here will diverge and likely fail with the shared conversion logic.

🐛 Proposed fix
-        mem_mib=57222,
+        mem_mib=57221,
🤖 Fix all issues with AI agents
In `@src/snakemake/io/__init__.py`:
- Line 1: The package init module (snakemake.io/__init__.py) is marked
executable but contains only imports; either remove the executable bit from this
module file (make it non-executable) if it’s intended only for import, or add a
proper shebang line (e.g., #!/usr/bin/env python3) at the top to declare it
executable; update the file permission or add the shebang consistently with
project conventions and CI/tooling expectations.

In `@src/snakemake/workflow.py`:
- Around line 1901-1940: Remove exception chaining when re-raising RuleException
for resource errors: in the block handling ruleinfo.resources, change the two
raises that use "from err" so they simply raise RuleException(err, rule=rule)
and raise RuleException(msg + str(err), rule=rule) respectively; keep the same
catch clauses for ResourceDuplicationError, ResourceConversionError, and
ResourceValidationError and preserve the original messages and rule parameter
but do not append "from err" to the raise statements (affecting the raise in the
ResourceDuplicationError and ResourceConversionError handlers).
🧹 Nitpick comments (1)
src/snakemake/resources.py (1)

890-897: Consider adding ClassVar annotation for the class constant.

The DEFAULTS dictionary is a class-level constant that should not be mutated. While this works correctly as-is, adding a ClassVar annotation would make the intent clearer and satisfy type checkers.

+from typing import ClassVar

+    DEFAULTS: ClassVar[dict[str, dict[str, str]]] = {
-    DEFAULTS = {

This centralizes the code for resource handling into the resource.py
file under a new Resource class. Notably, DefaultResources has been
removed, it's functionality rolled into the new class. The same class is
now used for handling global resource limits, default resources, and
rule resource constraints

As a result of this standarization, the following bugs/inconsistencies are fixed:

1) humanreadable resources can now be specified in a group job in
   cluster submit without crashing the program
2) if mem or disk are used as global resource limits, they will
   correctly restrict mem_mb, mem_mib, disk_mb, and disk_mib
3) similar to above when setting a global scope for mem and disk
4) humanreadable resources can now be used for global resource limits
5) disk_mib and mem_mib can be set in rule resource specifications,
   global limits, and in default resources, and will correctly set the
   _mb variants.
6) Providing the same effective resource twice (e.g. mem, mem_mb and/or
   mem_mib) in the same rule or in the global resource specification
   raises an error.
7) QOL: Resources in tests can be set using mappings
8) Runtime is handled as a human-readable resource
9) Original raw resource specifications are maintained to pass on to
   subprocesses

10) Fix handling of input-based resource lambdas

Previously, lambdas with an input parameter were skipped by inspecting
the arguments of the lambda via introspection. This did not work well
with the new refactor, which involves potentially deep nesting of these
lambdas. Instead, always run these input functions, relying on the fact
that missing input files will already result in TBDStrings, the same
outcome as the previous skip_evaluation.

Currently limited by second bug, where input file sizes cannot be
evaulated in the initial dag creation routine due to nesting of
asynchronous routines. For now, simply catch these nesting errors and
routine TBDString when they occur

11) Move resource expansion logic to resources.py

Better separation of concerns: in particular, handling of "magic"
resources "mem" and "disk" and their suffixed versions is handled
totally within `Resources`. `Rule.expand_resources` now only has logic
that requires direct knowledge of the rule.

12) Move humanfriendly from expr to main parse block

Previously only evaluated in the context of epxressions, preventing use
of human friendly resources with --resources. No reason for this, as
they can be parsed deterministically.

13) Sized resources are stored without a suffix

This makes much of the logic more straightforward, and facilitates
allowing callables on mem and disk to return strings which can then be
parsed as human-readable

14) Remove positive_integer arguments, CLI should always validate
    that integers are positive

15) Exclude the string versions of sized resources from the group layer
    function, then add in after
@pvandyken pvandyken force-pushed the fix/human-friendly-units-groups branch from bf37c8c to 8157552 Compare February 4, 2026 19:05
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Fix all issues with AI agents
In `@src/snakemake/resources.py`:
- Around line 225-233: Remove exception chaining when wrapping ResourceError and
WorkflowError: in the block that calls cls._get_layers (where ResourceError is
caught and you currently raise ResourceInsufficiencyError(additive_resources,
err.args[0]) from err) drop the "from err" so you simply raise
ResourceInsufficiencyError(additive_resources, err.args[0]). Apply the same
change for the two places that re-raise WorkflowError (the raise
WorkflowError(...) from err occurrences referenced at lines ~857 and ~862) so
those become plain raises without "from err". Ensure the caught exception
variable (err) is still used to extract the message but do not chain the
original exception.

In `@src/snakemake/rules.py`:
- Around line 1094-1169: In the evaluate(...) inner function, remove exception
chaining when re-raising ResourceValidationError and BaseException: change the
ResourceValidationError handler that does "raise WorkflowError(err, rule=self)
from err" to just "raise WorkflowError(err, rule=self)" and change the
BaseException handler that does "raise InputFunctionException(e, rule=self,
wildcards=wildcards) from e" to "raise InputFunctionException(e, rule=self,
wildcards=wildcards)"; keep all other behavior the same so evaluate,
ResourceValidationError and InputFunctionException usages are unchanged.

In `@test.txt`:
- Line 1: The new fixture file test.txt containing "3.0" appears unused; either
wire it into the test suite or remove it: search for any tests or helper modules
that should consume this fixture (tests that mention "test.txt" or use
fixture-loading utilities) and update the appropriate test(s) or loader function
to import/read test.txt (e.g., add a fixture path reference or call the existing
loadFixture utility), or delete test.txt if no consumer is required; ensure the
change is reflected in the test setup so CI no longer has a stray artifact.
🧹 Nitpick comments (1)
src/snakemake/logging.py (1)

68-73: Consider aligning the error message with the broader Mapping acceptance.
Right now the ValueError says “dict or Namedlist,” but you now accept any Mapping. Updating the message keeps debugging clearer.

🔧 Suggested tweak
-        raise ValueError(
-            "bug: format_dict applied to something neither a dict nor a Namedlist"
-        )
+        raise ValueError(
+            "bug: format_dict applied to something neither a Mapping nor a Namedlist"
+        )

@pvandyken pvandyken force-pushed the fix/human-friendly-units-groups branch from 8157552 to ff3b8c2 Compare February 4, 2026 20:17
@johanneskoester johanneskoester merged commit 4cada18 into snakemake:main Feb 5, 2026
112 of 113 checks passed
@github-project-automation github-project-automation bot moved this from In progress to Done in Snakemake Hackathon March 2025 Feb 5, 2026
johanneskoester pushed a commit that referenced this pull request Feb 5, 2026
🤖 I have created a release *beep* *boop*
---


##
[9.16.3](v9.16.2...v9.16.3)
(2026-02-05)


### Bug Fixes

* standardize handling of resources
([#3421](#3421))
([4cada18](4cada18))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
@twillis209
Copy link
Copy Markdown

Thank you for yet another contribution to snakemake.

I used to use the following in my profile with 3420 being the memory per CPU allocated on my then-HPC cluster:

default_resources:
    mem_mb: threads * 3420

I just upgraded to 9.16.3 and can no longer do this, which I think is by design as you write above:

BREAKING mem_mb and mem_mib can no longer be provided a string or a function that returns a string within a rule.

I'm sure there are good reasons for this, but can you suggest a workaround to achieve the same effect as the 'hack' above?

@fgvieira
Copy link
Copy Markdown
Contributor

fgvieira commented Feb 6, 2026

@pvandyken great work and thanks for not giving up on this PR! 😄

johanneskoester added a commit that referenced this pull request Mar 11, 2026
Fixes dynamic resource parsing regression introduced in #3421. Closes
#3970

### QC
<!-- Make sure that you can tick the boxes below. -->

* [x] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [x] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Preserve unresolved placeholder resource values during evaluation so
pending resource expressions remain intact and are not incorrectly
converted.

* **Tests**
* Added a test validating a minimal sequential workflow to confirm
resource handling.
* Updated an existing test to adjust embedded shell invocation structure
for more robust execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Johannes Koester <johannes.koester@uni-due.de>
@coderabbitai coderabbitai bot mentioned this pull request Mar 11, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

Development

Successfully merging this pull request may close these issues.

4 participants