feat: Modernize package configuration using Pixi#3369
feat: Modernize package configuration using Pixi#3369johanneskoester merged 31 commits intosnakemake:mainfrom
Conversation
📝 WalkthroughWalkthroughThe pull request introduces updates to several configuration files and code style. The Changes
Sequence Diagram(s)sequenceDiagram
participant Dev as Developer/PR
participant GA as GitHub Actions
participant Job as Jobs
Dev->>GA: Push PR update
GA->>GA: Evaluate conditions (PR merged? branch == main?)
alt Conditions Met
GA->>Job: Execute formatting, tests, build-container-image jobs
Job-->>GA: Return job results
GA->>GA: Execute testing-done job ("All tests passed")
else Conditions Not Met
GA-->>Dev: Skip conditional jobs
end
Tip ⚡🧪 Multi-step agentic review comment chat (experimental)
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
snakemake/template_rendering/yte.py (2)
10-13: Enhanced Readability with Multi-line Context ManagerThe refactor using a multi-line parenthesized
withstatement improves readability by clearly separating the file opening operations. This change preserves the functionality while making the code easier to maintain.
15-16: Consider Exception Chaining for Better TracebacksCurrently, the exception is raised with
raise WorkflowError("Failed to render yte template.", e). For improved traceback clarity, consider using Python’s exception chaining (e.g.,raise WorkflowError("Failed to render yte template.") from e) if theWorkflowErrorclass supports it.🧰 Tools
🪛 Ruff (0.8.2)
16-16: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
.gitignore (1)
34-37: Clarify and Validate New Ignore Patterns for Pixi Environments
The additions under the "# pixi environments" section correctly introduce ignoring for the.pixidirectory and for*.egg-infofiles. Note that an ignore pattern for*.egg-info/already exists on an earlier line. If the intent is to ignore both directories (with a trailing slash) and standalone files without the slash, a brief inline comment explaining this differentiation could help future maintainers.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (2)
pixi.lockis excluded by!**/*.lockpyproject.tomlis excluded by!pyproject.toml
📒 Files selected for processing (4)
.gitattributes(1 hunks).gitignore(1 hunks)setup.cfg(0 hunks)snakemake/template_rendering/yte.py(1 hunks)
💤 Files with no reviewable changes (1)
- setup.cfg
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
snakemake/template_rendering/yte.py
🔇 Additional comments (1)
.gitattributes (1)
9-10: Enhanced Configuration forpixi.lock
The new lines configuringpixi.lockto use binary merges, with YAML syntax highlighting and a generated flag, are well implemented. This setup helps prevent merge conflicts and ensures proper treatment by GitHub’s linguist.
matthewfeickert
left a comment
There was a problem hiding this comment.
I think this looks good overall. Just some high level comments.
matthewfeickert
left a comment
There was a problem hiding this comment.
LGTM. This is a super nice improvement. 👍
…nto jjjermiah/3347-modernize-project-config
…jjjermiah/snakemake into jjjermiah/3347-modernize-project-config
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
test-environment.yml (2)
7-7: Add setuptools as a top-level dependency
The inclusion ofsetuptoolsat the top level streamlines the environment setup and aligns with modern packaging practices. Consider whether you need to pin a specific version to ensure compatibility with your project.
61-62: Refactor cwl dependencies for improved clarity
Movingcwltoolandcwl-utilsfrom the nestedpipsection to the top-level dependencies simplifies dependency management and makes these packages first-class citizens in the environment specification. Please verify that these packages are available from your listed channels (or that an alternative installation method is available) to avoid potential resolution issues.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (17)
pixi.lockis excluded by!**/*.lockpyproject.tomlis excluded by!pyproject.tomlpyproject.tomlis excluded by!pyproject.tomlpyproject.tomlis excluded by!pyproject.tomlpyproject.tomlis excluded by!pyproject.tomlpixi.lockis excluded by!**/*.lockpyproject.tomlis excluded by!pyproject.tomlpyproject.tomlis excluded by!pyproject.tomlpyproject.tomlis excluded by!pyproject.tomlpixi.lockis excluded by!**/*.lockpyproject.tomlis excluded by!pyproject.tomlpyproject.tomlis excluded by!pyproject.tomlpyproject.tomlis excluded by!pyproject.tomlpixi.lockis excluded by!**/*.lockpyproject.tomlis excluded by!pyproject.tomlpyproject.tomlis excluded by!pyproject.tomlpyproject.tomlis excluded by!pyproject.toml
📒 Files selected for processing (11)
.gitattributes(1 hunks).gitignore(1 hunks)setup.cfg(0 hunks)snakemake/template_rendering/yte.py(1 hunks).gitignore(1 hunks)test-environment.yml(2 hunks).gitattributes(1 hunks).gitignore(1 hunks)setup.cfg(0 hunks)snakemake/template_rendering/yte.py(1 hunks).gitignore(1 hunks)
💤 Files with no reviewable changes (2)
- setup.cfg
- setup.cfg
🚧 Files skipped from review as they are similar to previous changes (8)
- .gitignore
- snakemake/template_rendering/yte.py
- .gitignore
- snakemake/template_rendering/yte.py
- .gitattributes
- .gitignore
- .gitignore
- .gitattributes
⏰ Context from checks skipped due to timeout of 90000ms (10)
- GitHub Check: testing-windows (10)
- GitHub Check: testing-windows (9)
- GitHub Check: testing-windows (8)
- GitHub Check: testing-windows (7)
- GitHub Check: testing-windows (6)
- GitHub Check: testing-windows (5)
- GitHub Check: testing-windows (4)
- GitHub Check: testing-windows (3)
- GitHub Check: testing-windows (2)
- GitHub Check: testing-windows (1)
There was a problem hiding this comment.
Actionable comments posted: 0
🔭 Outside diff range comments (1)
snakemake/ioutils/input.py (1)
30-31:⚠️ Potential issueImport missing WorkflowError and improve exception handling.
The
WorkflowErrorneeds to be imported, and exception handling should preserve the original error context.+from snakemake.exceptions import WorkflowError def extract_checksum(infile, **kwargs): try: import pandas as pd ... - raise WorkflowError("Pandas is required to extract checksum from file.") + raise WorkflowError("Pandas is required to extract checksum from file.") from err🧰 Tools
🪛 Ruff (0.8.2)
30-30: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
30-30: Undefined name
WorkflowError(F821)
🧹 Nitpick comments (5)
.github/workflows/main.yml (3)
41-55: Testing Job: Conditional Execution and Matrix Configuration
The added condition on line 42 ensures tests run only when the PR is not merged to main. Note that theexclude: []block (lines 52–54) is currently empty and has been flagged by static analysis. If no exclusions are needed, consider removing that block entirely to clean up the configuration.🧰 Tools
🪛 actionlint (1.7.4)
52-52: "exclude" section should not be empty
(syntax-check)
🪛 YAMLlint (1.35.1)
[error] 53-53: trailing spaces
(trailing-spaces)
211-218: Testing-Done Job: New Job and File Termination
The addition of thetesting-donejob provides a clear final notification when tests pass. However, static analysis indicates that there is no newline at the end of the file (line 218). Please add a newline at the end of the file.Below is an example diff to add the missing newline:
@@ -217,3 +217,4 @@ - run: echo "All tests passed." - \ No newline at end of file +🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 218-218: no new line character at the end of file
(new-line-at-end-of-file)
53-53: Clean Up Trailing Spaces
Static analysis flagged trailing spaces on line 53. Please remove these extra spaces to conform with YAML linting standards.🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 53-53: trailing spaces
(trailing-spaces)
snakemake/ioutils/input.py (2)
12-28: Replace lambda with a proper function definition.Using a lambda function for
fix_file_namereduces code readability. Consider using a proper function definition instead.- fix_file_name = lambda x: x.removeprefix("./") + def fix_file_name(x): + return x.removeprefix("./")🧰 Tools
🪛 Ruff (0.8.2)
16-16: Do not assign a
lambdaexpression, use adefRewrite
fix_file_nameas adef(E731)
12-31: Add docstrings to improve function documentation.The functions would benefit from docstrings explaining their purpose, parameters, and return values, especially given their integration into the Snakemake documentation.
def extract_checksum(infile, **kwargs): + """ + Extract checksum for a file from a checksum file. + + Parameters: + infile (str): Path to the checksum file + **kwargs: Additional arguments including 'file' - the filename to find checksum for + + Returns: + str: The extracted checksum + + Raises: + WorkflowError: If pandas is not available + """ try: import pandas as pd🧰 Tools
🪛 Ruff (0.8.2)
16-16: Do not assign a
lambdaexpression, use adefRewrite
fix_file_nameas adef(E731)
30-30: Within an
exceptclause, raise exceptions withraise ... from errorraise ... from Noneto distinguish them from errors in exception handling(B904)
30-30: Undefined name
WorkflowError(F821)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
CHANGELOG.mdis excluded by!CHANGELOG.md
📒 Files selected for processing (14)
.github/workflows/main.yml(1 hunks).github/workflows/codespell.yml(1 hunks).github/workflows/docs.yml(2 hunks).github/workflows/main.yml(6 hunks)docs/snakefiles/rules.rst(2 hunks)setup.py(1 hunks)snakemake/cli.py(1 hunks)snakemake/ioutils/__init__.py(2 hunks)snakemake/ioutils/input.py(1 hunks)snakemake/workflow.py(1 hunks)tests/test_ioutils/Snakefile(4 hunks)tests/test_ioutils/expected-results/c/1.txt(1 hunks)tests/test_ioutils/expected-results/results/switch~someswitch.column~sample.txt(1 hunks)tests/test_ioutils/samples.md5(1 hunks)
✅ Files skipped from review due to trivial changes (3)
- tests/test_ioutils/expected-results/results/switch
someswitch.columnsample.txt - tests/test_ioutils/samples.md5
- snakemake/cli.py
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/main.yml
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
setup.pysnakemake/workflow.pysnakemake/ioutils/input.pysnakemake/ioutils/__init__.py
🪛 Ruff (0.8.2)
snakemake/ioutils/input.py
16-16: Do not assign a lambda expression, use a def
Rewrite fix_file_name as a def
(E731)
30-30: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
30-30: Undefined name WorkflowError
(F821)
🪛 actionlint (1.7.4)
.github/workflows/main.yml
52-52: "exclude" section should not be empty
(syntax-check)
🪛 YAMLlint (1.35.1)
.github/workflows/main.yml
[error] 53-53: trailing spaces
(trailing-spaces)
[error] 218-218: no new line character at the end of file
(new-line-at-end-of-file)
⏰ Context from checks skipped due to timeout of 90000ms (28)
- GitHub Check: testing (10, 3.12, bash)
- GitHub Check: testing (10, 3.11, bash)
- GitHub Check: testing (8, 3.12, bash)
- GitHub Check: testing (8, 3.11, bash)
- GitHub Check: testing (7, 3.12, bash)
- GitHub Check: testing (7, 3.11, bash)
- GitHub Check: testing (6, 3.12, bash)
- GitHub Check: testing (6, 3.11, bash)
- GitHub Check: testing-windows (10)
- GitHub Check: testing (5, 3.12, bash)
- GitHub Check: testing-windows (9)
- GitHub Check: testing (5, 3.11, bash)
- GitHub Check: testing-windows (8)
- GitHub Check: testing (4, 3.12, bash)
- GitHub Check: testing (4, 3.11, bash)
- GitHub Check: testing-windows (7)
- GitHub Check: testing-windows (6)
- GitHub Check: testing (3, 3.12, bash)
- GitHub Check: testing-windows (5)
- GitHub Check: testing (3, 3.11, bash)
- GitHub Check: testing-windows (4)
- GitHub Check: testing (2, 3.12, bash)
- GitHub Check: testing-windows (3)
- GitHub Check: testing (2, 3.11, bash)
- GitHub Check: testing-windows (2)
- GitHub Check: testing (1, 3.12, bash)
- GitHub Check: testing-windows (1)
- GitHub Check: testing (1, 3.11, bash)
🔇 Additional comments (21)
setup.py (1)
33-33: Addition of unit test templates to package data looks good.This change includes template files from the
unit_tests/templates/directory in the package distribution, which aligns with the PR objective of modernizing the package configuration. This ensures that necessary test template files are properly packaged and available when needed..github/workflows/codespell.yml (1)
16-16: Validate Conditional Execution in Codespell Job
The added condition ensures that the codespell job will only run if the pull request isn’t merged or if the branch isn’t main. This update is consistent with our PR objectives..github/workflows/docs.yml (2)
16-16: Docs Job: Apply Conditional Execution
The new condition on line 16 guarantees that the docs job runs only if the pull request is not merged into main, aligning nicely with our overall workflow improvements.
41-41: Apidocs Job: Apply Conditional Execution
The condition on line 41 restricts the apidocs job from running after merge into main, mirroring the conditional logic applied in other workflow files..github/workflows/main.yml (3)
15-21: Formatting Job: Conditional Execution Update
The conditional on line 16 now restricts the formatting job to non-merged pull requests or non-main references, which is consistent with our intended CI/CD control flow.
157-165: Build Container Image Job: Conditional Execution
The condition on line 157 properly ensures that this job runs only under the non-merged/non-main conditions. This is consistent with the changes across the workflows.
167-174: Testing-Windows Job: Conditional Execution Update
The condition on line 167 appropriately restricts the Windows tests to non-merged PRs, and the additional steps to remove Unix-only dependencies are clearly implemented.tests/test_ioutils/expected-results/c/1.txt (1)
1-2: File content updated to match new MD5 checksum expectationThe file content has been updated from a simple 'a' to an MD5 checksum value, which aligns with the newly introduced checksum functionality in the codebase.
snakemake/workflow.py (1)
1294-1295: Added conditional cleanup based on configuration settingGood enhancement! The cleanup of storage objects is now conditional, based on the
keep_storage_localsetting. This prevents unnecessary cleanup operations when local storage should be retained, providing better control over the workflow's behavior.snakemake/ioutils/__init__.py (2)
8-8: Added imports for new utility functionsThe new imports for
parse_inputandextract_checksumfunctions from thesnakemake.ioutils.inputmodule are appropriate and follow the established import pattern in this file.
25-26: Registered new utility functions in global namespaceThe new utility functions
parse_inputandextract_checksumare correctly registered in the global namespace, making them available to Snakemake workflows. This follows the established pattern for exposing functions.tests/test_ioutils/Snakefile (6)
11-13: Added test assertion for extract_checksum functionalityGood test coverage! This assertion verifies that the newly added
extract_checksumfunction correctly retrieves a specific checksum from the MD5 file.
22-24: Updated rule input to include checksum fileThe rule now properly references the checksum file as an input dependency, which is necessary for the new checksum verification functionality.
26-29: Added parameters using new parse_input and extract_checksum functionsThe params section uses a lambda function to apply the new utility functions, demonstrating the proper usage pattern for parsing input files and extracting checksums. This is a good example of how to use the new functionality.
31-31: Updated shell command to use extracted checksumThe shell command now uses the dynamically computed checksum from params instead of a static string, properly testing the new functionality.
58-58: Added trailing comma for consistency in input declarationThe addition of a trailing comma after the input declaration improves code style consistency and makes future additions easier.
85-85: Reformatted shell command for consistencyThe shell command reformatting improves code style consistency throughout the file, though the functionality remains unchanged.
snakemake/ioutils/input.py (1)
1-9: Well-designed utility function for parsing input files!The
parse_inputfunction provides a flexible way to process input files with custom parsers. The function returns an inner function that follows Snakemake's expected pattern for input processing.docs/snakefiles/rules.rst (3)
550-571: Clear documentation for the new parse_input function.The documentation effectively explains the purpose, parameters, and provides a usage example for the
parse_inputfunction.
573-593: Well-structured documentation for extract_checksum function.The documentation for
extract_checksumis clear and includes a helpful usage example showing how to integrate withparse_input.
550-593:Details
✅ Verification successful
Verify consistency between documentation and implementation.
The function signatures in the documentation match the implementations, but it would be helpful to verify that the example code works as expected with the actual implementation.
🏁 Script executed:
#!/bin/bash # Check if the function parameters in docs match implementation echo "Implementation in input.py:" grep -A 2 "def parse_input" snakemake/ioutils/input.py grep -A 2 "def extract_checksum" snakemake/ioutils/input.py echo -e "\nDocumentation in rules.rst:" grep -A 4 "parse_input(" docs/snakefiles/rules.rst grep -A 4 "extract_checksum(" docs/snakefiles/rules.rstLength of output: 1490
Documentation and Implementation Signatures Verified and Consistent
The review checks confirmed that:
- The
parse_inputfunction is defined asdef parse_input(infile, parser, **kwargs):in the implementation, which aligns with the documentation’s signature (using different parameter names but equivalent semantics).- Likewise, the
extract_checksumfunction is implemented asdef extract_checksum(infile, **kwargs):, which effectively supports an extra keyword argument (used asfilein the documentation) as demonstrated by the example code.While the names differ slightly (i.e.
infileversusinput_itemand explicitfilein the docs), the behavior is consistent. For clarity, consider updating the documentation to explicitly note that both functions accept additional keyword arguments.
* fix: add conda and pip dependencies to pyproject.toml * fix: remove redundant test environment from pyproject.toml * fix: update test command in pyproject.toml to include additional test files * fix: add apt dependencies setup step in GitHub Actions workflow for Linux * fix: disable fail-fast in testing strategy for improved job execution * fix: add module check step in GitHub Actions for Linux and update pyproject.toml with conda channels * fix: remove unnecessary shell option from Linux test job in GitHub Actions * fix: update test target configuration in pyproject.toml to simplify structure * fix: add command to locate module in GitHub Actions workflow * fix: update test command in GitHub Actions to focus on specific test case * fix: remove unnecessary dependency on formatting job in GitHub Actions workflow * needs envmodues * fix: update pytest command syntax in pyproject.toml for clarity * fix: remove unnecessary module check from GitHub Actions workflow * fix: add a blank line for improved readability in tests.py * refactor: streamline GitHub Actions workflow by consolidating testing jobs and removing unnecessary steps let all tests within a job fail * fix: restore and reposition build-container-image job in GitHub Actions workflow * refactor: update test task definitions in pyproject.toml for clarity and organization * fix: add Windows test job to GitHub Actions workflow for improved cross-platform testing * fix: simplify command in Windows test job for improved readability * fix: remove unnecessary apt dependencies setup from GitHub Actions workflow now that its included in test feature * fix: update pytest command in pyproject.toml to exclude specific tests for improved test accuracy
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
.github/workflows/main.yml (2)
102-109: Fix trailing whitespace on line 109There's a trailing whitespace at the end of line 109, which should be removed.
- +🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 109-109: trailing spaces
(trailing-spaces)
128-128: Fix trailing whitespace on line 128There's a trailing whitespace at the end of line 128, which should be removed.
- needs: tests + needs: tests🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 128-128: trailing spaces
(trailing-spaces)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pyproject.tomlis excluded by!pyproject.toml
📒 Files selected for processing (3)
.github/workflows/main.yml(2 hunks)tests/tests.py(1 hunks)tests/tests_using_conda.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
tests/tests.pytests/tests_using_conda.py
🪛 YAMLlint (1.35.1)
.github/workflows/main.yml
[error] 109-109: trailing spaces
(trailing-spaces)
[error] 128-128: trailing spaces
(trailing-spaces)
⏰ Context from checks skipped due to timeout of 90000ms (29)
- GitHub Check: tests (10, windows-latest, py312, bash)
- GitHub Check: tests (10, windows-latest, py311, bash)
- GitHub Check: tests (10, ubuntu-latest, py312, bash)
- GitHub Check: tests (10, ubuntu-latest, py311, bash)
- GitHub Check: tests (7, ubuntu-latest, py311, bash)
- GitHub Check: tests (6, windows-latest, py312, bash)
- GitHub Check: tests (6, windows-latest, py311, bash)
- GitHub Check: tests (6, ubuntu-latest, py312, bash)
- GitHub Check: tests (6, ubuntu-latest, py311, bash)
- GitHub Check: tests (5, windows-latest, py312, bash)
- GitHub Check: tests (5, windows-latest, py311, bash)
- GitHub Check: tests (5, ubuntu-latest, py312, bash)
- GitHub Check: tests (5, ubuntu-latest, py311, bash)
- GitHub Check: tests (4, windows-latest, py312, bash)
- GitHub Check: tests (4, windows-latest, py311, bash)
- GitHub Check: tests (4, ubuntu-latest, py312, bash)
- GitHub Check: tests (4, ubuntu-latest, py311, bash)
- GitHub Check: tests (3, windows-latest, py312, bash)
- GitHub Check: tests (3, windows-latest, py311, bash)
- GitHub Check: tests (3, ubuntu-latest, py312, bash)
- GitHub Check: tests (3, ubuntu-latest, py311, bash)
- GitHub Check: tests (2, windows-latest, py312, bash)
- GitHub Check: tests (2, windows-latest, py311, bash)
- GitHub Check: tests (2, ubuntu-latest, py312, bash)
- GitHub Check: tests (2, ubuntu-latest, py311, bash)
- GitHub Check: tests (1, windows-latest, py312, bash)
- GitHub Check: tests (1, windows-latest, py311, bash)
- GitHub Check: tests (1, ubuntu-latest, py312, bash)
- GitHub Check: tests (1, ubuntu-latest, py311, bash)
🔇 Additional comments (10)
tests/tests_using_conda.py (1)
170-170: Good addition of theneeds_envmodulesmarkerAdding this decorator properly marks the test as requiring environment modules to be available, which aligns with the PR objective of modernizing the package configuration. This marker will help with test categorization and can be used to skip tests when environment modules aren't available.
tests/tests.py (1)
1358-1358: Good addition of theneeds_envmodulesmarkerAdding this decorator properly marks the test as requiring environment modules to be available, consistent with the similar change in
tests_using_conda.py. This helps with test categorization in the new Pixi-based configuration..github/workflows/main.yml (8)
24-31: Good implementation of Pixi for the formatting jobThe setup correctly installs Pixi using the
prefix-dev/setup-pixiaction and configures it to use the "quality" environment. The use ofcache: falseandlocked: falseis appropriate since there's no pixi.lock file yet.
32-33: Good update to the formatting commandThe command has been updated to use Pixi to run the formatting check, replacing the previous black command. This aligns with the PR objective of modernizing the package configuration using Pixi.
42-56: Good implementation of the test matrix strategyThe new test job uses a matrix strategy that covers different test groups, Python versions, and operating systems. The comment about test types referencing the pyproject.toml file is helpful for maintaining the configuration in the future.
70-77: Proper Pixi configuration for test environmentsThe Pixi setup correctly uses the matrix environment variable to determine which environment to install and configure. Same as with the formatting job, the use of
cache: falseandlocked: falseis appropriate.
110-114: Good OS-specific test commandsThe workflow correctly uses conditional steps based on the runner OS to execute appropriate test commands. For Windows, it uses the
test-simplecommand which is likely a subset of tests compatible with Windows.
115-124: Good addition of report generation testsAdding a specific step for report generation tests ensures that this important functionality is tested in the CI pipeline. The step is also conditionally executed only on Linux, where it's likely to be more reliable.
138-139: Good update to workflow dependenciesThe
testing-donejob now correctly depends on the newtestsjob, ensuring that it only runs after all tests have passed.
52-53:Details
❓ Verification inconclusive
Verify Windows testing compatibility
The matrix includes Windows, but there might be platform-specific tests that won't run on Windows. Consider validating that the
test-simplecommand on Windows covers all relevant scenarios or adding explicit exclusions for incompatible tests.You may want to run the following command to check if any test markers in the test files indicate Windows incompatibility:
🏁 Script executed:
#!/bin/bash # Look for tests that are skipped on Windows grep -r "skip_on_windows" tests/Length of output: 6894
Windows Testing Compatibility: Verify Coverage on Windows
The grep output confirms that tests expected to fail on Windows are already tagged with
@skip_on_windows(as seen in files liketests/conftest.py,tests/tests.py, andtests/tests_using_conda.py). This explicit exclusion is in line with handling known platform-specific issues.
- Action Required: Please ensure that the Windows test run (for example, via the
test-simplecommand) still exercises all the platform-independent scenarios. If any critical behavior isn’t covered due to these skips, consider adding additional Windows-specific tests or further refining the exclusion logic.
Current Status1. The CI/CD pipeline now uses the pixi test environments for
|
johanneskoester
left a comment
There was a problem hiding this comment.
Fantastic work!! One final thing: can you please update the testing guidelines here: https://snakemake.readthedocs.io/en/stable/project_info/contributing.html#testing-guidelines (i.e. in the corresponding place under docs/), to explain how to setup and run the tests just using pixi?
Let us do that in a separate PR, ok? |
There was a problem hiding this comment.
Actionable comments posted: 0
🔭 Outside diff range comments (1)
.github/workflows/main.yml (1)
140-147:⚠️ Potential issueCritical Conditional Syntax Issue in "testing-done" Job
There is a typo in the conditional for thetesting-donejob on line 141. The key is written asgithub.event_pull_request.mergedinstead of the expectedgithub.event.pull_request.merged. This discrepancy may result in the condition not evaluating as intended.Suggested Fix:
- if: github.event_pull_request.merged != true || github.ref != 'refs/heads/main' + if: github.event.pull_request.merged != true || github.ref != 'refs/heads/main'
🧹 Nitpick comments (4)
.github/workflows/main.yml (4)
47-56: Matrix Configuration and Docstring Typo
The matrix definition with multiple test groups, OS, and Python environments is well structured. Note that the inline comment referencespyprojec.toml—please update it topyproject.tomlfor clarity.
102-110: Trailing Whitespace in Linux Test Step
Static analysis has flagged trailing whitespace on line 110. Please remove any extra spaces to conform with YAML formatting best practices.🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 110-110: trailing spaces
(trailing-spaces)
111-119: Trailing Whitespace in Windows Test Step
Similarly, trailing whitespace is reported (e.g., on line 116). Please clean up the extra spaces in this step.🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 116-116: trailing spaces
(trailing-spaces)
130-133: Remove Trailing Whitespace in Build Container Image Job
Trailing whitespace is detected on line 133 within the build job. Removing these extra spaces will help maintain YAML cleanliness.🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 133-133: trailing spaces
(trailing-spaces)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pyproject.tomlis excluded by!pyproject.toml
📒 Files selected for processing (1)
.github/workflows/main.yml(2 hunks)
🧰 Additional context used
🪛 YAMLlint (1.35.1)
.github/workflows/main.yml
[error] 110-110: trailing spaces
(trailing-spaces)
[error] 116-116: trailing spaces
(trailing-spaces)
[error] 133-133: trailing spaces
(trailing-spaces)
⏰ Context from checks skipped due to timeout of 90000ms (20)
- GitHub Check: tests (10, windows-latest, py312, bash)
- GitHub Check: tests (10, windows-latest, py311, bash)
- GitHub Check: tests (9, windows-latest, py312, bash)
- GitHub Check: tests (9, windows-latest, py311, bash)
- GitHub Check: tests (8, windows-latest, py312, bash)
- GitHub Check: tests (8, windows-latest, py311, bash)
- GitHub Check: tests (7, windows-latest, py312, bash)
- GitHub Check: tests (7, windows-latest, py311, bash)
- GitHub Check: tests (6, windows-latest, py312, bash)
- GitHub Check: tests (6, windows-latest, py311, bash)
- GitHub Check: tests (5, windows-latest, py312, bash)
- GitHub Check: tests (5, windows-latest, py311, bash)
- GitHub Check: tests (4, windows-latest, py312, bash)
- GitHub Check: tests (4, windows-latest, py311, bash)
- GitHub Check: tests (3, windows-latest, py312, bash)
- GitHub Check: tests (3, windows-latest, py311, bash)
- GitHub Check: tests (2, windows-latest, py312, bash)
- GitHub Check: tests (2, windows-latest, py311, bash)
- GitHub Check: tests (1, windows-latest, py312, bash)
- GitHub Check: tests (1, windows-latest, py311, bash)
🔇 Additional comments (8)
.github/workflows/main.yml (8)
24-31: Pixi Installation in Formatting Job
The addition of the Pixi installation step usingprefix-dev/setup-pixi@v0.8.3for the "quality" environment looks good. Please just confirm that disabling caching (settingcache: false) is intentional.
32-34: Modernized Formatting Command
Replacing the direct use of Black with the Pixi formatting command (pixi run -e quality format --check --diff .) correctly reflects the PR’s modernization intent.
42-45: Informative Comment Block for Testing Job
The clearly delimited comment section for the testing job improves readability. No issues here.
70-77: Pixi Installation in Testing Job
The step that installs Pixi for each test environment mirrors the formatting job and is consistent.
78-85: AppArmor Namespace Adjustment for Linux
Disabling AppArmor namespace restrictions on Linux runners is appropriately conditional and necessary for the subsequent steps.
86-93: MinIO Setup for AWS S3 Testing
The configuration for setting up MinIO usingcomfuture/minio-action@v1is implemented correctly with the expected parameters.
94-101: MinIO Testing Step
The commands for testing the MinIO setup under a Linux environment are succinct and clear.
120-129: Report Generation Test Setup
The Report Generation Tests step is well configured, appropriately changing directory and invoking Pixi with the intended options.
|
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (5)
.github/workflows/main.yml (5)
32-33: Switch Formatting Check from Black to Pixi
The formatting check now uses the Pixi command (pixi run -e quality format --check --diff .) instead of invoking Black directly. Please verify that any associated documentation or PR comment messages (see later “Comment PR” step) are updated accordingly for consistency.
47-54: Matrix Configuration and Comment Accuracy
The matrix strategy is well defined with a comprehensive list of test groups, operating systems, and Python environments. Please note a minor typo in the comment on line 51—consider updating"pyprojec.toml"to"pyproject.toml". Also, ensure that includingwindows-latestaligns with your intended testing scope given some steps are Linux-specific.
102-109: Trailing Whitespace in Linux Test Step
The Linux test step command for running tests is correct overall. However, a trailing whitespace is detected on line 109. Removing this will help maintain YAML cleanliness.- --splitting-algorithm=least_duration␣ + --splitting-algorithm=least_duration🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 109-109: trailing spaces
(trailing-spaces)
118-127: Report Generation Tests and Shell Parameter Syntax
The report generation step is configured for Linux, and the sequence of commands is clear. One minor point: the shell parameter"bash -el {0}"is a bit unconventional. Consider verifying if the intended effect is achieved or if a more explicit format (for example using separate flags likebash -e -l {0}) would improve clarity.
128-132: Trailing Whitespace in Build-Container-Image Job
In thebuild-container-imagejob, line 131 shows trailing whitespace. Please remove this to adhere to YAML best practices.- needs: tests␣ + needs: tests🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 131-131: trailing spaces
(trailing-spaces)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
.github/workflows/main.yml(2 hunks)
🧰 Additional context used
🪛 YAMLlint (1.35.1)
.github/workflows/main.yml
[error] 109-109: trailing spaces
(trailing-spaces)
[error] 131-131: trailing spaces
(trailing-spaces)
⏰ Context from checks skipped due to timeout of 90000ms (38)
- GitHub Check: tests (10, windows-latest, py312, bash)
- GitHub Check: tests (10, windows-latest, py311, bash)
- GitHub Check: tests (10, ubuntu-latest, py312, bash)
- GitHub Check: tests (10, ubuntu-latest, py311, bash)
- GitHub Check: tests (9, windows-latest, py312, bash)
- GitHub Check: tests (9, windows-latest, py311, bash)
- GitHub Check: tests (9, ubuntu-latest, py312, bash)
- GitHub Check: tests (9, ubuntu-latest, py311, bash)
- GitHub Check: tests (8, windows-latest, py311, bash)
- GitHub Check: tests (8, ubuntu-latest, py312, bash)
- GitHub Check: tests (8, ubuntu-latest, py311, bash)
- GitHub Check: tests (7, windows-latest, py311, bash)
- GitHub Check: tests (7, ubuntu-latest, py312, bash)
- GitHub Check: tests (7, ubuntu-latest, py311, bash)
- GitHub Check: tests (6, windows-latest, py312, bash)
- GitHub Check: tests (6, windows-latest, py311, bash)
- GitHub Check: tests (6, ubuntu-latest, py312, bash)
- GitHub Check: tests (6, ubuntu-latest, py311, bash)
- GitHub Check: tests (5, windows-latest, py312, bash)
- GitHub Check: tests (5, windows-latest, py311, bash)
- GitHub Check: tests (5, ubuntu-latest, py312, bash)
- GitHub Check: tests (5, ubuntu-latest, py311, bash)
- GitHub Check: tests (4, windows-latest, py312, bash)
- GitHub Check: tests (4, windows-latest, py311, bash)
- GitHub Check: tests (4, ubuntu-latest, py312, bash)
- GitHub Check: tests (4, ubuntu-latest, py311, bash)
- GitHub Check: tests (3, windows-latest, py312, bash)
- GitHub Check: tests (3, windows-latest, py311, bash)
- GitHub Check: tests (3, ubuntu-latest, py312, bash)
- GitHub Check: tests (3, ubuntu-latest, py311, bash)
- GitHub Check: tests (2, windows-latest, py312, bash)
- GitHub Check: tests (2, windows-latest, py311, bash)
- GitHub Check: tests (2, ubuntu-latest, py312, bash)
- GitHub Check: tests (2, ubuntu-latest, py311, bash)
- GitHub Check: tests (1, windows-latest, py312, bash)
- GitHub Check: tests (1, windows-latest, py311, bash)
- GitHub Check: tests (1, ubuntu-latest, py312, bash)
- GitHub Check: tests (1, ubuntu-latest, py311, bash)
🔇 Additional comments (9)
.github/workflows/main.yml (9)
24-31: Install Pixi in Formatting Job
The Pixi installation step is well configured with the correct version and parameters. The use ofcache: falseandlocked: falseappears intentional (to avoid using apixi.lockfile).
42-44: Enhanced Section Separation for Testing Job
The added section header clearly marks the start of the testing configuration using Pixi. This improves readability and helps contextualize the subsequent job definitions.
57-64: Runner and Environment Variables Setup
Theruns-on: ${{ matrix.os }}directive and environment variable definitions are correctly set up. No issues noted here.
70-76: Pixi Installation in Tests Job
The Pixi installation step for the tests job mirrors the formatting job configuration and is correctly parameterized to deploy the proper environment from the matrix.
79-85: Linux-Specific AppArmor Configuration
The step to disable AppArmor restrictions for Apptainer is appropriately conditioned for Linux. Ensure that this command works as expected on all targeted Linux runners; occasionally, differences in distribution defaults might affect system control commands.
86-93: Setup MinIO for AWS S3 Testing
The MinIO setup step leverages a community action with clear parameters. The configuration appears correct and should suffice for AWS S3 testing.
94-101: MinIO Testing Step
The test step for MinIO appropriately sets the necessary environment variables and executes the command to create an S3 bucket. This implementation is straightforward and clear.
110-116: Windows Test Configuration Review
The Windows test step usestest-simple, which should be confirmed to align with your testing expectations on Windows. Given the mix of Linux-specific steps elsewhere, please double-check that the Windows tests are isolated and correctly configured.
139-145: Testing-Done Job Configuration
Thetesting-donejob is straightforward; it correctly depends on thetestsjob and echoes a success message. No issues noted in this section.
🤖 I have created a release *beep* *boop* --- ## [9.0.0](v8.30.0...v9.0.0) (2025-03-14) ### ⚠ BREAKING CHANGES * Logging refactor & add LoggerPluginInterface ([#3107](#3107)) ### Features * [#3412](#3412) - keep shadow folder of failed job if --keep-incomplete flag is set. ([#3430](#3430)) ([22978c3](22978c3)) * add flag --report-after-run to automatically generate the report after a successfull workflow run ([#3428](#3428)) ([b0a7f03](b0a7f03)) * add flatten function to IO utils ([#3424](#3424)) ([67fa392](67fa392)) * add helper functions to parse input files ([#2918](#2918)) ([63e45a7](63e45a7)) * Add option to print redacted file names ([#3089](#3089)) ([ba4d264](ba4d264)) * add support for validation of polars dataframe and lazyframe ([#3262](#3262)) ([c7473a6](c7473a6)) * added support for rendering dag with mermaid js ([#3409](#3409)) ([7bf8381](7bf8381)) * adding --replace-workflow-config to fully replace workflow configs (from config: directive) with --configfile, instead of merging them ([#3381](#3381)) ([47504a0](47504a0)) * Dynamic module name ([#3401](#3401)) ([024dc32](024dc32)) * Enable saving and reloading IOCache object ([#3386](#3386)) ([c935953](c935953)) * files added in rule params with workflow.source_path will be available in used containers ([#3385](#3385)) ([a6e45bf](a6e45bf)) * Fix keep_local in storage directive and more freedom over remote retrieval behaviour ([#3410](#3410)) ([67b4739](67b4739)) * inherit parameters of use rule and extend/replace individual items them when using 'with' directive ([#3365](#3365)) ([93e4b92](93e4b92)) * Logging refactor & add LoggerPluginInterface ([#3107](#3107)) ([86f1d6e](86f1d6e)) * Maximal file size for checksums ([#3368](#3368)) ([b039f8a](b039f8a)) * Modernize package configuration using Pixi ([#3369](#3369)) ([77992d8](77992d8)) * multiext support for named input/output ([#3372](#3372)) ([05e1378](05e1378)) * optionally auto-group jobs via temp files in case of remote execution ([#3378](#3378)) ([cc9bba2](cc9bba2)) ### Bug Fixes * `--delete-all-output` ignores `--dry-run` ([#3265](#3265)) ([23fef82](23fef82)) * 3342 faster touch runs and warning messages for non-existing files ([#3398](#3398)) ([cd9c3c3](cd9c3c3)) * add default value to max-jobs-per-timespan ([#3043](#3043)) ([2959abe](2959abe)) * checkpoints inside modules are overwritten ([#3359](#3359)) ([fba3ac7](fba3ac7)) * Convert Path to IOFile ([#3405](#3405)) ([c58684c](c58684c)) * Do not perform storage object cleanup with --keep-storage-local-copies set ([#3358](#3358)) ([9a6d14b](9a6d14b)) * edgecases of source deployment in case of remote execution ([#3396](#3396)) ([5da13be](5da13be)) * enhance error message formatting for strict DAG-building mode ([#3376](#3376)) ([a1c39ee](a1c39ee)) * fix bug in checkpoint handling that led to exceptions in case checkpoint output was missing upon rerun ([#3423](#3423)) ([8cf4a2f](8cf4a2f)) * force check all required outputs ([#3341](#3341)) ([495a4e7](495a4e7)) * group job formatting ([#3442](#3442)) ([f0b10a3](f0b10a3)) * in remote jobs, upload storage in topological order such that modification dates are preserved (e.g. in case of group jobs) ([#3377](#3377)) ([eace08f](eace08f)) * only skip eval when resource depends on input ([#3374](#3374)) ([4574c92](4574c92)) * Prevent execution of conda in apptainer when not explicitly requested in software deployment method ([#3388](#3388)) ([c43c5c0](c43c5c0)) * print filenames with quotes around them in RuleException ([#3269](#3269)) ([6baeda5](6baeda5)) * Re-evaluation of free resources ([#3399](#3399)) ([6371293](6371293)) * ReadTheDocs layout issue due to src directory change ([#3419](#3419)) ([695b127](695b127)) * robustly escaping quotes in generated bash scripts (v2) ([#3297](#3297)) ([#3389](#3389)) ([58720bd](58720bd)) * Show apptainer image URL in snakemake report ([#3407](#3407)) ([45f0450](45f0450)) * Update ReadTheDocs configuration for documentation build to use Pixi ([#3433](#3433)) ([3f227a6](3f227a6)) ### Documentation * Add pixi setup instructions to general use tutorial ([#3382](#3382)) ([115e81b](115e81b)) * fix contribution section heading levels, fix docs testing setup order ([#3360](#3360)) ([051dc53](051dc53)) * fix link to github.com/snakemake/poetry-snakemake-plugin ([#3436](#3436)) ([ec6d97c](ec6d97c)) * fix quoting ([#3394](#3394)) ([b40f599](b40f599)) * fix rerun-triggers default ([#3403](#3403)) ([4430e23](4430e23)) * fix typo 'safe' -> 'save' ([#3384](#3384)) ([7755861](7755861)) * mention code formatting in the contribution section ([#3431](#3431)) ([e8682b7](e8682b7)) * remove duplicated 'functions'. ([#3356](#3356)) ([7c595db](7c595db)) * update broken links documentation ([#3437](#3437)) ([e3d0d88](e3d0d88)) * Updating contributing guidelines with new pixi dev setup ([#3415](#3415)) ([8e95a12](8e95a12)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: snakemake-bot <snakemake-bot-admin@googlegroups.com>
resolves snakemake#3347 This PR is the first step in modernizing the project configuration by moving the `setup.cfg` sections to pyproject.toml, and defining environments using pixi. <details> <summary> pixi info </summary> ```console Project ------------ Name: snakemake Version: 8.30.0 Manifest file: /Users/bhklab/dev/snakemake-dev/snakemake/pyproject.toml Last updated: 12-03-2025 11:39:46 Environments ------------ Environment: default Features: default Channels: conda-forge, bioconda Dependency count: 1 Dependencies: python PyPI Dependencies: snakemake, appdirs, immutables, configargparse, connection-pool, docutils, gitpython, humanfriendly, jinja2, jsonschema, nbformat, packaging, psutil, pulp, pyyaml, requests, reretry, smart-open, snakemake-interface-executor-plugins, snakemake-interface-common, snakemake-interface-storage-plugins, snakemake-interface-report-plugins, tabulate, throttler, wrapt, yte, dpath, conda-inject Target platforms: linux-aarch64, osx-64, win-64, linux-64, osx-arm64 Environment: dev Features: test, docs, style, default Channels: conda-forge, bioconda Dependency count: 41 Dependencies: pytest, pytest-mock, pytest-cov, pytest-split, graphviz, oauth2client, numpy, glpk, pulp, boto3, responses, pytools, pandoc, crc32c, filelock, humanfriendly, pandas, pygments, pysftp, yte, ftputil, httpretty, polars, requests-mock, docutils, snakemake-storage-plugin-http, snakemake-storage-plugin-s3, snakemake-storage-plugin-fs, snakemake-executor-plugin-cluster-generic, conda, pip, sphinx, sphinx-argparse, sphinx_rtd_theme, sphinxawesome-theme, recommonmark, commonmark, myst-parser, sphinxcontrib-napoleon, black, python PyPI Dependencies: snakemake, appdirs, immutables, configargparse, connection-pool, docutils, gitpython, humanfriendly, jinja2, jsonschema, nbformat, packaging, psutil, pulp, pyyaml, requests, reretry, smart-open, snakemake-interface-executor-plugins, snakemake-interface-common, snakemake-interface-storage-plugins, snakemake-interface-report-plugins, tabulate, throttler, wrapt, yte, dpath, conda-inject Target platforms: linux-aarch64, osx-arm64, osx-64, linux-64, win-64 Tasks: test-complex, test-all, format, test-simple, docs, build-docs Environment: py311 Features: py311, test, default Channels: conda-forge, bioconda Dependency count: 32 Dependencies: python, pytest, pytest-mock, pytest-cov, pytest-split, graphviz, oauth2client, numpy, glpk, pulp, boto3, responses, pytools, pandoc, crc32c, filelock, humanfriendly, pandas, pygments, pysftp, yte, ftputil, httpretty, polars, requests-mock, docutils, snakemake-storage-plugin-http, snakemake-storage-plugin-s3, snakemake-storage-plugin-fs, snakemake-executor-plugin-cluster-generic, conda, pip PyPI Dependencies: snakemake, appdirs, immutables, configargparse, connection-pool, docutils, gitpython, humanfriendly, jinja2, jsonschema, nbformat, packaging, psutil, pulp, pyyaml, requests, reretry, smart-open, snakemake-interface-executor-plugins, snakemake-interface-common, snakemake-interface-storage-plugins, snakemake-interface-report-plugins, tabulate, throttler, wrapt, yte, dpath, conda-inject Target platforms: osx-64, linux-aarch64, linux-64, osx-arm64, win-64 Tasks: test-all, test-simple, test-complex Environment: py312 Features: py312, test, default Channels: conda-forge, bioconda Dependency count: 32 Dependencies: python, pytest, pytest-mock, pytest-cov, pytest-split, graphviz, oauth2client, numpy, glpk, pulp, boto3, responses, pytools, pandoc, crc32c, filelock, humanfriendly, pandas, pygments, pysftp, yte, ftputil, httpretty, polars, requests-mock, docutils, snakemake-storage-plugin-http, snakemake-storage-plugin-s3, snakemake-storage-plugin-fs, snakemake-executor-plugin-cluster-generic, conda, pip PyPI Dependencies: snakemake, appdirs, immutables, configargparse, connection-pool, docutils, gitpython, humanfriendly, jinja2, jsonschema, nbformat, packaging, psutil, pulp, pyyaml, requests, reretry, smart-open, snakemake-interface-executor-plugins, snakemake-interface-common, snakemake-interface-storage-plugins, snakemake-interface-report-plugins, tabulate, throttler, wrapt, yte, dpath, conda-inject Target platforms: win-64, osx-64, osx-arm64, linux-aarch64, linux-64 Tasks: test-complex, test-all, test-simple Environment: docs Features: docs, default Channels: conda-forge, bioconda Dependency count: 9 Dependencies: sphinx, sphinx-argparse, sphinx_rtd_theme, sphinxawesome-theme, recommonmark, commonmark, myst-parser, sphinxcontrib-napoleon, python PyPI Dependencies: snakemake, appdirs, immutables, configargparse, connection-pool, docutils, gitpython, humanfriendly, jinja2, jsonschema, nbformat, packaging, psutil, pulp, pyyaml, requests, reretry, smart-open, snakemake-interface-executor-plugins, snakemake-interface-common, snakemake-interface-storage-plugins, snakemake-interface-report-plugins, tabulate, throttler, wrapt, yte, dpath, conda-inject Target platforms: osx-arm64, win-64, linux-64, linux-aarch64, osx-64 Tasks: docs, build-docs Environment: quality Features: style, default Channels: conda-forge, bioconda Dependency count: 2 Dependencies: black, python PyPI Dependencies: snakemake, appdirs, immutables, configargparse, connection-pool, docutils, gitpython, humanfriendly, jinja2, jsonschema, nbformat, packaging, psutil, pulp, pyyaml, requests, reretry, smart-open, snakemake-interface-executor-plugins, snakemake-interface-common, snakemake-interface-storage-plugins, snakemake-interface-report-plugins, tabulate, throttler, wrapt, yte, dpath, conda-inject Target platforms: linux-aarch64, linux-64, win-64, osx-64, osx-arm64 Tasks: format Environment: publish Features: publish, default Channels: conda-forge, bioconda Dependency count: 4 Dependencies: setuptools, twine, python-build, python PyPI Dependencies: snakemake, appdirs, immutables, configargparse, connection-pool, docutils, gitpython, humanfriendly, jinja2, jsonschema, nbformat, packaging, psutil, pulp, pyyaml, requests, reretry, smart-open, snakemake-interface-executor-plugins, snakemake-interface-common, snakemake-interface-storage-plugins, snakemake-interface-report-plugins, tabulate, throttler, wrapt, yte, dpath, conda-inject Target platforms: osx-arm64, win-64, linux-aarch64, linux-64, osx-64 Tasks: check-build, build ``` </details> <details> <summary> pixi task ls </summary> ```console ➜ pixi task ls Tasks that can run on this machine: ----------------------------------- build, build-docs, check-build, docs, format, test-all, test-complex, test-simple - build Build the package into the dist/ directory - build-docs Build the documentation in the docs/ directory - check-build Check that the package can be uploaded - docs Serve the documentation on http://localhost:8000 - format Format the code - test-all Runs both simple and complex tests - test-complex Run all tests in the tests directory except tests.py - test-simple Run tests only for tests/tests.py ➜ pixi task ls -s Tasks per environment: ---------------------- py312: test-all, test-complex, test-simple quality: format publish: build, check-build docs: build-docs, docs py311: test-all, test-complex, test-simple default: dev: build-docs, docs, format, test-all, test-complex, test-simple ``` </details> Based on the github actions workflow which uses the old `test-environment.yaml` to install dependencies, which modify the `yaml` on-the-fly for windows testing, I also tried to define the `linux` specific test dependencies in the `[tool.pixi.feature.test.target.linux-64.dependencies]` For the most part I've tried to split up dependencies into re-usable features, otherwise, which can be better utilized across duplicated dependencies in the `test` and `docs` env ## Notes 1. I am not sure why the old `pyproject.toml` specified requiring `python < 3.11` for setuptools, https://github.com/snakemake/snakemake/blob/d1c369bcac77f9818373ca9a7e8940bf5a649c49/pyproject.toml#L3-L6 Should this be added back? 2. As explained in snakemake#3369 (comment) We ignore the lock file in library development to ensure tests reflect user environments while allowing developers to generate their own. 3. I was getting [this error in the test suite](https://github.com/snakemake/snakemake/actions/runs/13786010103/job/38554140354?pr=3369) which I managed to fix with 71163c8, not sure if there is a different reason for this though. ## Next Steps - [x] verify that the tests work in the test environment as expected - [x] verify the dependencies across the old and new configurations <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **Chores** - Improved file handling by adding new entries to `.gitattributes` and `.gitignore` for better management of the `pixi.lock` file. - Updated GitHub Actions workflows to utilize `Pixi` for formatting checks and testing, introducing a new job for testing with `Pixi`. - Removed the `setup.cfg` configuration file. - **Style** - Enhanced code formatting in file handling routines for improved maintainability. - **Tests** - Added decorators to test functions to indicate dependencies on specific environment modules. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Johannes Koester <johannes.koester@uni-due.de>
🤖 I have created a release *beep* *boop* --- ## [9.0.0](snakemake/snakemake@v8.30.0...v9.0.0) (2025-03-14) ### ⚠ BREAKING CHANGES * Logging refactor & add LoggerPluginInterface ([snakemake#3107](snakemake#3107)) ### Features * [snakemake#3412](snakemake#3412) - keep shadow folder of failed job if --keep-incomplete flag is set. ([snakemake#3430](snakemake#3430)) ([22978c3](snakemake@22978c3)) * add flag --report-after-run to automatically generate the report after a successfull workflow run ([snakemake#3428](snakemake#3428)) ([b0a7f03](snakemake@b0a7f03)) * add flatten function to IO utils ([snakemake#3424](snakemake#3424)) ([67fa392](snakemake@67fa392)) * add helper functions to parse input files ([snakemake#2918](snakemake#2918)) ([63e45a7](snakemake@63e45a7)) * Add option to print redacted file names ([snakemake#3089](snakemake#3089)) ([ba4d264](snakemake@ba4d264)) * add support for validation of polars dataframe and lazyframe ([snakemake#3262](snakemake#3262)) ([c7473a6](snakemake@c7473a6)) * added support for rendering dag with mermaid js ([snakemake#3409](snakemake#3409)) ([7bf8381](snakemake@7bf8381)) * adding --replace-workflow-config to fully replace workflow configs (from config: directive) with --configfile, instead of merging them ([snakemake#3381](snakemake#3381)) ([47504a0](snakemake@47504a0)) * Dynamic module name ([snakemake#3401](snakemake#3401)) ([024dc32](snakemake@024dc32)) * Enable saving and reloading IOCache object ([snakemake#3386](snakemake#3386)) ([c935953](snakemake@c935953)) * files added in rule params with workflow.source_path will be available in used containers ([snakemake#3385](snakemake#3385)) ([a6e45bf](snakemake@a6e45bf)) * Fix keep_local in storage directive and more freedom over remote retrieval behaviour ([snakemake#3410](snakemake#3410)) ([67b4739](snakemake@67b4739)) * inherit parameters of use rule and extend/replace individual items them when using 'with' directive ([snakemake#3365](snakemake#3365)) ([93e4b92](snakemake@93e4b92)) * Logging refactor & add LoggerPluginInterface ([snakemake#3107](snakemake#3107)) ([86f1d6e](snakemake@86f1d6e)) * Maximal file size for checksums ([snakemake#3368](snakemake#3368)) ([b039f8a](snakemake@b039f8a)) * Modernize package configuration using Pixi ([snakemake#3369](snakemake#3369)) ([77992d8](snakemake@77992d8)) * multiext support for named input/output ([snakemake#3372](snakemake#3372)) ([05e1378](snakemake@05e1378)) * optionally auto-group jobs via temp files in case of remote execution ([snakemake#3378](snakemake#3378)) ([cc9bba2](snakemake@cc9bba2)) ### Bug Fixes * `--delete-all-output` ignores `--dry-run` ([snakemake#3265](snakemake#3265)) ([23fef82](snakemake@23fef82)) * 3342 faster touch runs and warning messages for non-existing files ([snakemake#3398](snakemake#3398)) ([cd9c3c3](snakemake@cd9c3c3)) * add default value to max-jobs-per-timespan ([snakemake#3043](snakemake#3043)) ([2959abe](snakemake@2959abe)) * checkpoints inside modules are overwritten ([snakemake#3359](snakemake#3359)) ([fba3ac7](snakemake@fba3ac7)) * Convert Path to IOFile ([snakemake#3405](snakemake#3405)) ([c58684c](snakemake@c58684c)) * Do not perform storage object cleanup with --keep-storage-local-copies set ([snakemake#3358](snakemake#3358)) ([9a6d14b](snakemake@9a6d14b)) * edgecases of source deployment in case of remote execution ([snakemake#3396](snakemake#3396)) ([5da13be](snakemake@5da13be)) * enhance error message formatting for strict DAG-building mode ([snakemake#3376](snakemake#3376)) ([a1c39ee](snakemake@a1c39ee)) * fix bug in checkpoint handling that led to exceptions in case checkpoint output was missing upon rerun ([snakemake#3423](snakemake#3423)) ([8cf4a2f](snakemake@8cf4a2f)) * force check all required outputs ([snakemake#3341](snakemake#3341)) ([495a4e7](snakemake@495a4e7)) * group job formatting ([snakemake#3442](snakemake#3442)) ([f0b10a3](snakemake@f0b10a3)) * in remote jobs, upload storage in topological order such that modification dates are preserved (e.g. in case of group jobs) ([snakemake#3377](snakemake#3377)) ([eace08f](snakemake@eace08f)) * only skip eval when resource depends on input ([snakemake#3374](snakemake#3374)) ([4574c92](snakemake@4574c92)) * Prevent execution of conda in apptainer when not explicitly requested in software deployment method ([snakemake#3388](snakemake#3388)) ([c43c5c0](snakemake@c43c5c0)) * print filenames with quotes around them in RuleException ([snakemake#3269](snakemake#3269)) ([6baeda5](snakemake@6baeda5)) * Re-evaluation of free resources ([snakemake#3399](snakemake#3399)) ([6371293](snakemake@6371293)) * ReadTheDocs layout issue due to src directory change ([snakemake#3419](snakemake#3419)) ([695b127](snakemake@695b127)) * robustly escaping quotes in generated bash scripts (v2) ([snakemake#3297](snakemake#3297)) ([snakemake#3389](snakemake#3389)) ([58720bd](snakemake@58720bd)) * Show apptainer image URL in snakemake report ([snakemake#3407](snakemake#3407)) ([45f0450](snakemake@45f0450)) * Update ReadTheDocs configuration for documentation build to use Pixi ([snakemake#3433](snakemake#3433)) ([3f227a6](snakemake@3f227a6)) ### Documentation * Add pixi setup instructions to general use tutorial ([snakemake#3382](snakemake#3382)) ([115e81b](snakemake@115e81b)) * fix contribution section heading levels, fix docs testing setup order ([snakemake#3360](snakemake#3360)) ([051dc53](snakemake@051dc53)) * fix link to github.com/snakemake/poetry-snakemake-plugin ([snakemake#3436](snakemake#3436)) ([ec6d97c](snakemake@ec6d97c)) * fix quoting ([snakemake#3394](snakemake#3394)) ([b40f599](snakemake@b40f599)) * fix rerun-triggers default ([snakemake#3403](snakemake#3403)) ([4430e23](snakemake@4430e23)) * fix typo 'safe' -> 'save' ([snakemake#3384](snakemake#3384)) ([7755861](snakemake@7755861)) * mention code formatting in the contribution section ([snakemake#3431](snakemake#3431)) ([e8682b7](snakemake@e8682b7)) * remove duplicated 'functions'. ([snakemake#3356](snakemake#3356)) ([7c595db](snakemake@7c595db)) * update broken links documentation ([snakemake#3437](snakemake#3437)) ([e3d0d88](snakemake@e3d0d88)) * Updating contributing guidelines with new pixi dev setup ([snakemake#3415](snakemake#3415)) ([8e95a12](snakemake@8e95a12)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: snakemake-bot <snakemake-bot-admin@googlegroups.com>
resolves #3347
This PR is the first step in modernizing the project configuration by moving the
setup.cfgsections to pyproject.toml, and defining environments using pixi.pixi info
pixi task ls
Based on the github actions workflow which uses the old
test-environment.yamlto install dependencies, which modify theyamlon-the-fly for windows testing, I also tried to define thelinuxspecific test dependencies in the[tool.pixi.feature.test.target.linux-64.dependencies]For the most part I've tried to split up dependencies into re-usable features, otherwise, which can be better utilized across duplicated dependencies in the
testanddocsenvNotes
I am not sure why the old
pyproject.tomlspecified requiringpython < 3.11for setuptools,snakemake/pyproject.toml
Lines 3 to 6 in d1c369b
Should this be added back?
As explained in feat: Modernize package configuration using Pixi #3369 (comment)
We ignore the lock file in library development to ensure tests reflect user environments while allowing developers to generate their own.
I was getting this error in the test suite which I managed to fix with 71163c8, not sure if there is a different reason for this though.
Next Steps
Summary by CodeRabbit
Summary by CodeRabbit
Chores
.gitattributesand.gitignorefor better management of thepixi.lockfile.Pixifor formatting checks and testing, introducing a new job for testing withPixi.setup.cfgconfiguration file.Style
Tests