Comparing changes

…for cases where numpy/pandas data is passed to templates (#3227)  ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit - **Chores** - Updated the version constraint for the `yte` package to ensure compatibility with newer features.

### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit - **New Features** - Enhanced documentation structure for various Snakemake packages, including `snakemake.assets`, `snakemake.caching`, and others. - Introduced new utility functions and classes for improved path handling and file management. - Added new test cases to validate the functionality of the `subpath` utility. - New GitHub Actions workflow for automated documentation building. - **Bug Fixes** - Improved error handling and logging capabilities in the Snakemake API. - **Documentation** - Updated URLs and references in installation documentation. - Restructured internal API documentation for better clarity. - **Chores** - Removed outdated or redundant files and dependencies from the codebase.  --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

Fixes #3235 Goes together with snakemake/snakemake-interface-executor-plugins#79 This PR updates tests to catch the bug (should break at least `test_path_with_spaces` and `test_with_parentheses` and snakemake/snakemake-interface-executor-plugins#79 actually fixes it  ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Introduced a new rule that generates an output file with a predefined string. - **Tests** - Added a test case to verify the handling of paths containing spaces. - Enhanced temporary directory naming for clarity during test execution.  --------- Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de> Co-authored-by: Johannes Koester <johannes.koester@uni-due.de>

…data (#3148)  See #3014 (review) ### QC  * [ ] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [ ] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit - **Bug Fixes** - Improved logic for determining job execution conditions, ensuring accurate evaluations. - **Refactor** - Streamlined the `update_needrun` method for better readability and maintainability. - Consolidated checks for job metadata and conditions to eliminate redundancy. - **Style** - Minor adjustments to comments and formatting for enhanced clarity.  Co-authored-by: Anfeng Li <anfeng.li@cern.ch> Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de>

…ith source (#3164)  When running a workflow with a Snakefile using the `github` function (see below) which references an R script which sources a file an error like, "Cannot change the working directory" appears. This changes the source logic to only change the working directory if the script is not a URL. I also changed the code to use `on.exit(setwd(old_wd), add = TRUE)` instead of relying on the final `setwd(wd)` to be a little safer in case there's any errors later. That way the user's working directory is not changed if `snakemake@source(...)` is in a try-catch-except block. #### Snakefile ``` module module_name: snakefile: github("GITHUB_REPO", path="workflow/Snakefile", tag="v0.1.5") config: config ``` #### R script in module_name ``` ... # Fails on next line snakemake@source("file_hosted_on_github") ... ``` ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit - **New Features** - Improved handling of script directories in R scripts, ensuring reliability when sourcing scripts from various locations. - Enhanced logic for generating preambles in R scripts to reflect the new directory handling. - **Bug Fixes** - Resolved issues related to changing the working directory when the script directory is a URL. - **Documentation** - Minor adjustments made for clarity in comments and formatting.  Co-authored-by: rschauner <rschauner@fastmail.com>

@laf070810

…(thanks to @laf070810 for bringing this up) (#3237)  ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit - **New Features** - Introduced a new Snakemake workflow with rules for generating intermediate files and handling conditional failures. - Added a new test function to validate job metadata persistence during error scenarios. - **Bug Fixes** - Streamlined error handling in job completion processes to enhance clarity and efficiency. - **Documentation** - Updated expected results for tests to reflect new workflows and configurations. - **Chores** - Adjusted temporary directory handling in test functions for improved flexibility.

Some parameter types (e.g. `Path` or `np`) are not serializable when requesting extended benchmarks (and maybe in other other places #1425): ```bash $ snakemake test.out -j 1 --benchmark-extended -F Assuming unrestricted shared filesystem usage. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count ----- ------- 1 1 total 1 Select jobs to execute... Execute 1 jobs... [Thu Jul 11 11:07:47 2024] localrule 1: output: test.out jobid: 0 benchmark: test.jsonl reason: Forced execution resources: tmpdir=/tmp WorkflowError: TypeError: Object of type int64 is not JSON serializable [Thu Jul 11 11:07:50 2024] Error in rule 1: jobid: 0 output: test.out Exiting because a job execution failed. Look above for error message WorkflowError: At least one job did not complete successfully. [Thu Jul 11 11:07:50 2024] Error in rule 1: jobid: 0 output: test.out Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-07-11T110746.832532.snakemake.log WorkflowError: At least one job did not complete successfully. ``` Not sure why, but the tests all pass! :confused: ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit - **New Features** - Added new parameters for testing: `path` and `np` to enhance test configurations. - Introduced performance benchmarking for the `test_nonstr_params` function. - **Bug Fixes** - Updated the test to skip on Windows platforms, addressing compatibility issues.  --------- Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de>

### Description Currently parameters given to the R script preamble do not work for some (uncommon) cases: - special float values (NaN, inf, -inf) - complex numbers - integers (type preserved) - heterogeneous lists These cases are corrected in this PR. Heterogeneous lists are now supported by creating an R `list` instead of a R `vector`, if the Python `list` contents are not from the same type. Therefore, this change may break some workflows, if they rely on the R script input being a vector or the implicitly converted types (e.g. before this PR `TRUE` became `"TRUE"` if there is another string in the list). In my opinion, I would regard the implicit type conversion a bug, hence my change proposal. ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake). Co-authored-by: Johannes Koester <johannes.koester@uni-due.de>

… wildcards or params in their paths (#2751) Fixes #2750 ### Description Parse wildcards in script names before checking file timestamps.  ## Summary by CodeRabbit - **New Features** - Enhanced script path handling for improved flexibility in dynamic resolution based on provided parameters. - **Bug Fixes** - Improved control flow to ensure accurate script path determination before further operations.  --------- Co-authored-by: mawenzy <mawenzy@users.noreply.github.com> Co-authored-by: Filipe G. Vieira <1151762+fgvieira@users.noreply.github.com> Co-authored-by: Johannes Koester <johannes.koester@uni-due.de> Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de>

… methods (#3155) Updates `GitlabFile` to re-implement the `HostingProviderFile` methods `get_basedir` and `join` that create new class instances to include handling for the `GitlabFile` specific `host` attribute. ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit - **New Features** - Enhanced `GitlabFile` class with a default host parameter and added methods for better file path management. - Introduced `get_basedir` and `join` methods for improved functionality. - **Bug Fixes** - Updated `get_path_or_uri` method for consistent host attribute usage. - **Tests** - Added a new test to ensure the `host` attribute is correctly propagated in the `GitlabFile` class.  --------- Co-authored-by: Johannes Koester <johannes.koester@uni-due.de>

…ssing, accounting for temporary files. (#1765) When I run group jobs on a cluster, I'll often write temporary files to the node-specific local scratch folder. I'll label them `temp(...)` and the sub-snakemake instance will dutifully delete the files when no longer needed. Even without that deletion, the files would disappear with the node local scratch as soon as the group job finishes. The problem occurs when the parent snakemake instance re-assumes control. It checks for all the outputs from the group job, finds the temp files missing, and exits with an error. But these temp files are no longer needed anywhere in the workflow, and indeed, Snakemake's immediate next step if it were to find the temp files would be to delete them. This PR filters the group job outputs before the existence checks, removing any that are: - Not needed anywhere else in the workflow - Not target files In particular, only files that would otherwise be immediately removed are excluded from the existence check (at least, that's the idea). The idea is to remove a small frustration without causing any breaking or generally noticeable changes. I've no test yet, but I'll write one if the concept here is approved. ### QC  * [ ] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [ ] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit - **New Features** - Enhanced output checking mechanism to selectively ignore specific output files. - Improved handling of missing output files in job postprocessing, accounting for temporary files. - **Bug Fixes** - Refined logic for determining if files are needed by pending jobs, improving accuracy in output management.  --------- Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de> Co-authored-by: Johannes Koester <johannes.koester@uni-due.de>

@sposadac

@sposadac for the initial solution) (#3240)  ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit - **New Features** - Enhanced logging messages for ignored source files to provide clearer user guidance regarding directory structure. - Added functionality to handle configuration file arguments, including support for relative paths based on shared file system usage. - **Bug Fixes** - Improved consistency in how configuration file paths are recorded, ensuring they are stored as relative paths.

@laf070810

🤖 I have created a release *beep* *boop* --- ## [8.26.0](v8.25.5...v8.26.0) (2024-12-23) ### Features * add helpers for deferred input/output etc. item access ([#2927](#2927)) ([2cca9bc](2cca9bc)) ### Bug Fixes * convert parameters so they can be serialized ([#2925](#2925)) ([9e653fb](9e653fb)) * correct formatting of R preamble ([#2425](#2425)) ([5380cae](5380cae)) * fix modification checks for scripts and and notebooks containing wildcards or params in their paths ([#2751](#2751)) ([773568d](773568d)) * Improved handling of missing output files in group job postprocessing, accounting for temporary files. ([#1765](#1765)) ([bac06ba](bac06ba)) * mtime of script or notebook not triggering workflow without metadata ([#3148](#3148)) ([e8a0b83](e8a0b83)) * Pass `host` attribute to `GitlabFile` instantiation within class methods ([#3155](#3155)) ([9ef52de](9ef52de)) * problem with spaces in path ([#3236](#3236)) ([2d08c63](2d08c63)) * require current yte release which contains an important bug fix for cases where numpy/pandas data is passed to templates ([#3227](#3227)) ([c3339da](c3339da)) * rerun jobs if previously failed but rule was changed afterwards (thanks to [@laf070810](https://github.com/laf070810) for bringing this up) ([#3237](#3237)) ([1dc0084](1dc0084)) * use relpath for configfiles added to the source archive (thanks to [@sposadac](https://github.com/sposadac) for the initial solution) ([#3240](#3240)) ([bff3844](bff3844)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comparing changes

Open a pull request

Commits on Dec 11, 2024

Commits on Dec 21, 2024

Commits on Dec 23, 2024

This comparison is taking too long to generate.

Uh oh!