Skip to content

fix: convert parameters so they can be serialized#2925

Merged
johanneskoester merged 8 commits intosnakemake:mainfrom
fgvieira:params_serialize
Dec 21, 2024
Merged

fix: convert parameters so they can be serialized#2925
johanneskoester merged 8 commits intosnakemake:mainfrom
fgvieira:params_serialize

Conversation

@fgvieira
Copy link
Copy Markdown
Contributor

@fgvieira fgvieira commented Jun 23, 2024

Some parameter types (e.g. Path or np) are not serializable when requesting extended benchmarks (and maybe in other other places #1425):

$ snakemake test.out -j 1 --benchmark-extended -F
Assuming unrestricted shared filesystem usage.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job      count
-----  -------
1            1
total        1

Select jobs to execute...
Execute 1 jobs...

[Thu Jul 11 11:07:47 2024]
localrule 1:
    output: test.out
    jobid: 0
    benchmark: test.jsonl
    reason: Forced execution
    resources: tmpdir=/tmp

WorkflowError:
TypeError: Object of type int64 is not JSON serializable
[Thu Jul 11 11:07:50 2024]
Error in rule 1:
    jobid: 0
    output: test.out

Exiting because a job execution failed. Look above for error message
WorkflowError:
At least one job did not complete successfully.
[Thu Jul 11 11:07:50 2024]
Error in rule 1:
    jobid: 0
    output: test.out

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-07-11T110746.832532.snakemake.log
WorkflowError:
At least one job did not complete successfully.

Not sure why, but the tests all pass! 😕

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Summary by CodeRabbit

  • New Features

    • Added new parameters for testing: path and np to enhance test configurations.
    • Introduced performance benchmarking for the test_nonstr_params function.
  • Bug Fixes

    • Updated the test to skip on Windows platforms, addressing compatibility issues.

@fgvieira fgvieira marked this pull request as draft June 23, 2024 14:41
@fgvieira fgvieira changed the title fix: add test to check it fails fix: convert parameters so they can be serialized Jun 27, 2024
@fgvieira fgvieira marked this pull request as ready for review July 11, 2024 09:27
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Aug 7, 2024

@johanneskoester
Copy link
Copy Markdown
Contributor

I think this has become obselete with #3175. Please reopen if I am wrong.

@fgvieira
Copy link
Copy Markdown
Contributor Author

fgvieira commented Nov 12, 2024

Yes, it seems it has been fixed, but then we can just merged the PR to include the extra tests, no?

@fgvieira fgvieira reopened this Nov 12, 2024
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Nov 12, 2024

📝 Walkthrough
📝 Walkthrough

Walkthrough

The changes involve modifications to the Snakefile and the tests.py file. In the Snakefile, two parameters (path and np) are added to the params section, and the run section is updated to echo these parameters to an output file. In tests.py, the test_nonstr_params function is decorated to skip on Windows and now includes performance benchmarking in its execution.

Changes

File Change Summary
tests/test_nonstr_params/Snakefile - Added imports for numpy and Path.
- Added parameters path and np in params.
- Updated run section to echo params.path and params.np to "test.out". Removed the previous command that created an empty file.
tests/tests.py - Decorated test_nonstr_params with @skip_on_windows.
- Updated invocation of run to include benchmark_extended=True.

Suggested reviewers

  • johanneskoester

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 341a79a and 5e3786c.

📒 Files selected for processing (1)
  • tests/tests.py (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/tests.py

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@sonarqubecloud
Copy link
Copy Markdown

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Outside diff range and nitpick comments (1)
tests/test_nonstr_params/Snakefile (1)

10-12: Consider using a platform-independent temporary path

The hardcoded path "/tmp" might cause issues on Windows systems. Consider using a more portable solution.

-        path=Path("/tmp"),
+        path=Path(tempfile.gettempdir()),

Don't forget to add the import:

import tempfile
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 76d5329 and 341a79a.

⛔ Files ignored due to path filters (1)
  • tests/test_nonstr_params/expected-results/test.out is excluded by !**/*.out
📒 Files selected for processing (2)
  • tests/test_nonstr_params/Snakefile (1 hunks)
  • tests/tests.py (1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
tests/tests.py (1)

Pattern **/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

🔇 Additional comments (4)
tests/test_nonstr_params/Snakefile (3)

1-2: LGTM: Required imports added correctly

The imports for numpy and Path are necessary for the test case and are properly defined.


7-8: LGTM: Benchmark configuration added

The benchmark configuration is correctly set up to test the serialization functionality.


1-15: Verify if changes are still needed after PR #3175

Based on the PR comments, there might be an overlap with PR #3175. While these changes look correct, we should verify if they're still necessary.

#!/bin/bash
# Description: Check for related changes in PR #3175
# Look for similar parameter serialization fixes

# Check for related files in the main branch
gh pr view 3175 --json files -q '.files[].path' | grep -i "param\|serial"

# Look for related code changes
gh pr view 3175 --json files -q '.files[].patch' | grep -i "param\|serial\|json"
tests/tests.py (1)

464-466: LGTM: Changes align with PR objectives.

The modifications appropriately address the serialization issues:

  1. The @skip_on_windows decorator is correctly added as path handling differs on Windows.
  2. The benchmark_extended=True parameter enables extended benchmarking, which helps test the serialization of non-string parameters.

@sonarqubecloud
Copy link
Copy Markdown

@johanneskoester johanneskoester merged commit 9e653fb into snakemake:main Dec 21, 2024
@fgvieira fgvieira deleted the params_serialize branch December 22, 2024 20:22
johanneskoester pushed a commit that referenced this pull request Dec 23, 2024
🤖 I have created a release *beep* *boop*
---


##
[8.26.0](v8.25.5...v8.26.0)
(2024-12-23)


### Features

* add helpers for deferred input/output etc. item access
([#2927](#2927))
([2cca9bc](2cca9bc))


### Bug Fixes

* convert parameters so they can be serialized
([#2925](#2925))
([9e653fb](9e653fb))
* correct formatting of R preamble
([#2425](#2425))
([5380cae](5380cae))
* fix modification checks for scripts and and notebooks containing
wildcards or params in their paths
([#2751](#2751))
([773568d](773568d))
* Improved handling of missing output files in group job postprocessing,
accounting for temporary files.
([#1765](#1765))
([bac06ba](bac06ba))
* mtime of script or notebook not triggering workflow without metadata
([#3148](#3148))
([e8a0b83](e8a0b83))
* Pass `host` attribute to `GitlabFile` instantiation within class
methods ([#3155](#3155))
([9ef52de](9ef52de))
* problem with spaces in path
([#3236](#3236))
([2d08c63](2d08c63))
* require current yte release which contains an important bug fix for
cases where numpy/pandas data is passed to templates
([#3227](#3227))
([c3339da](c3339da))
* rerun jobs if previously failed but rule was changed afterwards
(thanks to [@laf070810](https://github.com/laf070810) for bringing this
up) ([#3237](#3237))
([1dc0084](1dc0084))
* use relpath for configfiles added to the source archive (thanks to
[@sposadac](https://github.com/sposadac) for the initial solution)
([#3240](#3240))
([bff3844](bff3844))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@coderabbitai coderabbitai bot mentioned this pull request Jul 24, 2025
2 tasks
kjohnsen pushed a commit to kjohnsen/snakemake that referenced this pull request Dec 15, 2025
<!--Add a description of your PR here-->

Some parameter types (e.g. `Path` or `np`) are not serializable when
requesting extended benchmarks (and maybe in other other places snakemake#1425):

```bash
$ snakemake test.out -j 1 --benchmark-extended -F
Assuming unrestricted shared filesystem usage.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job      count
-----  -------
1            1
total        1

Select jobs to execute...
Execute 1 jobs...

[Thu Jul 11 11:07:47 2024]
localrule 1:
    output: test.out
    jobid: 0
    benchmark: test.jsonl
    reason: Forced execution
    resources: tmpdir=/tmp

WorkflowError:
TypeError: Object of type int64 is not JSON serializable
[Thu Jul 11 11:07:50 2024]
Error in rule 1:
    jobid: 0
    output: test.out

Exiting because a job execution failed. Look above for error message
WorkflowError:
At least one job did not complete successfully.
[Thu Jul 11 11:07:50 2024]
Error in rule 1:
    jobid: 0
    output: test.out

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-07-11T110746.832532.snakemake.log
WorkflowError:
At least one job did not complete successfully.
```

Not sure why, but the tests all pass! :confused: 


### QC
<!-- Make sure that you can tick the boxes below. -->

* [x] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [x] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added new parameters for testing: `path` and `np` to enhance test
configurations.
- Introduced performance benchmarking for the `test_nonstr_params`
function.

- **Bug Fixes**
- Updated the test to skip on Windows platforms, addressing
compatibility issues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de>
kjohnsen pushed a commit to kjohnsen/snakemake that referenced this pull request Dec 15, 2025
🤖 I have created a release *beep* *boop*
---


##
[8.26.0](snakemake/snakemake@v8.25.5...v8.26.0)
(2024-12-23)


### Features

* add helpers for deferred input/output etc. item access
([snakemake#2927](snakemake#2927))
([2cca9bc](snakemake@2cca9bc))


### Bug Fixes

* convert parameters so they can be serialized
([snakemake#2925](snakemake#2925))
([9e653fb](snakemake@9e653fb))
* correct formatting of R preamble
([snakemake#2425](snakemake#2425))
([5380cae](snakemake@5380cae))
* fix modification checks for scripts and and notebooks containing
wildcards or params in their paths
([snakemake#2751](snakemake#2751))
([773568d](snakemake@773568d))
* Improved handling of missing output files in group job postprocessing,
accounting for temporary files.
([snakemake#1765](snakemake#1765))
([bac06ba](snakemake@bac06ba))
* mtime of script or notebook not triggering workflow without metadata
([snakemake#3148](snakemake#3148))
([e8a0b83](snakemake@e8a0b83))
* Pass `host` attribute to `GitlabFile` instantiation within class
methods ([snakemake#3155](snakemake#3155))
([9ef52de](snakemake@9ef52de))
* problem with spaces in path
([snakemake#3236](snakemake#3236))
([2d08c63](snakemake@2d08c63))
* require current yte release which contains an important bug fix for
cases where numpy/pandas data is passed to templates
([snakemake#3227](snakemake#3227))
([c3339da](snakemake@c3339da))
* rerun jobs if previously failed but rule was changed afterwards
(thanks to [@laf070810](https://github.com/laf070810) for bringing this
up) ([snakemake#3237](snakemake#3237))
([1dc0084](snakemake@1dc0084))
* use relpath for configfiles added to the source archive (thanks to
[@sposadac](https://github.com/sposadac) for the initial solution)
([snakemake#3240](snakemake#3240))
([bff3844](snakemake@bff3844))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants