Skip to content

fix: more robust handling of input metadata that avoids storing user or typoe specific local paths, and properly handles pipe or service files#3150

Merged
johanneskoester merged 2 commits intomainfrom
fix/input-metadata
Oct 17, 2024
Merged

fix: more robust handling of input metadata that avoids storing user or typoe specific local paths, and properly handles pipe or service files#3150
johanneskoester merged 2 commits intomainfrom
fix/input-metadata

Conversation

@johanneskoester
Copy link
Copy Markdown
Contributor

@johanneskoester johanneskoester commented Oct 17, 2024 •
edited by coderabbitai bot
Loading

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Summary by CodeRabbit

  • New Features

    • Updated record format version to enhance how records are managed and interpreted.
    • Improved input path handling for better clarity and functionality.

  • Bug Fixes

    • Ensured compatibility with the new record format version for input change detection.

  • Documentation

    • Enhanced metadata tracking to include the new record format version across job outputs.

@johanneskoester
fix: more robust handling of input metadata that avoids storing user …
4fcc0a6 
…or typoe specific local paths, and properly handles pipe or service files
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Oct 17, 2024 •
edited
Loading

Walkthrough

The changes in this pull request primarily affect the snakemake/persistence.py file, specifically the Persistence class. The RECORD_FORMAT_VERSION constant has been updated from 3 to 4, impacting record management. The _input method has been refactored to use a generator function for processing input paths, enhancing clarity. The _input_changed method now includes a check for the new record format version, and the finished method has been updated to record this version in job outputs. These modifications improve how input paths are processed and metadata is structured.

Changes

File Change Summary
snakemake/persistence.py - Updated RECORD_FORMAT_VERSION from 3 to 4.
- Refactored _input method to use get_paths() generator.
- Modified _input_changed method to check for format version 4 or higher.
- Updated finished method to include record_format_version in metadata.

Possibly related PRs

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between 6416c36 and f707780.

📒 Files selected for processing (1)
  • snakemake/persistence.py (3 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
snakemake/persistence.py (1)

Pattern **/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

🔇 Additional comments (3)
snakemake/persistence.py (3)

35-35: Update of RECORD_FORMAT_VERSION

The increment of RECORD_FORMAT_VERSION to 4 appropriately reflects the changes in the record format and ensures consistency across the system.

485-488: Proper version check in _input_changed method

The addition of the format version check in _input_changed enhances compatibility by ensuring that only files with the new record format version are considered, preventing potential issues with outdated metadata.

524-540: Improved input processing in _input method

The refactored _input method now robustly handles various input types, such as storage objects, pipes, services, and source cache entries. This enhancement increases the method's clarity and ensures accurate processing of input paths.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Oct 17, 2024

Please format your code with black: black snakemake tests/*.py.

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Oct 17, 2024

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@johanneskoester johanneskoester merged commit 756dc70 into main Oct 17, 2024
@johanneskoester johanneskoester deleted the fix/input-metadata branch October 17, 2024 14:48
@github-actions github-actions bot mentioned this pull request Oct 17, 2024
Merged
This was referenced Oct 17, 2024
Closed
Merged
johanneskoester pushed a commit that referenced this pull request Oct 18, 2024
@github-actions
chore(main): release 8.23.2 (#3149)
9504bf4 
🤖 I have created a release *beep* *boop*
---


##
[8.23.2](v8.23.1...v8.23.2)
(2024-10-18)


### Bug Fixes

* ignore derived parameters when storing job metadata and inferring
rerun necessity of jobs (this gets rid of spurious triggers caused by
e.g. changed resources, threads, remote storage configuration)
([#3154](#3154))
([73ce212](73ce212))
* more robust handling of input metadata that avoids storing user or
typoe specific local paths, and properly handles pipe or service files
([#3150](#3150))
([756dc70](756dc70))


### Documentation

* Fix missing spaces in CLI help text
([#3146](#3146))
([6416c36](6416c36))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@coderabbitai coderabbitai bot mentioned this pull request Oct 21, 2024
Merged
2 tasks
This was referenced Oct 30, 2024
Merged
Merged
@coderabbitai coderabbitai bot mentioned this pull request Nov 12, 2024
Merged
@coderabbitai coderabbitai bot mentioned this pull request Feb 27, 2025
Merged
2 tasks
@coderabbitai coderabbitai bot mentioned this pull request Aug 28, 2025
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@johanneskoester