Skip to content

feat: files added in rule params with workflow.source_path will be available in used containers #3385

Merged
johanneskoester merged 21 commits intosnakemake:mainfrom
SrPeter128:mount_cache_to_container
Mar 14, 2025
Merged

feat: files added in rule params with workflow.source_path will be available in used containers #3385
johanneskoester merged 21 commits intosnakemake:mainfrom
SrPeter128:mount_cache_to_container

Conversation

@SrPeter128
Copy link
Copy Markdown
Contributor

@SrPeter128 SrPeter128 commented Mar 11, 2025

If files are specified under params using workflow.source_path, they are also automatically available in the container. This was achieved by mounting the nextflow source cache in the container by default. This achieves a behavior which is rather natural, as the user probably expects that files which are specified in params using workflow.source_path are also available in the container.

Due to the fact that the source cache is normally located under /home//.cache/snakemake/snakemake/source-cache and the container thus gets access to the home directory despite the use of the --no-home flag used in this PR, the documentation explicitly points out that only trustworthy containers should be used.

QC

  • [ x] The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • [ x] The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Summary by CodeRabbit

This release improves container workflow execution and documentation, ensuring a smoother and more secure experience for users.

  • New Features
    • Enhanced container integration now automatically binds local cache directories to improve workflow performance.
    • Added new test scenarios to validate Snakemake execution with Apptainer deployment.
  • Documentation
    • Updated guidance on file accessibility within container environments, with added security recommendations to use trusted container images.
  • Tests
    • Introduced additional test functions to cover various Snakemake execution scenarios.
    • New input and expected output files added for comprehensive testing.

SrPeter128 added 2 commits March 11, 2025 15:14
…vailable in the container by default by mounting the .cache/snakemake/snakemake/source-cache/ if no other --bind for the container is given
…de the home folder from the standard mounted folders
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 11, 2025

📝 Walkthrough

Walkthrough

This pull request introduces new test functions and documentation enhancements for container-based Snakemake workflows. It adds two test functions in tests/tests.py to validate execution scenarios with Apptainer, including both passing and failing cases. New Snakefiles, input files, and expected output files have been created under separate test directories for these scenarios. The documentation in docs/snakefiles/deployment.rst clarifies how files from workflow.source_path are mounted via the Snakemake cache. Additionally, the Singularity deployment code is updated to mount the Snakemake cache if present, with minor whitespace cleanup in the workflow module.

Changes

File(s) Change Summary
tests/tests.py Added two new test functions (test_issue3361_pass and test_issue3361_fail) for Snakemake commands with Apptainer; both are skipped on Windows.
docs/snakefiles/.../deployment.rst Updated documentation explaining that files from workflow.source_path are mounted via the Snakemake cache and advising the use of trusted containers.
tests/test_issue3361_fail/... Added a new Snakefile, input file, expected output file, and supplementary input (some_other_input.txt) for testing a failure scenario via a Docker container.
tests/test_issue3361_pass/... Added a new Snakefile, input file, expected output file, and supplementary input (some_other_input.txt) for testing a passing scenario via a Docker container.
src/snakemake/deployment/singularity.py Updated the shellcmd function to check for and mount the Snakemake cache directory if available; added an import for get_appdirs.
src/snakemake/workflow.py Removed extraneous blank lines in the _sourcecache initialization and source_path method (no functional changes).

Sequence Diagram(s)

sequenceDiagram
  participant T as Test Function
  participant R as Runner (run method)
  participant S as Snakemake
  participant D as Docker Container

  T->>R: Invoke test_issue3361_pass/test_issue3361_fail
  R->>S: Execute "snakemake --sdm apptainer" with target ["all"]
  S->>D: Run workflow as defined in Snakefile (copy_in_container rule)
  D->>S: Return generated output
  S->>R: Relay command results
  R->>T: Validate output or expected error condition
Loading
sequenceDiagram
  participant SC as shellcmd Function
  participant GA as get_appdirs()
  participant FS as Filesystem
  participant EX as Command Executor

  SC->>GA: Query user cache directory
  GA-->>SC: Return cache directory path
  SC->>FS: Check existence of the cache directory
  alt Directory exists
    FS-->>SC: Directory found
    SC->>SC: Append bind mount argument for source cache
  else Directory not found
    SC->>SC: Log debug message (skip bind mount)
  end
  SC->>EX: Execute constructed shell command
Loading

Possibly related PRs

Suggested reviewers

  • johanneskoester

Tip

⚡🧪 Multi-step agentic review comment chat (experimental)
  • We're introducing multi-step agentic chat in review comments. This experimental feature enhances review discussions with the CodeRabbit agentic chat by enabling advanced interactions, including the ability to create pull requests directly from comments.
    - To enable this feature, set early_access to true under in the settings.

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between cafc3da and 2b5260a.

📒 Files selected for processing (2)
  • src/snakemake/workflow.py (0 hunks)
  • tests/tests.py (1 hunks)
💤 Files with no reviewable changes (1)
  • src/snakemake/workflow.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/tests.py
⏰ Context from checks skipped due to timeout of 90000ms (43)
  • GitHub Check: tests (10, windows-latest, py312, bash)
  • GitHub Check: tests (10, windows-latest, py311, bash)
  • GitHub Check: tests (10, ubuntu-latest, py312, bash)
  • GitHub Check: tests (10, ubuntu-latest, py311, bash)
  • GitHub Check: tests (9, windows-latest, py312, bash)
  • GitHub Check: tests (9, windows-latest, py311, bash)
  • GitHub Check: tests (9, ubuntu-latest, py312, bash)
  • GitHub Check: tests (9, ubuntu-latest, py311, bash)
  • GitHub Check: tests (8, windows-latest, py312, bash)
  • GitHub Check: tests (8, windows-latest, py311, bash)
  • GitHub Check: tests (8, ubuntu-latest, py312, bash)
  • GitHub Check: tests (8, ubuntu-latest, py311, bash)
  • GitHub Check: tests (7, windows-latest, py312, bash)
  • GitHub Check: tests (7, windows-latest, py311, bash)
  • GitHub Check: tests (7, ubuntu-latest, py312, bash)
  • GitHub Check: tests (7, ubuntu-latest, py311, bash)
  • GitHub Check: tests (6, windows-latest, py312, bash)
  • GitHub Check: tests (6, windows-latest, py311, bash)
  • GitHub Check: tests (6, ubuntu-latest, py312, bash)
  • GitHub Check: tests (6, ubuntu-latest, py311, bash)
  • GitHub Check: tests (5, windows-latest, py312, bash)
  • GitHub Check: tests (5, windows-latest, py311, bash)
  • GitHub Check: tests (5, ubuntu-latest, py312, bash)
  • GitHub Check: tests (5, ubuntu-latest, py311, bash)
  • GitHub Check: tests (4, windows-latest, py312, bash)
  • GitHub Check: tests (4, windows-latest, py311, bash)
  • GitHub Check: tests (4, ubuntu-latest, py312, bash)
  • GitHub Check: tests (4, ubuntu-latest, py311, bash)
  • GitHub Check: tests (3, windows-latest, py312, bash)
  • GitHub Check: tests (3, windows-latest, py311, bash)
  • GitHub Check: tests (3, ubuntu-latest, py312, bash)
  • GitHub Check: tests (3, ubuntu-latest, py311, bash)
  • GitHub Check: tests (2, windows-latest, py312, bash)
  • GitHub Check: tests (2, windows-latest, py311, bash)
  • GitHub Check: tests (2, ubuntu-latest, py312, bash)
  • GitHub Check: tests (2, ubuntu-latest, py311, bash)
  • GitHub Check: tests (1, windows-latest, py312, bash)
  • GitHub Check: tests (1, windows-latest, py311, bash)
  • GitHub Check: tests (1, ubuntu-latest, py312, bash)
  • GitHub Check: tests (1, ubuntu-latest, py311, bash)
  • GitHub Check: apidocs
  • GitHub Check: docs
  • GitHub Check: formatting

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
snakemake/deployment/singularity.py (1)

130-134: Fix typos in comments and ensure proper error handling.

The implementation correctly mounts the source cache directory by default, which aligns with the PR objective. However, there are typos in the comments, and it might be beneficial to add error handling for cases where the cache directory doesn't exist.

-# mount the snakemake cache into the container per detault so that
-# params included with workflow.source_path are allways mounted in the conteiner
+# mount the snakemake cache into the container by default so that
+# params included with workflow.source_path are always mounted in the container
 if len(args) == 0:
-    args += "--bind " + os.path.join(get_appdirs().user_cache_dir, "snakemake/source-cache")
+    cache_path = os.path.join(get_appdirs().user_cache_dir, "snakemake/source-cache")
+    if os.path.exists(cache_path):
+        args += "--bind " + cache_path

Additionally, consider adding a note in the PR description or documentation about the security implications of mounting the user's cache directory, especially when used with the --no-home flag as mentioned in the PR objectives.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 051dc53 and 7232d3e.

📒 Files selected for processing (7)
  • snakemake/deployment/singularity.py (2 hunks)
  • snakemake/workflow.py (1 hunks)
  • tests/test_issue3361/Snakefile (1 hunks)
  • tests/test_issue3361/expected-results/output.txt (1 hunks)
  • tests/test_issue3361/input.txt (1 hunks)
  • tests/test_issue3361/some_dir/some_other_input.txt (1 hunks)
  • tests/tests.py (1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • snakemake/workflow.py
  • snakemake/deployment/singularity.py
  • tests/tests.py
🔇 Additional comments (6)
tests/test_issue3361/input.txt (1)

1-2: LGTM! Appropriate test input file.

Simple test input file with the expected content for the container access test.

tests/test_issue3361/some_dir/some_other_input.txt (1)

1-2: LGTM! Appropriate test file in subdirectory.

Well-positioned test file that will be accessed via workflow.source_path from the container, which directly tests the functionality being implemented in this PR.

tests/test_issue3361/expected-results/output.txt (1)

1-3: LGTM! Properly formatted expected test output.

The expected output file correctly contains the concatenated content of both input files, which will be used to validate the test results.

tests/test_issue3361/Snakefile (1)

1-16: LGTM! Well-designed test workflow for container source path access.

This Snakefile effectively tests the PR's main functionality by:

  1. Creating a simple workflow with container execution
  2. Using workflow.source_path() to reference a file in a subdirectory
  3. Verifying the concatenated output matches the expected result

The test directly validates that files specified with workflow.source_path in params are accessible within the container environment.

tests/tests.py (1)

2051-2054: LGTM: Good test addition for the new feature.

The test properly validates that files specified with workflow.source_path are accessible in containers by running Snakemake with the Apptainer deployment method.

snakemake/deployment/singularity.py (1)

21-21: LGTM: Appropriate import for accessing user cache directory.

This import is necessary for accessing the user's cache directory where Snakemake stores its source cache.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
snakemake/deployment/singularity.py (2)

130-134: Fix spelling errors in the comments.

There are spelling errors in the comments that should be corrected.

-    # mount the snakemake cache into the container per detault so that
-    # params included with workflow.source_path are allways mounted in the conteiner
+    # mount the snakemake cache into the container per default so that
+    # params included with workflow.source_path are always mounted in the container
     if len(args) == 0:
         args += "--bind " + os.path.join(get_appdirs().user_cache_dir, "snakemake/source-cache")
🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 130-130:
detault ==> default


[failure] 131-131:
allways ==> always


[failure] 131-131:
conteiner ==> container

🪛 GitHub Actions: Codespell

[error] 130-130: codespell: 'detault' is a misspelling; suggested correction is 'default'.


130-134: Good implementation for automatically mounting source cache files.

The implementation correctly handles mounting the Snakemake source cache directory when no additional arguments are provided, ensuring that files referenced with workflow.source_path are available in containers. This addresses the stated PR objective.

Consider adding a more detailed comment or log message to make users aware of this behavior, especially considering the security implications of mounting directories from the user's home.

🧰 Tools
🪛 GitHub Check: Check for spelling errors

[failure] 130-130:
detault ==> default


[failure] 131-131:
allways ==> always


[failure] 131-131:
conteiner ==> container

🪛 GitHub Actions: Codespell

[error] 130-130: codespell: 'detault' is a misspelling; suggested correction is 'default'.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7232d3e and 16bce78.

📒 Files selected for processing (5)
  • snakemake/deployment/singularity.py (2 hunks)
  • snakemake/workflow.py (1 hunks)
  • docs/snakefiles/deployment.rst (1 hunks)
  • snakemake/deployment/singularity.py (1 hunks)
  • snakemake/workflow.py (0 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
  • snakemake/workflow.py
  • snakemake/workflow.py
  • snakemake/deployment/singularity.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • snakemake/deployment/singularity.py
🪛 GitHub Check: Check for spelling errors
snakemake/deployment/singularity.py

[failure] 130-130:
detault ==> default


[failure] 131-131:
allways ==> always


[failure] 131-131:
conteiner ==> container

🪛 GitHub Actions: Codespell
snakemake/deployment/singularity.py

[error] 130-130: codespell: 'detault' is a misspelling; suggested correction is 'default'.

🪛 GitHub Actions: CI
snakemake/deployment/singularity.py

[error] 19-19: Black formatting check failed. 2 files would be reformatted, 141 files would be left unchanged.

🔇 Additional comments (2)
snakemake/deployment/singularity.py (1)

21-21: Import added for supporting the cache mounting feature.

The import for get_appdirs is correctly added to access the user cache directory needed to implement the new functionality.

docs/snakefiles/deployment.rst (1)

455-457: Documentation clearly explains the new functionality and security considerations.

The added documentation clearly explains:

  1. That files mounted using params with workflow.source_path are automatically available in containers
  2. How this is implemented (by mounting the Snakemake cache)
  3. Important security considerations about container access to the home directory

This documentation aligns well with the code changes and provides users with the necessary information about both the functionality and security implications.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 16bce78 and f68d913.

📒 Files selected for processing (5)
  • snakemake/deployment/singularity.py (2 hunks)
  • tests/tests.py (1 hunks)
  • snakemake/deployment/singularity.py (1 hunks)
  • snakemake/deployment/singularity.py (2 hunks)
  • tests/tests.py (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (4)
  • snakemake/deployment/singularity.py
  • tests/tests.py
  • snakemake/deployment/singularity.py
  • tests/tests.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • snakemake/deployment/singularity.py
🪛 GitHub Check: Check for spelling errors
snakemake/deployment/singularity.py

[failure] 130-130:
detault ==> default


[failure] 131-131:
allways ==> always


[failure] 131-131:
conteiner ==> container

🪛 GitHub Actions: Codespell
snakemake/deployment/singularity.py

[error] 130-130: codespell: 'detault' is a misspelling of 'default'.

⏰ Context from checks skipped due to timeout of 90000ms (30)
  • GitHub Check: testing (10, 3.12, bash)
  • GitHub Check: testing (10, 3.11, bash)
  • GitHub Check: testing (9, 3.12, bash)
  • GitHub Check: testing (9, 3.11, bash)
  • GitHub Check: testing (8, 3.12, bash)
  • GitHub Check: testing (8, 3.11, bash)
  • GitHub Check: testing (7, 3.12, bash)
  • GitHub Check: testing (7, 3.11, bash)
  • GitHub Check: testing (6, 3.12, bash)
  • GitHub Check: testing (6, 3.11, bash)
  • GitHub Check: testing (5, 3.12, bash)
  • GitHub Check: testing (5, 3.11, bash)
  • GitHub Check: testing (4, 3.12, bash)
  • GitHub Check: testing (4, 3.11, bash)
  • GitHub Check: testing-windows (10)
  • GitHub Check: testing (3, 3.12, bash)
  • GitHub Check: testing-windows (9)
  • GitHub Check: testing-windows (8)
  • GitHub Check: testing (3, 3.11, bash)
  • GitHub Check: testing-windows (7)
  • GitHub Check: testing (2, 3.12, bash)
  • GitHub Check: testing-windows (6)
  • GitHub Check: testing (2, 3.11, bash)
  • GitHub Check: testing-windows (5)
  • GitHub Check: testing (1, 3.12, bash)
  • GitHub Check: testing-windows (4)
  • GitHub Check: testing (1, 3.11, bash)
  • GitHub Check: testing-windows (3)
  • GitHub Check: testing-windows (2)
  • GitHub Check: testing-windows (1)
🔇 Additional comments (1)
snakemake/deployment/singularity.py (1)

21-21: Good addition of necessary import.

Adding the import for get_appdirs from snakemake.common is necessary for the new functionality that mounts the source cache.

get_appdirs().user_cache_dir, "snakemake/source-cache"
)

cmd = "{} singularity {} exec --home {} {} {} {} -c '{}'".format(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but here it binds --home doesn't it? How can it be that this does not already include the cache dir?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback!

We just had a deeper look and it seems that the cwd is bound as "home" and not the home of the user. So we think binding the source-cache is still needed to make the sourced files accessible in the container. "--no-home" flag is then not needed since "home" is mounted as the current cwd which makes sense.

We also further tested the capability of singularity to write and see files in the higher level folders and this fails as expected. We think this makes the use of containers more save then expected and we also implemented a test for that.

We will provide the update in the next commit.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
snakemake/deployment/singularity.py (1)

130-136: Consider adding a user warning about container security implications

Given the PR objectives mention security concerns about giving containers access to the user's cache directory, consider adding a warning log message when this mount is added automatically, especially for users who might not be aware of this behavior.

if len(args) == 0:
+    logger.warning(
+        "Mounting Snakemake source cache into container. "
+        "Only use trusted containers as they will have access to cached workflow files."
+    )
    args += "--bind " + os.path.join(
        get_appdirs().user_cache_dir, "snakemake/source-cache"
    )
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 624ea71 and c631e8f.

📒 Files selected for processing (3)
  • docs/snakefiles/deployment.rst (1 hunks)
  • snakemake/deployment/singularity.py (2 hunks)
  • tests/tests.py (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • tests/tests.py
  • docs/snakefiles/deployment.rst
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • snakemake/deployment/singularity.py
🔇 Additional comments (1)
snakemake/deployment/singularity.py (1)

20-20: Import addition for accessing user cache directory

The new import of get_appdirs from snakemake.common is necessary for determining the location of the Snakemake source cache directory, which is used in the updated shellcmd function.

@sonarqubecloud
Copy link
Copy Markdown

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 856ce96 and d6bd8fa.

📒 Files selected for processing (3)
  • src/snakemake/deployment/singularity.py (2 hunks)
  • src/snakemake/workflow.py (0 hunks)
  • tests/tests.py (1 hunks)
💤 Files with no reviewable changes (1)
  • src/snakemake/workflow.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/tests.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest ...

**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

  • src/snakemake/deployment/singularity.py
⏰ Context from checks skipped due to timeout of 90000ms (18)
  • GitHub Check: tests (10, windows-latest, py312, bash)
  • GitHub Check: tests (10, windows-latest, py311, bash)
  • GitHub Check: tests (10, ubuntu-latest, py312, bash)
  • GitHub Check: tests (10, ubuntu-latest, py311, bash)
  • GitHub Check: tests (4, ubuntu-latest, py312, bash)
  • GitHub Check: tests (4, ubuntu-latest, py311, bash)
  • GitHub Check: tests (3, windows-latest, py312, bash)
  • GitHub Check: tests (3, windows-latest, py311, bash)
  • GitHub Check: tests (3, ubuntu-latest, py312, bash)
  • GitHub Check: tests (3, ubuntu-latest, py311, bash)
  • GitHub Check: tests (2, windows-latest, py312, bash)
  • GitHub Check: tests (2, windows-latest, py311, bash)
  • GitHub Check: tests (2, ubuntu-latest, py312, bash)
  • GitHub Check: tests (2, ubuntu-latest, py311, bash)
  • GitHub Check: tests (1, windows-latest, py312, bash)
  • GitHub Check: tests (1, windows-latest, py311, bash)
  • GitHub Check: tests (1, ubuntu-latest, py312, bash)
  • GitHub Check: tests (1, ubuntu-latest, py311, bash)
🔇 Additional comments (1)
src/snakemake/deployment/singularity.py (1)

20-20: Good addition of required import.

The import of get_appdirs from snakemake.common is appropriately added to support the new feature for mounting the Snakemake cache into containers.

Copy link
Copy Markdown
Contributor

@johanneskoester johanneskoester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!

@johanneskoester johanneskoester merged commit a6e45bf into snakemake:main Mar 14, 2025
49 checks passed
@SrPeter128 SrPeter128 deleted the mount_cache_to_container branch March 14, 2025 09:50
btraven00 pushed a commit to btraven00/snakemake-software-deployment-plugin-container that referenced this pull request Mar 14, 2025
As done recently in the singularity implementation, we need to mount the
cache folder in the path where XDG expects to find the cache.

The template will need to change when adding support for apptainer, but
we will converge on that later.

- Related: snakemake/snakemake#3385
johanneskoester pushed a commit that referenced this pull request Mar 14, 2025
🤖 I have created a release *beep* *boop*
---


##
[9.0.0](v8.30.0...v9.0.0)
(2025-03-14)


### ⚠ BREAKING CHANGES

* Logging refactor & add LoggerPluginInterface
([#3107](#3107))

### Features

* [#3412](#3412) - keep
shadow folder of failed job if --keep-incomplete flag is set.
([#3430](#3430))
([22978c3](22978c3))
* add flag --report-after-run to automatically generate the report after
a successfull workflow run
([#3428](#3428))
([b0a7f03](b0a7f03))
* add flatten function to IO utils
([#3424](#3424))
([67fa392](67fa392))
* add helper functions to parse input files
([#2918](#2918))
([63e45a7](63e45a7))
* Add option to print redacted file names
([#3089](#3089))
([ba4d264](ba4d264))
* add support for validation of polars dataframe and lazyframe
([#3262](#3262))
([c7473a6](c7473a6))
* added support for rendering dag with mermaid js
([#3409](#3409))
([7bf8381](7bf8381))
* adding --replace-workflow-config to fully replace workflow configs
(from config: directive) with --configfile, instead of merging them
([#3381](#3381))
([47504a0](47504a0))
* Dynamic module name
([#3401](#3401))
([024dc32](024dc32))
* Enable saving and reloading IOCache object
([#3386](#3386))
([c935953](c935953))
* files added in rule params with workflow.source_path will be available
in used containers
([#3385](#3385))
([a6e45bf](a6e45bf))
* Fix keep_local in storage directive and more freedom over remote
retrieval behaviour
([#3410](#3410))
([67b4739](67b4739))
* inherit parameters of use rule and extend/replace individual items
them when using 'with' directive
([#3365](#3365))
([93e4b92](93e4b92))
* Logging refactor & add LoggerPluginInterface
([#3107](#3107))
([86f1d6e](86f1d6e))
* Maximal file size for checksums
([#3368](#3368))
([b039f8a](b039f8a))
* Modernize package configuration using Pixi
([#3369](#3369))
([77992d8](77992d8))
* multiext support for named input/output
([#3372](#3372))
([05e1378](05e1378))
* optionally auto-group jobs via temp files in case of remote execution
([#3378](#3378))
([cc9bba2](cc9bba2))


### Bug Fixes

* `--delete-all-output` ignores `--dry-run`
([#3265](#3265))
([23fef82](23fef82))
* 3342 faster touch runs and warning messages for non-existing files
([#3398](#3398))
([cd9c3c3](cd9c3c3))
* add default value to max-jobs-per-timespan
([#3043](#3043))
([2959abe](2959abe))
* checkpoints inside modules are overwritten
([#3359](#3359))
([fba3ac7](fba3ac7))
* Convert Path to IOFile
([#3405](#3405))
([c58684c](c58684c))
* Do not perform storage object cleanup with --keep-storage-local-copies
set ([#3358](#3358))
([9a6d14b](9a6d14b))
* edgecases of source deployment in case of remote execution
([#3396](#3396))
([5da13be](5da13be))
* enhance error message formatting for strict DAG-building mode
([#3376](#3376))
([a1c39ee](a1c39ee))
* fix bug in checkpoint handling that led to exceptions in case
checkpoint output was missing upon rerun
([#3423](#3423))
([8cf4a2f](8cf4a2f))
* force check all required outputs
([#3341](#3341))
([495a4e7](495a4e7))
* group job formatting
([#3442](#3442))
([f0b10a3](f0b10a3))
* in remote jobs, upload storage in topological order such that
modification dates are preserved (e.g. in case of group jobs)
([#3377](#3377))
([eace08f](eace08f))
* only skip eval when resource depends on input
([#3374](#3374))
([4574c92](4574c92))
* Prevent execution of conda in apptainer when not explicitly requested
in software deployment method
([#3388](#3388))
([c43c5c0](c43c5c0))
* print filenames with quotes around them in RuleException
([#3269](#3269))
([6baeda5](6baeda5))
* Re-evaluation of free resources
([#3399](#3399))
([6371293](6371293))
* ReadTheDocs layout issue due to src directory change
([#3419](#3419))
([695b127](695b127))
* robustly escaping quotes in generated bash scripts (v2)
([#3297](#3297))
([#3389](#3389))
([58720bd](58720bd))
* Show apptainer image URL in snakemake report
([#3407](#3407))
([45f0450](45f0450))
* Update ReadTheDocs configuration for documentation build to use Pixi
([#3433](#3433))
([3f227a6](3f227a6))


### Documentation

* Add pixi setup instructions to general use tutorial
([#3382](#3382))
([115e81b](115e81b))
* fix contribution section heading levels, fix docs testing setup order
([#3360](#3360))
([051dc53](051dc53))
* fix link to github.com/snakemake/poetry-snakemake-plugin
([#3436](#3436))
([ec6d97c](ec6d97c))
* fix quoting
([#3394](#3394))
([b40f599](b40f599))
* fix rerun-triggers default
([#3403](#3403))
([4430e23](4430e23))
* fix typo 'safe' -&gt; 'save'
([#3384](#3384))
([7755861](7755861))
* mention code formatting in the contribution section
([#3431](#3431))
([e8682b7](e8682b7))
* remove duplicated 'functions'.
([#3356](#3356))
([7c595db](7c595db))
* update broken links documentation
([#3437](#3437))
([e3d0d88](e3d0d88))
* Updating contributing guidelines with new pixi dev setup
([#3415](#3415))
([8e95a12](8e95a12))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: snakemake-bot <snakemake-bot-admin@googlegroups.com>
@coderabbitai coderabbitai bot mentioned this pull request Sep 10, 2025
2 tasks
kjohnsen pushed a commit to kjohnsen/snakemake that referenced this pull request Dec 15, 2025
…ailable in used containers (snakemake#3385)

If files are specified under `params` using workflow.source_path, they
are also automatically available in the container. This was achieved by
mounting the nextflow source cache in the container by default. This
achieves a behavior which is rather natural, as the user probably
expects that files which are specified in params using
workflow.source_path are also available in the container.

Due to the fact that the source cache is normally located under
/home/<user>/.cache/snakemake/snakemake/source-cache and the container
thus gets access to the home directory despite the use of the --no-home
flag used in this PR, the documentation explicitly points out that only
trustworthy containers should be used.


### QC

* [ x] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [ x] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

This release improves container workflow execution and documentation,
ensuring a smoother and more secure experience for users.

- **New Features**
- Enhanced container integration now automatically binds local cache
directories to improve workflow performance.
- Added new test scenarios to validate Snakemake execution with
Apptainer deployment.
- **Documentation**
- Updated guidance on file accessibility within container environments,
with added security recommendations to use trusted container images.
- **Tests**
- Introduced additional test functions to cover various Snakemake
execution scenarios.
  - New input and expected output files added for comprehensive testing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
kjohnsen pushed a commit to kjohnsen/snakemake that referenced this pull request Dec 15, 2025
🤖 I have created a release *beep* *boop*
---


##
[9.0.0](snakemake/snakemake@v8.30.0...v9.0.0)
(2025-03-14)


### ⚠ BREAKING CHANGES

* Logging refactor & add LoggerPluginInterface
([snakemake#3107](snakemake#3107))

### Features

* [snakemake#3412](snakemake#3412) - keep
shadow folder of failed job if --keep-incomplete flag is set.
([snakemake#3430](snakemake#3430))
([22978c3](snakemake@22978c3))
* add flag --report-after-run to automatically generate the report after
a successfull workflow run
([snakemake#3428](snakemake#3428))
([b0a7f03](snakemake@b0a7f03))
* add flatten function to IO utils
([snakemake#3424](snakemake#3424))
([67fa392](snakemake@67fa392))
* add helper functions to parse input files
([snakemake#2918](snakemake#2918))
([63e45a7](snakemake@63e45a7))
* Add option to print redacted file names
([snakemake#3089](snakemake#3089))
([ba4d264](snakemake@ba4d264))
* add support for validation of polars dataframe and lazyframe
([snakemake#3262](snakemake#3262))
([c7473a6](snakemake@c7473a6))
* added support for rendering dag with mermaid js
([snakemake#3409](snakemake#3409))
([7bf8381](snakemake@7bf8381))
* adding --replace-workflow-config to fully replace workflow configs
(from config: directive) with --configfile, instead of merging them
([snakemake#3381](snakemake#3381))
([47504a0](snakemake@47504a0))
* Dynamic module name
([snakemake#3401](snakemake#3401))
([024dc32](snakemake@024dc32))
* Enable saving and reloading IOCache object
([snakemake#3386](snakemake#3386))
([c935953](snakemake@c935953))
* files added in rule params with workflow.source_path will be available
in used containers
([snakemake#3385](snakemake#3385))
([a6e45bf](snakemake@a6e45bf))
* Fix keep_local in storage directive and more freedom over remote
retrieval behaviour
([snakemake#3410](snakemake#3410))
([67b4739](snakemake@67b4739))
* inherit parameters of use rule and extend/replace individual items
them when using 'with' directive
([snakemake#3365](snakemake#3365))
([93e4b92](snakemake@93e4b92))
* Logging refactor & add LoggerPluginInterface
([snakemake#3107](snakemake#3107))
([86f1d6e](snakemake@86f1d6e))
* Maximal file size for checksums
([snakemake#3368](snakemake#3368))
([b039f8a](snakemake@b039f8a))
* Modernize package configuration using Pixi
([snakemake#3369](snakemake#3369))
([77992d8](snakemake@77992d8))
* multiext support for named input/output
([snakemake#3372](snakemake#3372))
([05e1378](snakemake@05e1378))
* optionally auto-group jobs via temp files in case of remote execution
([snakemake#3378](snakemake#3378))
([cc9bba2](snakemake@cc9bba2))


### Bug Fixes

* `--delete-all-output` ignores `--dry-run`
([snakemake#3265](snakemake#3265))
([23fef82](snakemake@23fef82))
* 3342 faster touch runs and warning messages for non-existing files
([snakemake#3398](snakemake#3398))
([cd9c3c3](snakemake@cd9c3c3))
* add default value to max-jobs-per-timespan
([snakemake#3043](snakemake#3043))
([2959abe](snakemake@2959abe))
* checkpoints inside modules are overwritten
([snakemake#3359](snakemake#3359))
([fba3ac7](snakemake@fba3ac7))
* Convert Path to IOFile
([snakemake#3405](snakemake#3405))
([c58684c](snakemake@c58684c))
* Do not perform storage object cleanup with --keep-storage-local-copies
set ([snakemake#3358](snakemake#3358))
([9a6d14b](snakemake@9a6d14b))
* edgecases of source deployment in case of remote execution
([snakemake#3396](snakemake#3396))
([5da13be](snakemake@5da13be))
* enhance error message formatting for strict DAG-building mode
([snakemake#3376](snakemake#3376))
([a1c39ee](snakemake@a1c39ee))
* fix bug in checkpoint handling that led to exceptions in case
checkpoint output was missing upon rerun
([snakemake#3423](snakemake#3423))
([8cf4a2f](snakemake@8cf4a2f))
* force check all required outputs
([snakemake#3341](snakemake#3341))
([495a4e7](snakemake@495a4e7))
* group job formatting
([snakemake#3442](snakemake#3442))
([f0b10a3](snakemake@f0b10a3))
* in remote jobs, upload storage in topological order such that
modification dates are preserved (e.g. in case of group jobs)
([snakemake#3377](snakemake#3377))
([eace08f](snakemake@eace08f))
* only skip eval when resource depends on input
([snakemake#3374](snakemake#3374))
([4574c92](snakemake@4574c92))
* Prevent execution of conda in apptainer when not explicitly requested
in software deployment method
([snakemake#3388](snakemake#3388))
([c43c5c0](snakemake@c43c5c0))
* print filenames with quotes around them in RuleException
([snakemake#3269](snakemake#3269))
([6baeda5](snakemake@6baeda5))
* Re-evaluation of free resources
([snakemake#3399](snakemake#3399))
([6371293](snakemake@6371293))
* ReadTheDocs layout issue due to src directory change
([snakemake#3419](snakemake#3419))
([695b127](snakemake@695b127))
* robustly escaping quotes in generated bash scripts (v2)
([snakemake#3297](snakemake#3297))
([snakemake#3389](snakemake#3389))
([58720bd](snakemake@58720bd))
* Show apptainer image URL in snakemake report
([snakemake#3407](snakemake#3407))
([45f0450](snakemake@45f0450))
* Update ReadTheDocs configuration for documentation build to use Pixi
([snakemake#3433](snakemake#3433))
([3f227a6](snakemake@3f227a6))


### Documentation

* Add pixi setup instructions to general use tutorial
([snakemake#3382](snakemake#3382))
([115e81b](snakemake@115e81b))
* fix contribution section heading levels, fix docs testing setup order
([snakemake#3360](snakemake#3360))
([051dc53](snakemake@051dc53))
* fix link to github.com/snakemake/poetry-snakemake-plugin
([snakemake#3436](snakemake#3436))
([ec6d97c](snakemake@ec6d97c))
* fix quoting
([snakemake#3394](snakemake#3394))
([b40f599](snakemake@b40f599))
* fix rerun-triggers default
([snakemake#3403](snakemake#3403))
([4430e23](snakemake@4430e23))
* fix typo 'safe' -&gt; 'save'
([snakemake#3384](snakemake#3384))
([7755861](snakemake@7755861))
* mention code formatting in the contribution section
([snakemake#3431](snakemake#3431))
([e8682b7](snakemake@e8682b7))
* remove duplicated 'functions'.
([snakemake#3356](snakemake#3356))
([7c595db](snakemake@7c595db))
* update broken links documentation
([snakemake#3437](snakemake#3437))
([e3d0d88](snakemake@e3d0d88))
* Updating contributing guidelines with new pixi dev setup
([snakemake#3415](snakemake#3415))
([8e95a12](snakemake@8e95a12))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: snakemake-bot <snakemake-bot-admin@googlegroups.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Files cached with workflow.source_path are not available in apptainer

2 participants