Skip to content

test: Add pytest markers & skips, update docs to reflect changes#3417

Merged
johanneskoester merged 12 commits intosnakemake:mainfrom
jjjermiah:jjjermiah/improve-docs-and-testing
Mar 14, 2025
Merged

test: Add pytest markers & skips, update docs to reflect changes#3417
johanneskoester merged 12 commits intosnakemake:mainfrom
jjjermiah:jjjermiah/improve-docs-and-testing

Conversation

@jjjermiah
Copy link
Copy Markdown
Contributor

@jjjermiah jjjermiah commented Mar 13, 2025

  • Some tests that that are related to s3 storage keep failing locally. I believe these require the S3 credentials (available as secretes in CI) so I've marked them with pytest as needs_s3 and added a section to the docs regarding this.

    • I could instead have a marker skip automatically but unsure how this works at the moment and dont want to have it start skipping tests in CI/CD.
  • I've also updated the .test_durations files since it hasnt been updated in some time, and there are more tests.

Side note: the docs havent updated due to the src layout not registering in ReadtheDocs. fixed: #3419

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Summary by CodeRabbit

Summary by CodeRabbit

  • Documentation

    • Introduced a new guide on using test markers, providing clear instructions for running or excluding tests based on environment needs.
  • Tests

    • Revised and reorganized test configurations with adjusted timings and added numerous new test cases.
    • Enhanced test annotations to indicate dependencies on external resources and platform-specific conditions for improved clarity during execution.
    • Added support for macOS in the testing workflow, expanding the environment options.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 13, 2025

📝 Walkthrough

Walkthrough

The changes update the JSON-like test durations record by removing, adding, and modifying several test entries. Notable modifications include removing test_singularity, adjusting durations for tests such as test_issue1331 and test_pipes2, and introducing many new tests. The documentation has been enhanced by adding a "Marked tests" section that explains how to use pytest markers (e.g., needs_envmodules, needs_s3) to manage test execution. Additionally, test functions in tests/tests.py now include the @pytest.mark.needs_s3 decorator, and extraneous comments have been removed.

Changes

File Change Summary
.test_durations Removed the test_singularity entry; updated durations for tests such as test_issue1331, test_pipes2, and test_github_issue1261; added numerous new test entries including test_queue_input, test_queue_input_forceall, test_queue_input_dryrun, and many more with their respective duration values.
docs/project_info/contributing.rst Added a new "Marked tests" section that explains the use of pytest markers (e.g., needs_envmodules, needs_s3) to exclude tests. This section includes example commands for filtering markers and listing available markers.
tests/tests.py Added the @pytest.mark.needs_s3 decorator to several test functions (test_default_storage, test_default_storage_local_job, test_storage, test_output_file_cache_storage); added @pytest.mark.skipif decorators to multiple tests to specify conditions under which they should be skipped on macOS. Removed outdated comment lines from functions like test_module_complex, test_module_complex2, and test_module_with_script.
.github/workflows/main.yml Updated the matrix configuration to include macos-latest for running tests; added a new step for executing tests on macOS.
tests/conftest.py Added a new constant ON_MACOS to check for the macOS platform, allowing for platform-specific test configurations.

Suggested reviewers

  • johanneskoester
  • cademirch

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between da34327 and 763b505.

📒 Files selected for processing (2)
  • .github/workflows/main.yml (2 hunks)
  • tests/tests.py (11 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • .github/workflows/main.yml
  • tests/tests.py
⏰ Context from checks skipped due to timeout of 90000ms (45)
  • GitHub Check: tests (10, macos-latest, py312)
  • GitHub Check: tests (10, macos-latest, py311)
  • GitHub Check: tests (10, windows-latest, py312)
  • GitHub Check: tests (10, windows-latest, py311)
  • GitHub Check: tests (10, ubuntu-latest, py312)
  • GitHub Check: tests (10, ubuntu-latest, py311)
  • GitHub Check: tests (9, windows-latest, py312)
  • GitHub Check: tests (9, windows-latest, py311)
  • GitHub Check: tests (9, ubuntu-latest, py312)
  • GitHub Check: tests (9, ubuntu-latest, py311)
  • GitHub Check: tests (8, windows-latest, py312)
  • GitHub Check: tests (8, windows-latest, py311)
  • GitHub Check: tests (8, ubuntu-latest, py312)
  • GitHub Check: tests (8, ubuntu-latest, py311)
  • GitHub Check: tests (7, windows-latest, py312)
  • GitHub Check: tests (7, windows-latest, py311)
  • GitHub Check: tests (7, ubuntu-latest, py312)
  • GitHub Check: tests (7, ubuntu-latest, py311)
  • GitHub Check: tests (6, windows-latest, py312)
  • GitHub Check: tests (6, windows-latest, py311)
  • GitHub Check: tests (6, ubuntu-latest, py312)
  • GitHub Check: tests (6, ubuntu-latest, py311)
  • GitHub Check: tests (5, windows-latest, py312)
  • GitHub Check: tests (5, windows-latest, py311)
  • GitHub Check: tests (5, ubuntu-latest, py312)
  • GitHub Check: tests (5, ubuntu-latest, py311)
  • GitHub Check: tests (4, windows-latest, py312)
  • GitHub Check: tests (4, windows-latest, py311)
  • GitHub Check: tests (4, ubuntu-latest, py312)
  • GitHub Check: tests (4, ubuntu-latest, py311)
  • GitHub Check: tests (3, windows-latest, py312)
  • GitHub Check: tests (3, windows-latest, py311)
  • GitHub Check: tests (3, ubuntu-latest, py312)
  • GitHub Check: tests (3, ubuntu-latest, py311)
  • GitHub Check: tests (2, macos-latest, py312)
  • GitHub Check: tests (2, windows-latest, py312)
  • GitHub Check: tests (2, windows-latest, py311)
  • GitHub Check: tests (2, ubuntu-latest, py312)
  • GitHub Check: tests (2, ubuntu-latest, py311)
  • GitHub Check: tests (1, macos-latest, py312)
  • GitHub Check: tests (1, macos-latest, py311)
  • GitHub Check: tests (1, windows-latest, py312)
  • GitHub Check: tests (1, windows-latest, py311)
  • GitHub Check: tests (1, ubuntu-latest, py312)
  • GitHub Check: tests (1, ubuntu-latest, py311)

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@jjjermiah jjjermiah changed the title tests: Add pytest markers for S3 requirements and update documentation test: Add pytest markers for S3 requirements and update documentation Mar 13, 2025
@jjjermiah jjjermiah marked this pull request as ready for review March 13, 2025 16:17
@jjjermiah jjjermiah added the documentation Improvements or additions to documentation label Mar 13, 2025
@jjjermiah jjjermiah self-assigned this Mar 13, 2025
@cademirch cademirch self-requested a review March 13, 2025 17:10
Copy link
Copy Markdown
Contributor

@cademirch cademirch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The queue test failure is weird but I think is unrelated to this. These changes are great, will make testing way easier, especially with the pixi support you added earlier. Great work.

@jjjermiah jjjermiah changed the title test: Add pytest markers for S3 requirements and update documentation test: Add pytest markers & skips, update docs to reflect changes Mar 14, 2025
@johanneskoester johanneskoester merged commit 275983e into snakemake:main Mar 14, 2025
68 checks passed
@github-project-automation github-project-automation bot moved this from In review to Done in Snakemake Hackathon March 2025 Mar 14, 2025
kjohnsen pushed a commit to kjohnsen/snakemake that referenced this pull request Dec 15, 2025
…kemake#3417)

<!--Add a description of your PR here-->
- Some tests that that are related to s3 storage keep failing locally. I
believe these require the S3 credentials (available as secretes in CI)
so I've marked them with pytest as `needs_s3` and added a section to the
docs regarding this.
- I could instead have a marker skip automatically but unsure how this
works at the moment and dont want to have it start skipping tests in
CI/CD.

- I've also updated the `.test_durations` files since it hasnt been
updated in some time, and there are more tests.

Side note: the docs havent updated due to the src layout not registering
in ReadtheDocs. fixed: snakemake#3419

### QC
<!-- Make sure that you can tick the boxes below. -->

* [x] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [x] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Summary by CodeRabbit

- **Documentation**
- Introduced a new guide on using test markers, providing clear
instructions for running or excluding tests based on environment needs.

- **Tests**
- Revised and reorganized test configurations with adjusted timings and
added numerous new test cases.
- Enhanced test annotations to indicate dependencies on external
resources and platform-specific conditions for improved clarity during
execution.
- Added support for macOS in the testing workflow, expanding the
environment options.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

Development

Successfully merging this pull request may close these issues.

3 participants