Skip to content

feat: introduce --workflow-profile for additional workflow specific profiles that overwrite global profiles; add ability to define key-value CLI flags like --set-threads or --set-resources as multi-level dictionaries in profile config yaml files#2310

Merged
johanneskoester merged 15 commits intomainfrom
feat/profile-overlays
Jun 21, 2023
Merged

Conversation

@johanneskoester
Copy link
Copy Markdown
Contributor

@johanneskoester johanneskoester commented Jun 20, 2023

Description

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

johanneskoester and others added 5 commits June 20, 2023 12:48
@johanneskoester
feat: introduce --workflow-profile and --no-workflow-profile for addi…
fac0e38 
…tional workflow specific profiles that overwrite global profiles; add ability to define key-value CLI flags like --set-threads or --set-resources as multi-level dictionaries in profile config yaml files.
@johanneskoester
fmt
c32a5c9
@johanneskoester
Merge branch 'main' into feat/profile-overlays
2bf8c48
@johanneskoester
docs
24dcf81
@johanneskoester
Merge branch 'feat/profile-overlays' of github.com:snakemake/snakemak…
7576e23 
…e into feat/profile-overlays
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jun 20, 2023

Please format your code with black: black snakemake tests/*.py.

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Jun 20, 2023

Please retry analysis of this Pull-Request directly on SonarCloud.

@cmeesters
Copy link
Copy Markdown
Member

cmeesters commented Jun 20, 2023 •
edited
Loading

Hoi @johanneskoester

You are WAY too fast for my taste ;-)

Anyway, I like this very much indeed. I will test it by the end of the week (just that some more meetings are in the way to do this any sooner). Allow me a minor question: Why the original --no-workflow-profile flag and currently the explicitly required None to the --workflow-profile flag? If there is no such flag, it could just be ignored. Or am I missing something here?

@johanneskoester johanneskoester changed the title feat: introduce --workflow-profile and --no-workflow-profile for additional workflow specific profiles that overwrite global profiles; add ability to define key-value CLI flags like --set-threads or --set-resources as multi-level dictionaries in profile config yaml files feat: introduce --workflow-profile for additional workflow specific profiles that overwrite global profiles; add ability to define key-value CLI flags like --set-threads or --set-resources as multi-level dictionaries in profile config yaml files Jun 21, 2023
@johanneskoester
Copy link
Copy Markdown
Contributor Author

johanneskoester commented Jun 21, 2023 •
edited
Loading

Hoi @johanneskoester

You are WAY too fast for my taste ;-)

Anyway, I like this very much indeed. I will test it by the end of the week (just that some more meetings are in the way to do this any sooner). Allow me a minor question: Why the original --no-workflow-profile flag and currently the explicitly required None to the --workflow-profile flag? If there is no such flag, it could just be ignored. Or am I missing something here?

This is because of the behavior that if --workflow-profile is not specified the profile stored at workflow/profiles/default (if Snakefile is at workflow/Snakefile is automatically used. This default behavior is important to allow workflow designers to ensure certain flags are set. E.g. some default constraints on resources like for example:

resources:
  biomart_api_calls: 1

To ensure that some rule that queries biomart is not executed many times in parallel (which would lead to being blocked).
If there is no such profile next to the Snakefile, it will of course not be used and --workflow-profile none is equivalent to not specifying it at all.

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Jun 21, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 1 Code Smell

No Coverage information No Coverage information
0.0% 0.0% Duplication

@johanneskoester johanneskoester merged commit 9675c17 into main Jun 21, 2023
@johanneskoester johanneskoester deleted the feat/profile-overlays branch June 21, 2023 17:34
@github-actions github-actions bot mentioned this pull request Jun 21, 2023
Merged
johanneskoester pushed a commit that referenced this pull request Jun 22, 2023
@github-actions
chore(main): release 7.29.0 (#2303)
f1b8342 
🤖 I have created a release *beep* *boop*
---


##
[7.29.0](v7.28.3...v7.29.0)
(2023-06-21)


### Features

* introduce --workflow-profile for additional workflow specific profiles
that overwrite global profiles; add ability to define key-value CLI
flags like --set-threads or --set-resources as multi-level dictionaries
in profile config yaml files
([#2310](#2310))
([9675c17](9675c17))


### Bug Fixes

* addressing [#2197](#2197)
by allowing 256 character account names in slurm
([#2198](#2198))
([ab58c65](ab58c65))
* removed distutils from snakemake
([#2312](#2312))
([9b8c362](9b8c362))
* Update __init__.py to move "file" param to "print"
([#2291](#2291))
([92352b6](92352b6))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@johanneskoester @cmeesters