Manifest-driven shed repository definitions#143
Manifest-driven shed repository definitions#143jmchilton merged 5 commits intogalaxyproject:masterfrom
Conversation
```
repositories:
cs-cat1:
include:
- cat1.xml
- macros.xml
- test-data
cs-cat2:
include:
- cat2.xml
- macros.xml
- test-data
```
Adding tests for tar ball and repository creation to verify this.
``exclude`` now works in addition to ``ignore`` in ``.shed.yml`` for consistency with ``include``.
An example might look like: ``` owner: "iuc" remote_repository_url: "https://github.com/galaxyproject/planemo/tree/master/tests/data/repos/multi_repos_flat_flag" homepage_url: "http://planemo.readthedocs.org/en/latest/" categories: - "Text Manipulation" auto_tool_repositories: name_template: "cs-{{ tool_id }}" description_template: "The tool {{ tool_name }} from the cat tool suite." ```
An example of creating a .shed.yml that produces just a single repository with one suite in it might be:
```
owner: devteam
suite:
name: suite_1
description: "A suite of Galaxy tools designed to work with version 1.2 of the SAMtools package."
include_repositories:
- name: data_manager_sam_fasta_index_builder
owner: devteam
- name: bam_to_sam
owner: devteam
- name: sam_to_bam
owner: devteam
- name: samtools_bedcov
owner: devteam
```
In this case we are defining explicit dependent repositories but it can also be used with .shed.yml files that define other repositries. For instance if used with ``auto_tool_repositories`` these will automatically be included in the suite.
```
owner: "iuc"
remote_repository_url: "https://github.com/galaxyproject/planemo/tree/master/tests/data/repos/multi_repos_flat_flag"
homepage_url: "http://planemo.readthedocs.org/en/latest/"
categories:
- "Text Manipulation"
auto_tool_repositories:
name_template: "cs-{{ tool_id }}"
description_template: "The tool {{ tool_name }} from the cat tool suite."
suite:
name: "suite_cat"
description: "A suite of Cat tools."
long_description: "A longer description of all the cat tools."
```
Think these semantics are a little better.
Previously custom include statements must have plain strings - files, directories, or globs relative to the .shed.yml file. This has now been extended to allow more complex source and destination selection.
The following is taken from the added test data and demonstrates pulling in and renaming a single file from outside the .shed.yml directory and copying a whole directory into a new directory.
```
repositories:
cs-cat1:
description: "The tool Cat 1 from the cat tool suite."
include:
- cat1.xml
- macros.xml
- test-data
- source: ../shared_files/CITATION
destination: CITATION.txt
- source: ../shared_files/extra_test_data/**
strip_components: 3 # drop "..", "shared_files", "extra_test_data" from source
destination: test-data
```
There was a problem hiding this comment.
Does this belong in shed util?
There was a problem hiding this comment.
Oh, I meant the shed.py file in planemo where a bunch of TS interactivity
was stuck
man. 27. apr. 2015, 08.32 skrev John Chilton notifications@github.com:
In planemo/shed.py
#143 (comment):
- repository_dependencies.repo_pairs = list(repo_pairs) + list(extra_pairs)
- repo = {
"_files": {REPO_DEPENDENCIES_CONFIG_NAME: str(repository_dependencies)},"include": [],"name": name,"description": description,- }
- if long_description:
repo["long_description"] = long_description- repos[name] = repo
+def find_repository(tsi, owner, name):
What is shed util?
—
Reply to this email directly or view it on GitHub
https://github.com/galaxyproject/planemo/pull/143/files#r29145707.
There was a problem hiding this comment.
That is this file :). Though I would like to break it up (#135).
There was a problem hiding this comment.
I...uh...how could I possibly be this blind. So sorry @jmchilton
|
I would like to do a planemo release tonight - preferably including this pull request. Let me know if this |
|
+1 |
|
Awesome - thanks for the review @erasche. |
Manifest-driven shed repository definitions
Currently there exists a tension between what is best for developers (storing all tools in a single repository - e.g. ncbi_blast_plus or bedtools) and what is best for Galaxy users (storing a single repository per tool and collecting them together with a suite - e.g. samtools or gatk). More discussion here.
This pull request extends the semantics of
.shed.ymlin a attempt to resolve this tension and make the best practice for Galaxy users trivial to manage for developers. Previously each.shed.ymlcould only correspond to a single Tool Shed repository and it would collect all files in a directory (except an optional list of ignored files). This pull request extends theshed_createandshed_uploadcommands to allow.shed.ymlfiles to correspond to any number of actual Tool Shed repositories each with fully customizable file includes and excludes.While there is a great deal customization allowed - two new keys
auto_tool_repositoriesandsuiteprovide shortcuts to quickly and implicity define repositories for for each individual tool in the directory and build a suite for those. Consider the following (admittedly idealized) samtools example:This example assumes the
.shed.ymlfile is placed in a "flat" directory with each samtools tool wrapper and planemo will create and update repositories for each individual tool given the specified templates inauto_tool_repositories. Thesuitekey here will auto-generate a suite repository for all of these tools and will automatically created the correspondingrepository_dependencies.xmlto populate it with (this is generated duringshed_uploadand never needs to exist in your repository).Again this example is admittedly idealized, but if
auto_tool_repositoriesis not specified, arepositorieslist can be specified instead. There are some examples of this in the test data included with this pull request:-This demonstrates complex inclusions files from sub-directories and renaming.
The test data also includes some more advanced usages of the
suitekey as well - specifically using it without auto_tool_repositories as a generic replacement forrepository_dependencies.xmland adding additional dependent repositories in addition to the ones defined by the.shed.ymlfile.Implements #26.