Skip to content

Commit 347c622

Browse files
jmchiltonjmchilton
committed
CWL Documentation Overhaul.
- Rename "Dependencies and Docker" tutorial to "Dependencies and Containers" for Galaxy tools (singularity and such has come a long way). - Update everything for CWL 1.0 and the latest Planemo release. - Add tutorial for "Dependencies and Conda" for CWL that mirrors the Galaxy version of this document. - Add tutorial for "Dependencies and Containers" for CWL that mirrors the Galaxy version of this document. - Add seqtk_complete_cwl project template for above tutorials to mirror the Galaxy versions. - Add conda_testing_cwl, conda_answers_cwl, and conda_testing_cwl project templates for above tutorials that mirror the Galaxy versions. - Re-do README to link to the above tutorials and feature CWL more prominently. - Small updates to Galaxy variants of these documents. - Improvements and expansions to documentation testing scripts.
1 parent 67fe8cd commit 347c622

File tree

64 files changed

+2365
-306
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+2365
-306
lines changed

‎README.rst‎

Lines changed: 76 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
:figwidth: 100%
55
:target: https://github.com/galaxyproject/planemo
66

7-
Command-line utilities to assist in building and publishing Galaxy_ tools.
7+
Command-line utilities to assist in developing Galaxy_ and `Common Workflow Language`_ tools.
88

99
.. image:: https://readthedocs.org/projects/planemo/badge/?version=latest
1010
:target: http://planemo.readthedocs.io/en/latest/?badge=latest
@@ -30,9 +30,6 @@ Command-line utilities to assist in building and publishing Galaxy_ tools.
3030
Quick Start
3131
-----------
3232

33-
This quick start demonstrates using ``planemo`` commands to help
34-
develop Galaxy tools.
35-
3633
-----------------
3734
Obtaining
3835
-----------------
@@ -48,26 +45,26 @@ install with ``pip``. Planemo requires pip 7.0 or newer.
4845
$ pip install planemo
4946

5047
For information on updating Planemo, installing the latest development release,
51-
or installing planemo via bioconda - checkout the `installation
48+
or installing Planemo via Bioconda_ - checkout the `installation
5249
<http://planemo.readthedocs.io/en/latest/installation.html>`__
5350
documentation.
5451

5552
Planemo is also available as a `virtual appliance
5653
<https://planemo.readthedocs.io/en/latest/appliance.html>`_ bundled
57-
with a preconfigured Galaxy server and set up for Galaxy tool development.
58-
You can choose from open virtualization format (OVA_, .ova), Docker_,
59-
or Vagrant_ appliances.
54+
with a preconfigured Galaxy server and set up for Galaxy_ and
55+
`Common Workflow Language`_ tool development.
56+
You can choose from open virtualization format (OVA_, .ova) or Docker_
57+
appliances.
6058

61-
--------------
62-
Basics
63-
--------------
59+
-----------------
60+
Basics - Galaxy
61+
-----------------
6462

6563
This quick start will assume you have a directory with one or more Galaxy
6664
tool XML files. If no such directory is available, one can be quickly created for
67-
demonstrating ``planemo`` as follows ``mkdir mytools; cd mytools; planemo
68-
project_init --template=demo``.
65+
demonstrating ``planemo`` as follows ``project_init --template=demo mytools; cd mytools``.
6966

70-
Planemo can check tools containing XML for common problems and best
67+
Planemo can check tool XML files for common problems and best
7168
practices using the ``lint`` `command <http://planemo.readthedocs.org/en/latest/commands.html#lint-command>`_
7269
(also aliased as ``l``).
7370

@@ -101,16 +98,8 @@ can be used to test a tool or a directory of tools.
10198

10299
$ planemo test --galaxy_root=../galaxy randomlines.xml
103100

104-
If no ``--galaxy_root`` is defined, ``planemo`` will check for a default in
105-
`~/.planemo.yml
106-
<http://planemo.readthedocs.org/en/latest/configuration.html>`_ and finally
107-
search the tool's parent directories for a Galaxy root directory.
108-
Planemo can also download and configure a disposable Galaxy instance for
109-
testing. Pass ``--install_galaxy`` instead of ``--galaxy_root``.
110-
111-
::
112-
113-
$ planemo t --install_galaxy
101+
If no ``--galaxy_root`` is defined, Planemo will download and configure
102+
a disposable Galaxy instance for testing.
114103

115104
Planemo will create a HTML output report in the current directory named
116105
``tool_test_output.html`` (override with ``--test_output``). See an
@@ -132,6 +121,57 @@ with ``--install_galaxy``.
132121
For more information on building Galaxy tools in general please check out
133122
`Building Galaxy Tools Using Planemo`_.
134123

124+
----------------------------------
125+
Basics - Common Workflow Language
126+
----------------------------------
127+
128+
This quick start will assume you have a directory with one or more `Common Workflow
129+
Language`_ YAML files. If no such directory is available, one can be quickly created for
130+
demonstrating ``planemo`` as follows ``planemo project_init --template=seqtk_complete_cwl mytools; cd mytools``.
131+
132+
Planemo can check tools YAML files for common problems and best
133+
practices using the ``lint`` `command <http://planemo.readthedocs.org/en/latest/commands.html#lint-command>`_
134+
(also aliased as ``l``).
135+
136+
::
137+
138+
$ planemo lint
139+
140+
Like many ``planemo`` commands - by default this will search the
141+
current directory and use all tool files it finds. It can be explicitly
142+
passed a path to tool files or a directory of tool files.
143+
144+
::
145+
146+
$ planemo l seqtk_seq.cwl
147+
148+
The ``lint`` command takes in additional options related to
149+
reporting levels, exit code, etc. These options are described
150+
in the `docs <http://planemo.readthedocs.org/en/latest/commands.html#lint-command>`_
151+
or (like with all commands) can be accessed by passing ``--help`` to it.
152+
153+
::
154+
155+
$ planemo l --help
156+
Usage: planemo lint [OPTIONS] TOOL_PATH
157+
158+
Once tools are syntactically correct - it is time to test. The ``test``
159+
`command <http://planemo.readthedocs.org/en/latest/commands.html#test-command>`__
160+
can be used to test a CWL tool, workflow, or a directories thereof.
161+
162+
::
163+
164+
$ planemo test --engine cwltool seqtk_seq.cwl
165+
166+
Planemo will create a HTML output report in the current directory named
167+
``tool_test_output.html``. Check out the file ``seqtk_seq_tests.yml`` for
168+
an example of Planemo test for a CWL tool. A test consists of any number of
169+
jobs (with input descriptions) and corresponding output assertions.
170+
171+
Checkout the `Commmon Workflow User Guide`_ for more information on developing
172+
CWL tools in general and `Building Common Workflow Language Tools`_ for more
173+
information on using Planemo to develop CWL tools.
174+
135175
---------
136176
Tool Shed
137177
---------
@@ -144,29 +184,17 @@ Conda
144184
------
145185

146186
Planemo can help develop tools and Conda packages in unison.
147-
Check out `Dependencies and Conda`_ for more information.
148-
149-
--------
150-
Docker
151-
--------
152-
153-
Planemo can help develop tools deployable via Docker.
154-
Check out `Dependencies and Docker`_ for more information.
155-
156-
--------------------------
157-
Common Workflow Language
158-
--------------------------
159-
160-
Planemo includes experimental support for running a subset of valid
161-
`Common Workflow Language`_ (CWL) tools using either the reference implementation cwltool or
162-
a fork of Galaxy enhanced to run CWL tools.
163-
164-
::
187+
Check out the `Galaxy <http://planemo.readthedocs.io/en/latest/writing_advanced.html#dependencies-and-conda>`__ or `CWL
188+
<http://planemo.readthedocs.io/en/latest/writing_advanced_cwl.html#dependencies-and-conda-cwl>`__ version of the "Dependencies and Conda" tutorial
189+
for more information.
165190

166-
$ planemo project_init --template cwl_draft3_spec
167-
$ planemo serve --cwl cwl_draft3_spec/cat1-tool.cwl
191+
-----------------------
192+
Docker and Containers
193+
-----------------------
168194

169-
Checko out `Building Common Workflow Language Tools`_ for more information.
195+
Planemo can help develop tools that run in "Best Practice" containers for
196+
scientific workflows. Check out the `Galaxy <http://planemo.readthedocs.io/en/latest/writing_advanced.html#dependencies-and-containers>`__ or `CWL
197+
<http://planemo.readthedocs.io/en/latest/writing_advanced_cwl.html#dependencies-and-containers-cwl>`__ version of the "Dependencies and Containers" tutorial for more information.
170198

171199
.. _Galaxy: http://galaxyproject.org/
172200
.. _GitHub: https://github.com/
@@ -178,8 +206,8 @@ Checko out `Building Common Workflow Language Tools`_ for more information.
178206
.. _`tools-iuc`: https://github.com/galaxyproject/tools-iuc
179207
.. _Building Galaxy Tools Using Planemo: http://planemo.readthedocs.io/en/latest/writing_standalone.html
180208
.. _Publishing to the Tool Shed: http://planemo.readthedocs.org/en/latest/publishing.html
181-
.. _Dependencies and Conda: http://planemo.readthedocs.io/en/latest/writing_advanced.html#dependencies-and-conda
182-
.. _Dependencies and Docker: http://planemo.readthedocs.io/en/latest/writing_advanced.html#dependencies-and-docker
183-
.. _Common Workflow Language: http://common-workflow-language.github.io
209+
.. _Common Workflow Language: https://www.commonwl.org/
210+
.. _Commmon Workflow User Guide: http://www.commonwl.org/user_guide/
184211
.. _Building Common Workflow Language Tools: http://planemo.readthedocs.io/en/latest/writing_cwl_standalone.html
185212
.. _OVA: https://en.wikipedia.org/wiki/Open_Virtualization_Format
213+
.. _Bioconda: https://github.com/bioconda/bioconda-recipes

‎docs/_writing_conda_fleeqtk.rst‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
This is the skeleton of a tool wrapping the parody bioinformatics software package fleeqtk_.
2+
fleeqtk is a fork of the project seqtk_ that many Planemo tutorials are built around and the
3+
example tool should hopefully be fairly familiar. fleeqtk version 1.3 can be downloaded
4+
from `here <https://github.com/jmchilton/fleeqtk/archive/v1.3.tar.gz>`__ and built using
5+
``make``. The result of ``make`` includes a single executable ``fleeqtk``.
6+
7+
.. _fleeqtk: https://github.com/jmchilton/fleeqtk
8+
.. _seqtk: https://github.com/lh3/seqtk

‎docs/_writing_conda_init.rst‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
.. note:: Planemo requires a Conda installation to target with its various Conda
2+
related commands. A properly configured Conda installation can be initialized
3+
with the ``conda_init`` command. This should only need to be executed once
4+
per development machine.
5+
6+
::
7+
8+
$ planemo conda_init
9+
galaxy.tools.deps.conda_util INFO: Installing conda, this may take several minutes.
10+
wget -q --recursive -O /var/folders/78/zxz5mz4d0jn53xf0l06j7ppc0000gp/T/conda_installLW5zn1.sh https://repo.continuum.io/miniconda/Miniconda3-4.3.31-MacOSX-x86_64.sh
11+
bash /var/folders/78/zxz5mz4d0jn53xf0l06j7ppc0000gp/T/conda_installLW5zn1.sh -b -p /Users/john/miniconda3
12+
PREFIX=/Users/john/miniconda3
13+
installing: python-3.6.3-h47c878a_7 ...
14+
Python 3.6.3 :: Anaconda, Inc.
15+
installing: ca-certificates-2017.08.26-ha1e5d58_0 ...
16+
installing: conda-env-2.6.0-h36134e3_0 ...
17+
installing: libcxxabi-4.0.1-hebd6815_0 ...
18+
installing: tk-8.6.7-h35a86e2_3 ...
19+
installing: xz-5.2.3-h0278029_2 ...
20+
installing: yaml-0.1.7-hc338f04_2 ...
21+
installing: zlib-1.2.11-hf3cbc9b_2 ...
22+
installing: libcxx-4.0.1-h579ed51_0 ...
23+
installing: openssl-1.0.2n-hdbc3d79_0 ...
24+
installing: libffi-3.2.1-h475c297_4 ...
25+
installing: ncurses-6.0-hd04f020_2 ...
26+
installing: libedit-3.1-hb4e282d_0 ...
27+
installing: readline-7.0-hc1231fa_4 ...
28+
installing: sqlite-3.20.1-h7e4c145_2 ...
29+
installing: asn1crypto-0.23.0-py36h782d450_0 ...
30+
installing: certifi-2017.11.5-py36ha569be9_0 ...
31+
installing: chardet-3.0.4-py36h96c241c_1 ...
32+
installing: idna-2.6-py36h8628d0a_1 ...
33+
installing: pycosat-0.6.3-py36hee92d8f_0 ...
34+
installing: pycparser-2.18-py36h724b2fc_1 ...
35+
installing: pysocks-1.6.7-py36hfa33cec_1 ...
36+
installing: python.app-2-py36h54569d5_7 ...
37+
installing: ruamel_yaml-0.11.14-py36h9d7ade0_2 ...
38+
installing: six-1.11.0-py36h0e22d5e_1 ...
39+
installing: cffi-1.11.2-py36hd3e6348_0 ...
40+
installing: setuptools-36.5.0-py36h2134326_0 ...
41+
installing: cryptography-2.1.4-py36h842514c_0 ...
42+
installing: wheel-0.30.0-py36h5eb2c71_1 ...
43+
installing: pip-9.0.1-py36h1555ced_4 ...
44+
installing: pyopenssl-17.5.0-py36h51e4350_0 ...
45+
installing: urllib3-1.22-py36h68b9469_0 ...
46+
installing: requests-2.18.4-py36h4516966_1 ...
47+
installing: conda-4.3.31-py36_0 ...
48+
installation finished.
49+
/Users/john/miniconda3/bin/conda install -y --override-channels --channel iuc --channel bioconda --channel conda-forge --channel defaults --channel r conda=4.3.33 conda-build=2.1.18
50+
Fetching package metadata ...................
51+
Solving package specifications: .
52+
53+
Package plan for installation in environment /Users/john/miniconda3:
54+
55+
The following NEW packages will be INSTALLED:
56+
57+
beautifulsoup4: 4.6.0-py36_0 conda-forge
58+
conda-build: 2.1.18-py36_0 conda-forge
59+
conda-verify: 2.0.0-py36_0 conda-forge
60+
filelock: 3.0.4-py36_0 conda-forge
61+
jinja2: 2.10-py36_0 conda-forge
62+
markupsafe: 1.0-py36_0 conda-forge
63+
pkginfo: 1.4.2-py36_0 conda-forge
64+
pycrypto: 2.6.1-py36_1 conda-forge
65+
pyyaml: 3.12-py36_1 conda-forge
66+
67+
The following packages will be UPDATED:
68+
69+
conda: 4.3.31-py36_0 --> 4.3.33-py36_0 conda-forge
70+
71+
beautifulsoup4 100% |###################################################################| Time: 0:00:00 782.08 kB/s
72+
filelock-3.0.4 100% |###################################################################| Time: 0:00:00 7.95 MB/s
73+
markupsafe-1.0 100% |###################################################################| Time: 0:00:00 5.82 MB/s
74+
pkginfo-1.4.2- 100% |###################################################################| Time: 0:00:00 1.18 MB/s
75+
pycrypto-2.6.1 100% |###################################################################| Time: 0:00:00 1.69 MB/s
76+
pyyaml-3.12-py 100% |###################################################################| Time: 0:00:00 3.31 MB/s
77+
conda-verify-2 100% |###################################################################| Time: 0:00:00 6.91 MB/s
78+
jinja2-2.10-py 100% |###################################################################| Time: 0:00:00 2.81 MB/s
79+
conda-4.3.33-p 100% |###################################################################| Time: 0:00:00 621.27 kB/s
80+
conda-build-2. 100% |###################################################################| Time: 0:00:00 2.16 MB/s
81+
Conda installation succeeded - Conda is available at '/Users/john/miniconda3/bin/conda'

‎docs/_writing_conda_new.rst‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
2+
----------------------------------------------------------------
3+
Building New Conda Packages
4+
----------------------------------------------------------------
5+
6+
Frequently packages your tool will require are not found in Bioconda_
7+
or conda-forge yet. In these cases, it is likely best to contribute
8+
your package to one of these projects. Unless the tool is exceedingly
9+
general Bioconda_ is usually the correct starting point.
10+
11+
.. note:: Many things that are not strictly or even remotely "bio" have
12+
been accepted into Bioconda_ - including tools for image analysis,
13+
natural language processing, and cheminformatics.
14+
15+
To get quickly learn to write Conda_ recipes for typical Galaxy tools,
16+
please read the following pieces of external documentation.
17+
18+
- `Contributing to Bioconda <https://bioconda.github.io/contributing.html>`__ in particular focusing on
19+
20+
- `One time setup <https://bioconda.github.io/contrib-setup.html>`__
21+
- `Contributing a recipe <https://bioconda.github.io/contribute-a-recipe.html>`__ (through "Write a Recipe")
22+
- `Building conda packages <https://conda.io/docs/building/bpp.html#>`__ in particular
23+
24+
- `Building conda packages with conda skeleton <https://conda.io/docs/build_tutorials/pkgs.html>`__ (the best approach for common scripting languages such as R and Python)
25+
- `Building conda packages from scratch <https://conda.io/docs/build_tutorials/pkgs2.html>`__
26+
- `Building conda packages for general code projects <https://conda.io/docs/build_tutorials/postgis.html>`__
27+
- `Using conda build <https://conda.io/docs/building/recipe.html>`__
28+
- Then return to the Bioconda documentation and read
29+
30+
- The rest of "Contributing a recipe" continuing from `Testing locally <https://bioconda.github.io/contribute-a-recipe.html#test-locally>`__
31+
- And finally `Guidelines for bioconda recipes <https://bioconda.github.io/guidelines.html>`__
32+
33+
These guidelines in particular can be skimmed depending on your recipe type, for
34+
instance that document provides specific advice for:
35+
36+
- `Python <https://bioconda.github.io/guidelines.html#python>`__
37+
- `R (CRAN) <https://bioconda.github.io/guidelines.html#r-cran>`__
38+
- `R (Bioconductor) <https://bioconda.github.io/guidelines.html#r-bioconductor>`__
39+
- `Perl <https://bioconda.github.io/guidelines.html#perl>`__
40+
- `C/C++ <https://bioconda.github.io/guidelines.html#c-c>`__
41+
42+
To go a little deeper, you may want to read:
43+
44+
- `Specification for meta.yaml <https://conda.io/docs/building/meta-yaml.html>`__
45+
- `Environment variables <https://conda.io/docs/building/environment-vars.html>`__
46+
- `Custom channels <https://conda.io/docs/custom-channels.html>`__
47+
48+
And finally to debug problems the `Bioconda troubleshooting <https://bioconda.github.io/troubleshooting.html>`__
49+
documentation may prove useful.
50+
51+
----------------------------------------------------------------
52+
Exercise - Build a Recipe
53+
----------------------------------------------------------------
54+
55+
If you have just completed the exercise above - this exercise can be found in parent folder. Get
56+
there with ``cd ../exercise2``. If not, the exercise can be downloaded with

‎docs/_writing_conda_recipe_complete.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
2+
Congratulations on writing a Conda recipe and building a package! Upon succesfully building
3+
and testing such a Bioconda package, you would normally push your branch to Github
4+
and open a pull request. This step is skipped here as to not pollute Bioconda with unneeded
5+
software packages.

‎docs/_writing_conda_search.rst‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
2+
----------------------------------------------------------------
3+
Finding Existing Conda Packages
4+
----------------------------------------------------------------
5+
6+
How did we know what software name and software version to use? We found the existing
7+
packages available for Conda and referenced them. To do this yourself, you can simply
8+
use the planemo command ``conda_search``. If we do a search for ``seqt`` it will show
9+
all the software and all the versions available matching that search term - including
10+
``seqtk``.
11+
12+
::
13+
14+
$ planemo conda_search seqt
15+
/Users/john/miniconda3/bin/conda search --override-channels --channel iuc --channel bioconda --channel conda-forge --channel defaults '*seqt*'
16+
Loading channels: done
17+
# Name Version Build Channel
18+
bioconductor-htseqtools 1.26.0 r3.4.1_0 bioconda
19+
bioconductor-seqtools 1.10.0 r3.3.2_0 bioconda
20+
bioconductor-seqtools 1.10.0 r3.4.1_0 bioconda
21+
bioconductor-seqtools 1.12.0 r3.4.1_0 bioconda
22+
seqtk r75 0 bioconda
23+
seqtk r82 0 bioconda
24+
seqtk r82 1 bioconda
25+
seqtk r93 0 bioconda
26+
seqtk 1.2 0 bioconda
27+
seqtk 1.2 1 bioconda
28+
29+
.. note:: The Planemo command ``conda_search`` is a light wrapper around the underlying
30+
``conda search`` command but configured to use the same channels and other options as
31+
Planemo and Galaxy. The following Conda command would also work to search::
32+
33+
$ $HOME/miniconda3/bin/conda -c iuc -c bioconda -c conda-forge '*seqt*'
34+
35+
For Conda versions 4.3.X or less, the search invocation would be something a bit
36+
different::
37+
38+
$ $HOME/miniconda3/bin/conda -c iuc -c bioconda -c conda-forge seqt
39+
40+
41+
Alternatively the Anaconda_ website can be used to search for packages. Typing ``seqtk``
42+
into the search form on that page and clicking the top result will bring on to `this page
43+
<https://anaconda.org/bioconda/seqtk>`__ with information about the Bioconda package.
44+
45+
When using the website to search though, you need to aware of what channel you are using. By
46+
default, Planemo and Galaxy will search a few different Conda channels. While it is possible
47+
to configure a local Planemo or Galaxy to target different channels - the current best practice
48+
is to add tools to the existing channels.
49+
50+
The existing channels include:
51+
52+
* Bioconda (`github <https://github.com/bioconda/bioconda-recipes>`__ | `conda <https://anaconda.org/bioconda>`__) - best practice channel for various bioinformatics packages.
53+
* Conda-Forge (`github <https://github.com/conda-forge/staged-recipes>`__ | `conda <https://anaconda.org/conda-forge>`__) - best practice channel for general purpose and widely useful computing packages and libraries.
54+
* iuc (`github <https://github.com/galaxyproject/conda-iuc>`__ | `conda <https://anaconda.org/iuc>`__) - best practice channel for other more Galaxy specific packages.

0 commit comments

Comments
 (0)