Skip to content

Commit 7be1bf5

Browse files
jmchiltonjmchilton
committed
Implement Engine and Runnable abstractions.
``Runnable``s are things such as Galaxy tools, Galaxy workflows, CWL tools, CWL workflows and ``Engine``s provide operations over ``Runnable``s. There are two Engines right now - cwltool and Galaxy and one command that uses the Engine abstraction (``run`` - formerly ``cwl_run``). Exit code handling is improved to so fixed exit codes result when unknown runnables are encountered or when a runnable is matched with an incompatible engine. A dependency on ``aenum`` is added to get enable nice enum functionality in planemo to describe runnable types. The ``Engine`` abstraction implements running Galaxy and CWL tools so far (with either Galaxy or cwltool) but this commit contain a lot of infrastructure for running Galaxy tools and workflows of either type. In particular, workflows will now be uploaded as needed and IDs tracked, they can be executed, and what remains is to collect outputs. The generalized run command implements #449 and #364 - at least in some initial fashion. Initial infrastructure for a generic test command across engine and runnable types is also present (xref #448). If ``planemo test`` encounters a runnable type other than galaxy_tool and directory - such as a cwl tool, a workflow, etc... it will use a new planemo engine-driven testing procedure. To do this an engine "job" is executed, the results collected, and the results "checked". Currently, Galaxy workflows do not have their results collected and the only "check" available is exact non-File matches and sha1 checksums as used by ``cwltest.py``. A complete suite of checks matching what is available in Galaxy will hopefully be implemented shortly. This new engine-driven testing produces a structured test data structure that should match what is generated by Galaxy regardless of which engine is used. The upshot of this is that existing test report infrastructure in planemo should be reusable. ``planemo test_report`` and ``planemo share_test`` should work for any ``Engine`` and any ``RunnableType``. Despite not yet using ``Engine``s infrastructure, the ``serve`` command and related infrastructure for Galaxy has been updated to consume runnables instead of paths. An upshot of this is that serve has been updated to allow serving workflows (either ``.ga`` files directly or Format 2 Workflows through the new gxformat2 library). In order to serve, run, and test gxformat2 workflows - planemo now depends on the gxformat2 library (https://github.com/jmchilton/gxformat2). The depenedency on bioblend has been constrained to be newer so that workflow invocation client stuff can be used. The commit also contains general improvements for CWL support - in particular the cwl-runner implementation driven by planemo is much enhanced and now handles (in at least a minimal fashion) all arguments required to be used by the conformance test code. This commit also introduces a test script ``scripts/test_confromance_cwl.bash`` that runs the common workflow language conformance tests against Planemo/Galaxy. Fixes #453. This commit also includes a variety of improvements to ``run`` and ``serve`` tests. - Refactor serve test cases to allow more reuse. - Fix profile test to allow rerunning on the same machine more than once. - Docstrings.
1 parent 4d29bf1 commit 7be1bf5

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

63 files changed

+2344
-426
lines changed

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,11 @@ develop-eggs
1919
lib
2020
lib64
2121

22+
# For building cwl-runner package.
2223
cwl-runner/dist
2324
cwl-runner/build
25+
# For running CWL conformance tests.
26+
common-workflow-language
2427

2528
# Installer logs
2629
pip-log.txt

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,9 @@ clean-test: ## remove test and coverage artifacts
4242
rm -f .coverage
4343
rm -fr htmlcov/
4444

45+
install:
46+
python setup.py install && cd cwl-runner && python setup.py install
47+
4548
setup-venv: ## setup a development virutalenv in current directory
4649
if [ ! -d $(VENV) ]; then virtualenv $(VENV); exit; fi;
4750
$(IN_VENV) pip install -r requirements.txt && pip install -r dev-requirements.txt

‎cwl-runner/cwl-runner‎

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,42 @@
1-
#!/bin/sh
1+
#!/bin/bash
22

3-
planemo cwl_run "$@"
3+
PLANEMO_BIN="${PLANEMO_BIN:-planemo}"
4+
PLANEMO_ARGS="${PLANEMO_ARGS:-}"
5+
6+
VERBOSE_ARG="--verbose"
7+
USE_CONTAINER=0
8+
9+
# Pop args meant for common_startup.sh
10+
while :
11+
do
12+
case "$1" in
13+
--version|--help)
14+
$PLANEMO_BIN "$1"
15+
exit $?
16+
;;
17+
--no-container)
18+
USE_CONTAINER=1
19+
shift
20+
;;
21+
--quiet)
22+
VERBOSE_ARG=""
23+
shift
24+
;;
25+
--basedir*)
26+
BASEDIR=`echo $1 | sed -e 's/^[^=]*=//g'`
27+
#echo `pwd`
28+
#cd "$BASEDIR"
29+
shift
30+
;;
31+
"")
32+
break
33+
;;
34+
*)
35+
PLANEMO_ARGS="$PLANEMO_ARGS $1"
36+
shift
37+
;;
38+
esac
39+
done
40+
41+
# TODO: Do something with USE_CONTAINER
42+
$PLANEMO_BIN $VERBOSE_ARG run $PLANEMO_ARGS

‎docs/planemo.commands.rst‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -68,14 +68,6 @@ planemo.commands.cmd_create_gist module
6868
:undoc-members:
6969
:show-inheritance:
7070

71-
planemo.commands.cmd_cwl_run module
72-
-----------------------------------
73-
74-
.. automodule:: planemo.commands.cmd_cwl_run
75-
:members:
76-
:undoc-members:
77-
:show-inheritance:
78-
7971
planemo.commands.cmd_cwl_script module
8072
--------------------------------------
8173

@@ -140,6 +132,14 @@ planemo.commands.cmd_project_init module
140132
:undoc-members:
141133
:show-inheritance:
142134

135+
planemo.commands.cmd_run module
136+
-------------------------------
137+
138+
.. automodule:: planemo.commands.cmd_run
139+
:members:
140+
:undoc-members:
141+
:show-inheritance:
142+
143143
planemo.commands.cmd_serve module
144144
---------------------------------
145145

‎docs/planemo.cwl.rst‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,6 @@ planemo.cwl package
44
Submodules
55
----------
66

7-
planemo.cwl.client module
8-
-------------------------
9-
10-
.. automodule:: planemo.cwl.client
11-
:members:
12-
:undoc-members:
13-
:show-inheritance:
14-
157
planemo.cwl.run module
168
----------------------
179

‎docs/planemo.engine.rst‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
planemo.engine package
2+
======================
3+
4+
Submodules
5+
----------
6+
7+
planemo.engine.cwltool module
8+
-----------------------------
9+
10+
.. automodule:: planemo.engine.cwltool
11+
:members:
12+
:undoc-members:
13+
:show-inheritance:
14+
15+
planemo.engine.factory module
16+
-----------------------------
17+
18+
.. automodule:: planemo.engine.factory
19+
:members:
20+
:undoc-members:
21+
:show-inheritance:
22+
23+
planemo.engine.galaxy module
24+
----------------------------
25+
26+
.. automodule:: planemo.engine.galaxy
27+
:members:
28+
:undoc-members:
29+
:show-inheritance:
30+
31+
planemo.engine.interface module
32+
-------------------------------
33+
34+
.. automodule:: planemo.engine.interface
35+
:members:
36+
:undoc-members:
37+
:show-inheritance:
38+
39+
40+
Module contents
41+
---------------
42+
43+
.. automodule:: planemo.engine
44+
:members:
45+
:undoc-members:
46+
:show-inheritance:

‎docs/planemo.galaxy.rst‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,14 @@ Subpackages
1111
Submodules
1212
----------
1313

14+
planemo.galaxy.activity module
15+
------------------------------
16+
17+
.. automodule:: planemo.galaxy.activity
18+
:members:
19+
:undoc-members:
20+
:show-inheritance:
21+
1422
planemo.galaxy.api module
1523
-------------------------
1624

@@ -51,6 +59,14 @@ planemo.galaxy.serve module
5159
:undoc-members:
5260
:show-inheritance:
5361

62+
planemo.galaxy.workflows module
63+
-------------------------------
64+
65+
.. automodule:: planemo.galaxy.workflows
66+
:members:
67+
:undoc-members:
68+
:show-inheritance:
69+
5470

5571
Module contents
5672
---------------

‎docs/planemo.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,13 @@ Subpackages
88

99
planemo.commands
1010
planemo.cwl
11+
planemo.engine
1112
planemo.galaxy
1213
planemo.linters
1314
planemo.reports
1415
planemo.shed
1516
planemo.shed2tap
17+
planemo.test
1618
planemo.xml
1719

1820
Submodules
@@ -114,6 +116,14 @@ planemo.options module
114116
:undoc-members:
115117
:show-inheritance:
116118

119+
planemo.runnable module
120+
-----------------------
121+
122+
.. automodule:: planemo.runnable
123+
:members:
124+
:undoc-members:
125+
:show-inheritance:
126+
117127
planemo.shed_lint module
118128
------------------------
119129

‎docs/planemo.test.rst‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
planemo.test package
2+
====================
3+
4+
Submodules
5+
----------
6+
7+
planemo.test.results module
8+
---------------------------
9+
10+
.. automodule:: planemo.test.results
11+
:members:
12+
:undoc-members:
13+
:show-inheritance:
14+
15+
16+
Module contents
17+
---------------
18+
19+
.. automodule:: planemo.test
20+
:members:
21+
:undoc-members:
22+
:show-inheritance:

‎planemo/cli.py‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,10 @@
1111
OptionSource,
1212
)
1313
from planemo.galaxy import profiles
14+
from planemo.exit_codes import ExitCodeException
1415
from planemo import __version__
1516

16-
PYTHON_2_7_COMMANDS = ["cwl_run", "cwl_script"]
17+
PYTHON_2_7_COMMANDS = ["run", "cwl_script"]
1718
IS_PYTHON_2_7 = sys.version_info[0] == 2 and sys.version_info[1] >= 7
1819

1920

@@ -25,12 +26,6 @@
2526
}
2627

2728

28-
class ExitCodeException(Exception):
29-
30-
def __init__(self, exit_code):
31-
self.exit_code = exit_code
32-
33-
3429
class Context(object):
3530
"""Describe context of Planemo computation.
3631

0 commit comments

Comments
 (0)