Externally Executable Tool Tests by jmchilton · Pull Request #5545 · galaxyproject/galaxy

jmchilton · 2018-02-15T21:31:11Z

At a high-level this PR adds API and library (galaxy-lib) functionality to allow tool tests to be executed from outside the test framework. A wrapper around the galaxy-lib script has been placed in scripts/ for easy usage from Galaxy's git directory.

python scripts/run_tool_test.py \ 
             --key aef9a18deb993ead39317e439a4d7d5f \
             --admin-key aef9a18deb993ead39317e439a4d7d5f \ 
             --tool-id cat1 \
             [--url 'http://localhost:8080'] \
             [--text-index 0] \
             [--tool-version 1.0.0]

Many years ago and prior to the tool shed and the API, GALAXY_TEST_EXTERNAL could be used to replicate this and run tool tests against a running Galaxy but it would require the tool test framework to load the same toolbox in memory as running on the remote server. While I believe this modality should be fixed (GALAXY_TEST_EXTERNAL works for various other tests and should work for tool tests again now) this new approach is API driven and does not require the tool testing client to load Galaxy tools or toolbox into memory. This allows tests to be driven from outside of Galaxy's test framework and outside of Galaxy all together (from galaxy-lib for example - and hopefully eventually Planemo for the added goodies it provides).

More details about this PR:

Adds a new internal abstraction and API for determining test data paths for tools. This is now used by the TDT and for tool testing functionality. This should fix bugs in the TDT webhook related to how it finds test data - in particular the tool shed code to find test-data directories is now unified across approaches and correct for the TDT.

This API also allows eliminating various conditional paths through the test framework that used to hold onto tool shed data that is now resolved at runtime by Galaxy.
Adds a tool API endpoint to provide summary information for all tools. This is used to generate test cases in the test framework.
Adds an API endpoint for fetch descriptions of test data for particular tools. This is marked as experimental because I'm not super happy with how the internal data structures mapped to JSON.
Add a test case for installing a tool from the tool shed and then executing a tool test against it (huge thanks to @mvdbeek for taking care of the basics of setting up this integration - I've refactored his data manager test code to implement 96% this test).

xref galaxyproject/ephemeris#76
xref #5474

jmchilton · 2018-02-21T02:25:00Z

lib/galaxy/tools/__init__.py

+
+        # If a tool shed tool, try to find where the repository was installed. Is there a cleaner way
+        # to do this @mvdbeek?
+        if hasattr(self, 'tool_shed') and self.tool_shed:


@mvdbeek Does this piece of code to find a shed tool's test-data directory from inside of Galaxy make sense to you? In particular is there a better way to recover what I am calling repository_dir here - the test-data stuff a little bit lower is Greg code from the test framework so I assume that is right.

Hmm, I'm probably missing something here, but shouldn't the test-data dir live always next to self.tool_dir ?
I'm not sure why we need to walk up to the repository_dir to descend down again to find test-data ?

So this is code stolen from the previous method of resolving tool shed test data - https://github.com/galaxyproject/galaxy/blob/dev/test/base/tool_shed_util.py#L25. I guess there is descended from the repository directory in case the test-data was found next to a tool installed below the repository top of the repository:

repo/ - mytools/ - tool1_1.xml - tool1_2.xml - test-data/

In that case though you are right the test-data would still live next to the tool. My thought was that it would be good to move to the top of the directory in case tools are stored in directories below the top-level but the test-data is stored at the top-level. Repositories like this:

repo/ - README.txt - test-data/ - mytools1/ - tool1_1.xml - tool1_2.xml - mytools2/ - tool2_1.xml

That seems like a reasonable repository layout for a repository with a lot of tools - I don't know though. What do you think? I'm perfectly willing to just let IUC declare that test-data needs to be right next to the tool in the repository bundle for this to work - that is fine to me.

I see, does this alternative layout work with planemo ? If it doesn't I'd assume it's not a very common layout, in which case we could perhaps first check if filename exists in self.tool_dir before doing something more complicated.

wrt repository_dir we've got this:

galaxy/lib/galaxy/model/tool_shed_install/__init__.py

Lines 294 to 301 in a2b8e44

def repo_path(self, app):

tool_shed = common_util.remove_protocol_and_port_from_tool_shed_url(self.tool_shed)

for shed_tool_conf_dict in app.toolbox.dynamic_confs(include_migrated_tool_conf=True):

tool_path = shed_tool_conf_dict['tool_path']

relative_path = os.path.join(tool_path, tool_shed, 'repos', self.owner, self.name, self.installed_changeset_revision)

if os.path.exists(relative_path):

return relative_path

return None

I'm not sure the logic of relying on self.installed_changeset_revision will work in all cases (Could perhaps explain a subset of help images missing every now and then?!)

https://github.com/galaxyproject/planemo/blob/master/planemo/galaxy/config.py#L999

Planemo would allow use of both the pwd and the tool directory if they are different. So I think I imagined it working this way previously. Also --test-data is configurable on the command-line for testing and can be placed where ever in the suite using .shed.yml. Happy to call it unnecessary complex though.

Alright, that makes sense. I don't think it's too complex, this offers some nice flexibility.

hexylena · 2018-02-21T13:35:37Z

Will test tomorrow against freiburg. This is exciting. Will help validate all the tools we installed via https://github.com/usegalaxy-eu/usegalaxy-eu-tools/ after starting jenkins jobs for that yesterday.

jmchilton · 2018-02-21T13:39:48Z

@erasche That script doesn't produce xunit or anything yet - just runs one test and produces an exit code (and Python exception if the test fails) - I hope to build the reporting and aggregation half of the puzzle into Planemo I think which does produce nice XUnit and HTML reports already. You may want to wait for that before putting this into production - up to you.

hexylena · 2018-02-21T13:44:33Z

No xunit is fine for now, just happy to have some evidence that all the tools we've installed are functional (and happy to be able to provide some more testing of this.)

mvdbeek · 2018-02-23T14:53:59Z

lib/galaxy/tools/__init__.py

+                # Eventually we should find a directory named self.changeset_revision
+                # but just in case we don't (differences between self.installed_changeset and self.changeset_revision)
+                # check for repository name also.
+                if repository_dir_name == self.changeset_revision or repository_dir_name == self.repository_name:


The directory named self.repository_name should come first, judging by my disk layout: ~/src/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/minimap2/4070f129540a/minimap2
If that's good enough I wouldn't bother checking for self.changeset_revision(and adjust the comment).

Okay - I'll make this revision. Thanks!

See galaxyproject/galaxy#5545 for details.

This work is focused on "test" and "run" commands for Galaxy tools and workflows, though it should enable other combinations of artifacts and commands such as serving workflows and various CWL artifact operations against that fork of Galaxy. The tool piece of this requires an unreleased version of galaxy-lib (galaxyproject/galaxy-lib#91) and unmerged Galaxy modifications to expose a APIs for external tool testing (galaxyproject/galaxy#5545). Workflows are oddly enough likely closer to working with this WIP. Implements galaxyproject#592. Implements galaxyproject#508.

Can be shared between tests and TDTs. A step toward not requiring one to load the tool to test the tool (important for external testing of tools).

Should enable greater reuse from outside the test framework (e.g. standalone scripts and Planemo).

Working toward making interactor stand-alone and independent of the test framework.

This gets us a lot closer to being able to construct a ToolTestBuilder from simple dictionaries generated from the API for instance or from any tool really.

Use this is in tool testing. Prevents needing to load / have tools in memory to run tool tests - should allow external tool testing.

Previously this was set in the test framework but that doesn't help running Galaxies.

…shed.

…ools.

@mvdbeek

Based on conversations with @mvdbeek on https://github.com/galaxyproject/galaxy/pull/5545/files.

Capture more information about the test.

This work is focused on "test" and "run" commands for Galaxy tools and workflows, though it should enable other combinations of artifacts and commands such as serving workflows and various CWL artifact operations against that fork of Galaxy. The tool piece of this requires an unreleased version of galaxy-lib (galaxyproject/galaxy-lib#91) and unmerged Galaxy modifications to expose a APIs for external tool testing (galaxyproject/galaxy#5545). Workflows are oddly enough likely closer to working with this WIP. Implements galaxyproject#592. Implements galaxyproject#508.

jmchilton added kind/enhancement status/WIP kind/refactoring cleanup or refactoring of existing code, no functional changes area/testing area/tool-framework labels Feb 15, 2018

jmchilton force-pushed the external_tool_tests branch 4 times, most recently from ebea680 to b565834 Compare February 20, 2018 20:30

jmchilton changed the title ~~[WIP] Refactor toward externally executable tool tests.~~ [WIP] Externally Executable Tool Tests Feb 20, 2018

jmchilton force-pushed the external_tool_tests branch 2 times, most recently from d5d63b4 to d603baf Compare February 21, 2018 02:21

jmchilton commented Feb 21, 2018

View reviewed changes

jmchilton changed the title ~~[WIP] Externally Executable Tool Tests~~ Externally Executable Tool Tests Feb 21, 2018

jmchilton added status/review and removed status/WIP labels Feb 21, 2018

jmchilton force-pushed the external_tool_tests branch from d603baf to 628a90b Compare February 21, 2018 02:42

galaxybot added this to the 18.05 milestone Feb 21, 2018

mvdbeek reviewed Feb 23, 2018

View reviewed changes

jmchilton force-pushed the external_tool_tests branch 2 times, most recently from 0177eb4 to 35eb206 Compare February 26, 2018 01:59

jmchilton added a commit to jmchilton/galaxy-lib that referenced this pull request Feb 26, 2018

Bring in Galaxy updates to run externally-driven tool tests.

d652665

See galaxyproject/galaxy#5545 for details.

jmchilton mentioned this pull request Feb 26, 2018

Bring in Galaxy updates to run externally-driven tool tests. galaxyproject/galaxy-lib#91

Merged

jmchilton mentioned this pull request Feb 26, 2018

Implement external Galaxy engine. galaxyproject/planemo#781

Merged

jmchilton added 2 commits February 27, 2018 10:43

Create fewer objects when instantiating tools.

c06e3db

New abstraction to obtain tool test data paths.

1d63bd7

Can be shared between tests and TDTs. A step toward not requiring one to load the tool to test the tool (important for external testing of tools).

jmchilton added 14 commits February 27, 2018 10:43

Eliminate shed tools dict no longer used.

f7d8d50

Eliminate tool shed test helper no longer used.

26170e5

Refactor more tool test verification code into galaxy-lib.

bf856b3

Should enable greater reuse from outside the test framework (e.g. standalone scripts and Planemo).

Refactor polling code from testcase into more standalone interactor.

07e9daf

Working toward making interactor stand-alone and independent of the test framework.

Refactor much more tool testing code into galaxy-lib.

fa372a9

Refactor ToolTestBuilder to not require an in-memory Tool object.

7067416

This gets us a lot closer to being able to construct a ToolTestBuilder from simple dictionaries generated from the API for instance or from any tool really.

Serialize tool test descriptions through the API.

434920c

Use this is in tool testing. Prevents needing to load / have tools in memory to run tool tests - should allow external tool testing.

Refactor tool test outline construction to allow more reuse.

0ae038b

Setup a default GALAXY_TEST_FILE_DIR in case unset in the Galaxy itself.

9668406

Previously this was set in the test framework but that doesn't help running Galaxies.

Refactor data manager test to allow more integration tests with tool …

07008b2

…shed.

Integration test case for running tool tests against shed installed t…

e442aa5

…ools.

Revise repository directory calculation in tool class.

093fdf3

Based on conversations with @mvdbeek on https://github.com/galaxyproject/galaxy/pull/5545/files.

Better default configuration of Galaxy test data directory.

ac96c01

Add structured output json example to galaxy.tools.verify.script.

ec15086

Capture more information about the test.

jmchilton changed the title ~~Externally Executable Tool Tests~~ [WIP] Externally Executable Tool Tests Feb 27, 2018

jmchilton added area/API and removed status/review labels Feb 27, 2018

galaxybot added the status/WIP label Feb 27, 2018

jmchilton force-pushed the external_tool_tests branch from 35eb206 to bb075a6 Compare February 27, 2018 16:27

Stronger security around test data path resolution, now with tests.

dbfde96

jmchilton force-pushed the external_tool_tests branch from bb075a6 to dbfde96 Compare February 27, 2018 16:46

jmchilton changed the title ~~[WIP] Externally Executable Tool Tests~~ Externally Executable Tool Tests Feb 28, 2018

jmchilton added status/review and removed status/WIP labels Feb 28, 2018

jmchilton mentioned this pull request Mar 1, 2018

Externally Executable Tool Tests 2 (with better API) #5628

Merged

jmchilton closed this Mar 7, 2018

jmchilton mentioned this pull request Mar 8, 2018

Tool Testing Fix and Enhancements #5667

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Externally Executable Tool Tests#5545

Externally Executable Tool Tests#5545
jmchilton wants to merge 17 commits intogalaxyproject:devfrom
jmchilton:external_tool_tests

jmchilton commented Feb 15, 2018 •

edited

Loading

Uh oh!

jmchilton Feb 21, 2018

Uh oh!

mvdbeek Feb 23, 2018

Uh oh!

jmchilton Feb 23, 2018

Uh oh!

mvdbeek Feb 23, 2018

Uh oh!

jmchilton Feb 23, 2018

Uh oh!

mvdbeek Feb 23, 2018

Uh oh!

hexylena commented Feb 21, 2018

Uh oh!

jmchilton commented Feb 21, 2018 •

edited

Loading

Uh oh!

hexylena commented Feb 21, 2018

Uh oh!

mvdbeek Feb 23, 2018 •

edited

Loading

Uh oh!

jmchilton Feb 23, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	def repo_path(self, app):
	tool_shed = common_util.remove_protocol_and_port_from_tool_shed_url(self.tool_shed)
	for shed_tool_conf_dict in app.toolbox.dynamic_confs(include_migrated_tool_conf=True):
	tool_path = shed_tool_conf_dict['tool_path']
	relative_path = os.path.join(tool_path, tool_shed, 'repos', self.owner, self.name, self.installed_changeset_revision)
	if os.path.exists(relative_path):
	return relative_path
	return None

Conversation

jmchilton commented Feb 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmchilton Feb 21, 2018

Choose a reason for hiding this comment

Uh oh!

mvdbeek Feb 23, 2018

Choose a reason for hiding this comment

Uh oh!

jmchilton Feb 23, 2018

Choose a reason for hiding this comment

Uh oh!

mvdbeek Feb 23, 2018

Choose a reason for hiding this comment

Uh oh!

jmchilton Feb 23, 2018

Choose a reason for hiding this comment

Uh oh!

mvdbeek Feb 23, 2018

Choose a reason for hiding this comment

Uh oh!

hexylena commented Feb 21, 2018

Uh oh!

jmchilton commented Feb 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hexylena commented Feb 21, 2018

Uh oh!

mvdbeek Feb 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmchilton Feb 23, 2018

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jmchilton commented Feb 15, 2018 •

edited

Loading

jmchilton commented Feb 21, 2018 •

edited

Loading

mvdbeek Feb 23, 2018 •

edited

Loading