planemo test silently ignoring tool with tests #765
Description
I was seeing a silent failure locally (running on CentOS, where I let planemo test create its own Galaxy instance), but have now reproduced it on TravisCI (were I pre-create the Galaxy instance in order to try specific branches).
Consider this job: https://travis-ci.org/peterjc/kodoja_galaxy/jobs/338520395 (using Galaxy master)
$ planemo --version
planemo, version 0.47.0
The command "planemo --version" exited with 0.
$ planemo test --galaxy_root ${HOME}/galaxy-${GALAXY_BRANCH} --skip_venv ${TRAVIS_BUILD_DIR}
Testing using galaxy_root /home/travis/galaxy-master
...
----------------------------------------------------------------------
XML: /tmp/tmpkENNF8/xunit.xml
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
I am expecting the tests to fail (as I've not yet told Galaxy how to find the output file).
The commit prior planemo test was finding the three tests https://travis-ci.org/peterjc/kodoja_galaxy/jobs/338510306 and they failed due to the test-data/*.loc file being loaded:
$ planemo --version
planemo, version 0.47.0
The command "planemo --version" exited with 0.
$ planemo test --galaxy_root ${HOME}/galaxy-${GALAXY_BRANCH} --skip_venv ${TRAVIS_BUILD_DIR}
Testing using galaxy_root /home/travis/galaxy-master
...
----------------------------------------
Kodoja database search ( kodoja_search ) > Test-1 ... ERROR
Kodoja database search ( kodoja_search ) > Test-2 ... ERROR
Kodoja database search ( kodoja_search ) > Test-3 ... ERROR
======================================================================
ERROR: Kodoja database search ( kodoja_search ) > Test-1
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/galaxy-master/test/functional/test_toolbox.py", line 317, in test_tool
self.do_it(td)
File "/home/travis/galaxy-master/test/functional/test_toolbox.py", line 68, in do_it
raise e
RunToolException: Error creating a job for these tool inputs - Specify a dataset of the required format / build.
======================================================================
ERROR: Kodoja database search ( kodoja_search ) > Test-2
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/galaxy-master/test/functional/test_toolbox.py", line 317, in test_tool
self.do_it(td)
File "/home/travis/galaxy-master/test/functional/test_toolbox.py", line 68, in do_it
raise e
RunToolException: Error creating a job for these tool inputs - Specify a dataset of the required format / build.
======================================================================
ERROR: Kodoja database search ( kodoja_search ) > Test-3
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/galaxy-master/test/functional/test_toolbox.py", line 317, in test_tool
self.do_it(td)
File "/home/travis/galaxy-master/test/functional/test_toolbox.py", line 68, in do_it
raise e
RunToolException: Error creating a job for these tool inputs - Specify a dataset of the required format / build.
----------------------------------------------------------------------
XML: /tmp/tmpQoCkQ5/xunit.xml
----------------------------------------------------------------------
Ran 3 tests in 1.425s
FAILED (errors=3)
The only change on my side was a single commit peterjc/kodoja_galaxy@cd0113e which setup the *.loc files manually.
My guess is an exception of some sort is being caught and ignored, rather than reported as a test failure?
[Cross reference https://github.com/galaxyproject/galaxy/issues/5466 for why these TravisCI jobs are failing with the Galaxy dev branch]
[Cross reference https://github.com/galaxyproject/galaxy/pull/5465 for improving the exception message for the data table parameter]