Inconsistent planemo test output (silent failures) #291
Description
I've been trying planemo test with TravisCI on a sub-set of https://github.com/peterjc/pico_galaxy in a new experimental repository https://github.com/peterjc/galaxy_mira
See: https://travis-ci.org/peterjc/galaxy_mira/builds/79507686 (overview of test run 18)
Sub-job 18.1, https://travis-ci.org/peterjc/galaxy_mira/jobs/79507687 and sub-job 18.2 https://travis-ci.org/peterjc/galaxy_mira/jobs/79507689 look good (Python 2.6 using galaxy dev and master):
All 17 test(s) executed passed.
mira_4_0_bait[0]: passed
mira_4_0_bait[1]: passed
mira_4_0_bait[2]: passed
mira_4_0_convert[0]: passed
mira_4_0_convert[1]: passed
mira_4_0_de_novo[0]: passed
mira_4_0_de_novo[1]: passed
mira_4_0_mapping[0]: passed
mira_4_0_mapping[1]: passed
mira_4_9_bait[0]: passed
mira_4_9_bait[1]: passed
mira_4_9_bait[2]: passed
mira_4_9_bait[3]: passed
mira_4_9_bait[4]: passed
mira_4_9_bait[5]: passed
mira_4_9_bait[6]: passed
mira_assembler[0]: passed
The command "planemo test ${TRAVIS_BUILD_DIR}/tools/" exited with 0.
Sub job 18.3 https://travis-ci.org/peterjc/galaxy_mira/jobs/79507691 is confused (Python 2.7 and galaxy dev), note this says mira_4_9_bait[5] failed.
All 17 test(s) executed passed.
mira_4_0_bait[0]: passed
mira_4_0_bait[1]: passed
mira_4_0_bait[2]: passed
mira_4_0_convert[0]: passed
mira_4_0_convert[1]: passed
mira_4_0_de_novo[0]: passed
mira_4_0_de_novo[1]: passed
mira_4_0_mapping[0]: passed
mira_4_0_mapping[1]: passed
mira_4_9_bait[0]: passed
mira_4_9_bait[1]: passed
mira_4_9_bait[2]: passed
mira_4_9_bait[3]: passed
mira_4_9_bait[4]: passed
mira_4_9_bait[5]: failed
mira_4_9_bait[6]: passed
mira_assembler[0]: passed
The command "planemo test ${TRAVIS_BUILD_DIR}/tools/" exited with 0.
Sub job 18.4 https://travis-ci.org/peterjc/galaxy_mira/jobs/79507693 is also confused (Python 2.7 and Galaxy stable), note this says mira_4_0_mapping[1] failed:
All 17 test(s) executed passed.
mira_4_0_bait[0]: passed
mira_4_0_bait[1]: passed
mira_4_0_bait[2]: passed
mira_4_0_convert[0]: passed
mira_4_0_convert[1]: passed
mira_4_0_de_novo[0]: passed
mira_4_0_de_novo[1]: passed
mira_4_0_mapping[0]: passed
mira_4_0_mapping[1]: failed
mira_4_9_bait[0]: passed
mira_4_9_bait[1]: passed
mira_4_9_bait[2]: passed
mira_4_9_bait[3]: passed
mira_4_9_bait[4]: passed
mira_4_9_bait[5]: passed
mira_4_9_bait[6]: passed
mira_assembler[0]: passed
The command "planemo test ${TRAVIS_BUILD_DIR}/tools/" exited with 0.
Notice some of these test runs had a failure (not consistently), yet planemo still says all 17 tests passed, and returns 0 overall.