Revisiting this issue with some numbers from using TravisCI on large tool repositories: If we want to continue to use automated testing like TravisCI with large multi-tool repositories, then we need to have some sort of selective testing based on the recent git history (ideally spot changes since the last test run, and test any potentially affected tool).

The devteam and IUC have a couple of large GitHub repositories containing dozens of individual Galaxy tools or tool collections, as do I:

Even the BLAST+ collection maps to several Tool Shed entries, and the main collection now has many tools in it:

https://github.com/peterjc/galaxy_blast

As previously noted, these big repositories have advantages like being easy for new tool developers to search for usage patterns.

As you know I've been using TravisCI to run continuous integration functional testing of my Galaxy tools, but this works better with shorter test runs. Currently galaxy_blast takes about 30 minutes
per target (including about 1 minute downloading and setting things up), while pico_galaxy takes about 40 minutes (including a larger set of dependencies to download and setup):

It is possible to test multiple targets in parallel (eg Python 2.6 vs 2.7, and Galaxy dev vs master branch), but four targets per build seems greedy with these long run times.

As an experiment yesterday I split out the MIRA wrappers from pico_galaxy into their own repository - TravisCI testing of this takes about 8 minutes per target (about 2 minutes of which is downloading and setting up planemo, Galaxy, and three versions of MIRA):

https://github.com/peterjc/galaxy_mira
https://travis-ci.org/peterjc/galaxy_mira/builds

[As an aside I have tried using "planemo test" here but there are some teething issues, e.g.
https://github.com//issues/291 ]

Purely in terms of more immediate feedback from TravisCI, splitting up larger repositories like pico_galaxy seems like a worthwhile effort.

Note that while the ratio depends on the tools and tests, setting up planemo and the Galaxy instance is a small fraction of the run times in these examples.

In the absence of incremental testing, it is tempting to split things up even more so that one Tool Shed repository maps to a single GitHub repository, at which point the setup overhead (downloading and installing planemo, galaxy, and the tool dependencies) may be much more noticeable.

Explore Incremental Continuous Integration Testing #7

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions