Implement beta support for discovering mulled containers for dependencies. by jmchilton · Pull Request #2986 · galaxyproject/galaxy

jmchilton · 2016-09-29T20:58:49Z

tl;dr - Free community managed Docker containers for all tools whose requirements are resolvable with bioconda.

Brings in the latest mulled work that was migrated into galaxy-lib and made to work with combinations of requirements, local caching, non-bioconda channels, etc....

The local test can be run with the following commands:

virtualenv planemo-venv
. planemo-venv/bin/activate
pip install --upgrade pip 
pip install planemo 
planemo serve --galaxy_root . --mulled_containers test/functional/tools/mulled_example_multi_1.xml
# Open localhost:9090 and run the example tool - you will see Galaxy build the container on the fly.

Then open a web browser and run the new test tool - it should find the container you built and use it without any explicit markup required in the tool XML.

This demonstrates the local cached mulled containers - but remote ones on quay.io will also be used dynamically if GALAXY_CONFIG_ENABLE_BETA_MULLED_CONTAINERS is enabled - such as the ones @bgruening built for every package in bioconda.

Update: Replaced old manual test description with much more automated approach using new release of planemo leveraging all this magic. Old instructions were:

virtualenv .galaxy-lib-venv
. .galaxy-lib-venv/bin/activate
pip install galaxy-lib
mulled-build-tool build test/functional/tools/mulled_example_multi_1.xml  # Builds a mulled container locally for tool.
export GALAXY_CONFIG_ENABLE_BETA_MULLED_CONTAINERS=True
emacs config/job_conf.xml # Configure a Docker-enabled destination for mulled_example_multi_1
GALAXY_RUN_WITH_TEST_TOOLS=1 sh run.sh

bgruening · 2016-09-30T20:16:01Z

test/functional/tools/mulled_example_multi_1.xml

@@ -0,0 +1,18 @@
+<tool id="mulled_example_multi_1" name="mulled_example_multi_1" version="0.1.0">
+    <command><![CDATA[


Should this tool test anything? It also fails planemo lint ;)

All the test tools fail planemo lint. As for the tests - I don't have any because this tool really depends on the dependency resolvers working which none of the other framework test tools do. I want them testing the core tool framework - not the ... lets say dependency resolution ecosystem.

bgruening · 2016-09-30T20:23:10Z

Obviously I'm a huge fan of this and because it contains code from us I will not vote. Just one thing this PR contains some useful tools for handling/searching mulled containers and I'm not sure these should be under lib/, but I see that these are useful in galaxy-lib for planemo etc...

Moreover, thanks a lot to @thriqon who's master thesis made all this possible!

mvdbeek · 2016-10-02T11:32:43Z

lib/galaxy/tools/deps/container_resolvers/mulled.py

+        else:
+            name = image_name(targets)
+            for cached_image in cached_images:
+                if name == cached_image.hash:


cached_image can be aCachedMulledImageSingleTarget that doesn't have a hash attribute, right? if cached_image.multi_target and name == cached_image.hash: should be OK

mvdbeek · 2016-10-02T11:43:22Z

lib/galaxy/tools/deps/mulled/util.py

+        raise Exception("requets library is unavailable, functionality not available.")
+
+    url = 'https://quay.io/api/v1/repository/%s/%s' % (namespace, pkg_name)
+    response = requests.get(url, timeout=None)


this fails with:

galaxy.tools.deps.mulled.util WARNING 2016-10-02 11:41:39,416 {u'status': 401, u'error_message': u'Requires authentication', u'title': u'invalid_token', u'error_type': u'invalid_token', u'detail': u'Requires authentication', u'type': u'https://quay.io/api/v1/error/invalid_token'}

~~fwiw, I'm using something like https://gist.github.com/mvdbeek/5e48d2c21c1440e22bda895769f5e450 for v2 registries.~~
seems that you get a 401 also if the repository doesn't exist ...

mvdbeek · 2016-10-02T14:02:29Z

Amazing! If there is only a single requirement to resolve it even works with my WIP udocker implementation.

@mvdbeek

Mentioned by @mvdbeek here galaxyproject/galaxy#2986 (comment).

jmchilton · 2016-10-03T21:01:12Z

Okay - I've pushed through another iteration on this. I've:

Fixed the problem referenced by @mvdbeek if the mulled dependency is missing on quay.io.
Added another container resolver that will actually just build the container as needed - it comes after checking if they are locally cached or available on quay.io.
Made it so that involucro - the tool that handles the Docker interactions when building mulled containers - will install on-demand.
Added more documentation to galaxy.ini.sample.
Added the destination option requires_container - to force job destinations to run containerized jobs and fail if a relevant container is not configured (either by the tool author or deployer). This will enable planemo to force containers to be available during testing and quickly signal an obvious error if they are not.
Update galaxy-lib with some other changes to enable all of this.

jmchilton · 2016-10-05T05:00:14Z

Did a release of planemo which again greatly reduces the barrier to testing this PR. The new --mulled_containers flag should take care of configuring the resolvers and setting a job_conf.xml with Docker enabled. I update the PR description with new testing instructions.

natefoo · 2016-10-05T14:49:48Z

👍 this is amazing.

bgruening · 2016-10-05T15:06:31Z

Oh, we have a 👍 @jmchilton we have a 👍 :)
I think I need to adopt it to the new namespace, but hey we have a 👍 !!!!

@bgruening

…for dependencies. Contains all mulled fixes and enhancements through 16.10.7 including several bug fixes from @bgruening.

This is great for security in production. For my own greedy concerns, I'm adding it for development however. I don't want someone to be testing mulled containers and accidently run the job locally.

jmchilton · 2016-10-08T14:03:30Z

Updated to include the latest fixes from @bgruening's work downstream in galaxy-lib (through version 16.10.7 of that library) and rebased into two clean, atomic commits.

Add initial documentation about the new mulled features.

martenson · 2016-10-10T18:29:27Z

to trigger retesting

jmchilton · 2016-10-11T00:47:46Z

Thanks for the reviews all and thanks for the merge @martenson!

bgruening · 2016-10-11T06:57:37Z

🎉 🎈 I think I take a week off - this PR is awesome!
Docker for free for everyone!

Similar to galaxyproject#2986, implement a mechanism that allows tool dependencies to be cached. If the `use_cached_dependency_manager` option is set to True in galaxy.ini, we build a hash of the combination of a tools' requirements, and store the resulting environment in a directory specified by the `tool_dependency_cache_dir` option in galaxy.ini.

@stevecassidy

Fix thanks to @stevecassidy. Fixes galaxyproject#3207. Broken with introduction of container resolution stradegies in galaxyproject#2986.

jmchilton added kind/feature area/jobs area/tool-framework labels Sep 29, 2016

galaxybot added this to the 16.10 milestone Sep 29, 2016

bgruening reviewed Sep 30, 2016

View reviewed changes

jmchilton added the status/WIP label Sep 30, 2016

jmchilton changed the title ~~Implement beta support for discovering mulled containers for dependencies.~~ [WIP] Implement beta support for discovering mulled containers for dependencies. Sep 30, 2016

mvdbeek reviewed Oct 2, 2016

View reviewed changes

jmchilton added a commit to galaxyproject/galaxy-lib that referenced this pull request Oct 3, 2016

Fix not checking multi-target in mulled cache resolver.

ea64c91

Mentioned by @mvdbeek here galaxyproject/galaxy#2986 (comment).

jmchilton added status/review and removed status/WIP labels Oct 3, 2016

jmchilton changed the title ~~[WIP] Implement beta support for discovering mulled containers for dependencies.~~ Implement beta support for discovering mulled containers for dependencies. Oct 3, 2016

jmchilton force-pushed the mulled branch from 63795a0 to 1c2a352 Compare October 4, 2016 13:16

jmchilton mentioned this pull request Oct 4, 2016

Mulled support galaxyproject/planemo#585

Merged

jmchilton force-pushed the mulled branch from 9246b86 to b026d97 Compare October 8, 2016 13:58

jmchilton added 2 commits October 8, 2016 10:01

Implement beta support for dynamically discovering mulled containers …

5a72449

…for dependencies. Contains all mulled fixes and enhancements through 16.10.7 including several bug fixes from @bgruening.

Implement option to require containerization of jobs.

e389e19

This is great for security in production. For my own greedy concerns, I'm adding it for development however. I don't want someone to be testing mulled containers and accidently run the job locally.

jmchilton force-pushed the mulled branch from b026d97 to e389e19 Compare October 8, 2016 14:02

bgruening and others added 3 commits October 9, 2016 12:40

Add initial documentation about the new mulled features.

654b5e9

Merge pull request #50 from jmchilton/mulled_docs

4b5114b

Add initial documentation about the new mulled features.

Bring in latest galaxy-lib fixes.

f0b988d

martenson closed this Oct 10, 2016

martenson reopened this Oct 10, 2016

martenson added 2 commits October 10, 2016 19:19

remove duplicated option from config sample

20dcc4d

minor changes to mulled admin docs

bcabbd8

martenson approved these changes Oct 10, 2016

View reviewed changes

martenson merged commit 7db241f into galaxyproject:dev Oct 11, 2016

mvdbeek mentioned this pull request Oct 28, 2016

Add cached dependency manager #3106

Merged

jmchilton added a commit to jmchilton/galaxy that referenced this pull request Dec 20, 2016

Fix explicit Docker container resolution.

05cd16f

Fix thanks to @stevecassidy. Fixes galaxyproject#3207. Broken with introduction of container resolution stradegies in galaxyproject#2986.

jmchilton mentioned this pull request Dec 20, 2016

[16.10] Fix explicit Docker container resolution. #3346

Merged

jmchilton mentioned this pull request Feb 27, 2017

Holistic Approach to Container Caching #3673

Open

5 tasks

jmchilton mentioned this pull request May 5, 2017

[17.05] Fixes for containerized job executions. #4033

Merged

		@@ -0,0 +1,18 @@
		<tool id="mulled_example_multi_1" name="mulled_example_multi_1" version="0.1.0">
		<command><![CDATA[

Conversation

jmchilton commented Sep 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bgruening Sep 30, 2016

Choose a reason for hiding this comment

Uh oh!

jmchilton Sep 30, 2016

Choose a reason for hiding this comment

Uh oh!

bgruening commented Sep 30, 2016

Uh oh!

mvdbeek Oct 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mvdbeek Oct 2, 2016

Choose a reason for hiding this comment

Uh oh!

mvdbeek Oct 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mvdbeek commented Oct 2, 2016

Uh oh!

jmchilton commented Oct 3, 2016

Uh oh!

jmchilton commented Oct 5, 2016

Uh oh!

natefoo commented Oct 5, 2016

Uh oh!

bgruening commented Oct 5, 2016

Uh oh!

jmchilton commented Oct 8, 2016

Uh oh!

martenson commented Oct 10, 2016

Uh oh!

jmchilton commented Oct 11, 2016

Uh oh!

bgruening commented Oct 11, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jmchilton commented Sep 29, 2016 •

edited

Loading

mvdbeek Oct 2, 2016 •

edited

Loading

mvdbeek Oct 2, 2016 •

edited

Loading