Add API to install resolver tool dependencies by mvdbeek · Pull Request #3222 · galaxyproject/galaxy

mvdbeek · 2016-11-27T16:53:29Z

and to build a tool dependency cache (if activated in galaxy.ini).

This brings us closer to managing tool dependencies for non-toolshed tools.
In addition this is currently the only way to build a cached environment, except for re-installing a tool.

An example to install dependencies for the twobit converter:

import bioblend.galaxy
url = 'http://localhost:8080/'
api_key = 'admin_api_key'
tool_id = 'CONVERTER_fasta_to_2bit'
endpoint = "api/tools/%s/install_dependencies" % tool_id
gi = bioblend.galaxy.GalaxyInstance(url, api_key)
gi.make_post_request("/".join((url, endpoint)), payload={'id': tool_id})

If use_cached_dependency_manager is activated in the galaxy.ini,
a cached environment can be built like this:

endpoint = "api/tools/%s/build_dependency_cache" % tool_id
gi.make_post_request("/".join((url, endpoint)), payload={'id': tool_id})

and to build a tool dependency cache (if activated in galaxy.ini). An example to install dependencies for the twobit converter: ``` import bioblend.galaxy url = 'http://localhost:8080/' api_key = 'admin_api_key' tool_id = 'CONVERTER_fasta_to_2bit' endpoint = "api/tools/%s/install_dependencies" % tool_id gi = bioblend.galaxy.GalaxyInstance(url, api_key) gi.make_post_request("/".join((url, endpoint)), payload={'id': tool_id}) ``` If `use_cached_dependency_manager` is activated in the galaxy.ini, a cached environment can be built like this: ``` endpoint = "api/tools/%s/build_dependency_cache" % tool_id gi.make_post_request("/".join((url, endpoint)), payload={'id': tool_id}) ```

bgruening · 2016-11-27T20:10:59Z

@mvdbeek this is a great idea! Can we include an automatics conda clean in this endpoint or should this be a separate API endpoint.

mvdbeek · 2016-11-27T21:23:06Z

Can we include an automatics conda clean in this endpoint or should this be a separate API endpoint.

I don't know, I guess I would put this under the resolvers endpoint. In there we could have a general cleanup function, that would map to conda clean, or in the case of docker to docker rm/rmi. Does that make sense?

bgruening · 2016-11-27T21:27:47Z

Mh ... I think we need to clean at the end of bulk-installations and not after every single installation.
To make it clear, here I mean conda clean to remove the tarballs etc ... not the installation. This will free a lot of space.
I don't know of anything comparable in the Docker world.

bgruening · 2016-11-27T21:30:16Z

Overall, this is a nice PR and needed, so +1 from me.

mvdbeek · 2016-11-27T21:45:09Z

conda clean is not bound to a particular environment or tool, right?
Does a POST to /api/dependencies_resolvers/clean or cleanup seem reasonable?
I agree, for docker it is not very apparent what would need to be cleaned, but if we ever support apt, guix or sth. else this could be extended to map to the respective cleanup function.

Then again, I'm just hacking away here, input from people that know something about API design is more than welcome.

bgruening · 2016-11-27T21:50:13Z

conda clean is not bound to an environment, like apt-get clean. /api/dependencies_resolvers/clean seems like a good idea.

This will remove a pre-existing cached dependency directory.

…lation

mvdbeek · 2016-11-27T23:09:52Z

@bgruening Alright, I've added the /api/dependencies_resolvers/clean endpoint and cherry-picked the conda_clean function from #2931 .

bgruening · 2016-11-27T23:15:41Z

Cool, thanks!

jmchilton · 2016-11-28T20:48:13Z

Awesome sauce - thanks a bunch @mvdbeek!

martenson · 2016-11-28T22:01:48Z

does merging this make #2931 obsolete?

[16.10] Backport #3106 and #3222: Cached conda environments and API to manage them

martenson · 2016-12-01T14:39:50Z

Do these changes need to be reflected in Conda FAQ https://docs.galaxyproject.org/en/master/admin/conda_faq.html ?

We need to keep that resource up to date and top notch.

@mvdbeek @jmchilton

mvdbeek · 2016-12-01T16:35:08Z

Yes, I'll try to get to this.

nsoranzo · 2016-12-06T19:03:24Z

lib/galaxy/tools/deps/__init__.py

        resolved_dependencies = self.requirements_to_dependencies(requirements, **kwds)
        cacheable_dependencies = [dep for req, dep in resolved_dependencies.items() if dep.cacheable]
        hashed_requirements_dir = self.get_hashed_requirements_path(cacheable_dependencies)
+        if kwds.get('force_rebuild', False) and os.path.exists(hashed_requirements_dir):


@mvdbeek What happens if an admin installs a tool which has the same set of requirements of a previously installed-and-cached environment (and force_rebuild is False)?

This is actually a bit inconsistent now (given that initially there was no way to build/rebuild a cache):
If you install a tool with the same set of requirements, you do build the cache again, since this was the only way to fix a broken cache or update a cache with new packages (short of deleting the cache).
But now I think we should expose this in the details section of the tool installation and extend the install repository API endpoint for this parameter. Does that sound like a good idea?

In fact we should have a UI to manage these things independently of the install process. The pieces are all there, but I don't feel up to the task of building a "Manage tool dependencies" page. That would be a huge win IMO. I have a clear picture in my head how this should look like, but looking through the codebase I don't really see something that I could base my work on like I did for the conda install process.

If you install a tool with the same set of requirements, you do build the cache again, since this was the only way to fix a broken cache

I think that should be done only when using the new force_rebuild option, and yes, we need an UI for that!

or update a cache with new packages (short of deleting the cache).

Do you mean updating a package that is already in the cached env or adding a new package to it? I don't think the second should happen, because that should have a different hash. And for the first, rebuilding is probably better.

So, I'd change this code to be:

if os.path.exists(hashed_requirements_dir): if kwds.get('force_rebuild', False): try: shutil.rmtree(hashed_requirements_dir) except Exception: log.warning("Could not delete cached requirements directory '%s'" % hashed_requirements_dir) pass else: log.debug("Cached environment %s already exists, skipping build", hashed_requirements_dir) return [dep.build_cache(hashed_requirements_dir) for dep in cacheable_dependencies]

In fact we should have a UI to manage these things independently of the install process. The pieces are all there, but I don't feel up to the task of building a "Manage tool dependencies" page.

I'd love that! Hopefully someone from the core Galaxy Team can help?

So, I'd change this code to be:

yep, that looks pefect. Do you want to open a PR? Otherwise I can include this with a Documentation update.

Thank @mvdbeek, I'll open a PR later.

mvdbeek added 2 commits November 27, 2016 17:00

Add integration test-case for install tool dependency API endpoint

e905b8b

mvdbeek added area/admin area/API area/tool-dependencies kind/feature status/review labels Nov 27, 2016

mvdbeek added this to the 17.01 milestone Nov 27, 2016

Install the correct converter dependency

3e48f37

mvdbeek added the status/WIP label Nov 27, 2016

mvdbeek and others added 4 commits November 27, 2016 22:55

Add force_rebuild param to build_dependency_cache

8ff7ffa

This will remove a pre-existing cached dependency directory.

Add dependency_resolvers/clean endpoint

3ebbb37

add a conda clean function to remove downloaded tarballs after instal…

23c6b1c

…lation

Connect clean endpoint properly and add -y to exec_clean

00ca0a6

mvdbeek removed the status/WIP label Nov 27, 2016

jmchilton merged commit fd0a52a into galaxyproject:dev Nov 28, 2016

martenson mentioned this pull request Nov 28, 2016

WIP: add a conda clean function #2931

Closed

mvdbeek mentioned this pull request Nov 29, 2016

Add cached dependency manager #3106

Merged

martenson added a commit that referenced this pull request Nov 29, 2016

Merge pull request #3227 from mvdbeek/cached_deps_16.10

d6f0e6b

[16.10] Backport #3106 and #3222: Cached conda environments and API to manage them

nsoranzo reviewed Dec 6, 2016

View reviewed changes

mvdbeek mentioned this pull request Jan 27, 2017

[16.10] Backport #3106 and #3222: Cached conda environments and API to manage them #3227

Merged

mvdbeek deleted the api_cache_env branch June 12, 2018 12:56

Conversation

mvdbeek commented Nov 27, 2016

Uh oh!

bgruening commented Nov 27, 2016

Uh oh!

mvdbeek commented Nov 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bgruening commented Nov 27, 2016

Uh oh!

bgruening commented Nov 27, 2016

Uh oh!

mvdbeek commented Nov 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bgruening commented Nov 27, 2016

Uh oh!

mvdbeek commented Nov 27, 2016

Uh oh!

bgruening commented Nov 27, 2016

Uh oh!

jmchilton commented Nov 28, 2016

Uh oh!

martenson commented Nov 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martenson commented Dec 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mvdbeek commented Dec 1, 2016

Uh oh!

nsoranzo Dec 6, 2016

Choose a reason for hiding this comment

Uh oh!

mvdbeek Dec 7, 2016 • edited by nsoranzo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mvdbeek Dec 7, 2016

Choose a reason for hiding this comment

Uh oh!

nsoranzo Dec 7, 2016

Choose a reason for hiding this comment

Uh oh!

mvdbeek Dec 7, 2016

Choose a reason for hiding this comment

Uh oh!

nsoranzo Dec 7, 2016

Choose a reason for hiding this comment

Uh oh!

nsoranzo Dec 9, 2016

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mvdbeek commented Nov 27, 2016 •

edited

Loading

mvdbeek commented Nov 27, 2016 •

edited

Loading

martenson commented Nov 28, 2016 •

edited

Loading

martenson commented Dec 1, 2016 •

edited

Loading

mvdbeek Dec 7, 2016 •

edited by nsoranzo

Loading