[16.10] Backport #3106 and #3222: Cached conda environments and API to manage them#3227
Merged
martenson merged 13 commits intogalaxyproject:release_16.10from Nov 29, 2016
Merged
Conversation
Similar to galaxyproject#2986, implement a mechanism that allows tool dependencies to be cached. If the `use_cached_dependency_manager` option is set to True in galaxy.ini, we build a hash of the combination of a tools' requirements, and store the resulting environment in a directory specified by the `tool_dependency_cache_dir` option in galaxy.ini.
to updates or changes in depedencies, folder structure and resolver configuration. Instead of hashing name, type and version of a dependency, hash the json representation of the dependencies returned by the dependency resolver, which include the path to the environment and the depedency type. This is only applied to resolvers whose cacheable attribute is set to True (conda-only, currently).
and only activate cached environments if they exist.
and override __eq__ for ToolRequirement, to simplify checking if ToolRequirements are already installed/cached.
and to build a tool dependency cache (if activated in galaxy.ini).
An example to install dependencies for the twobit converter:
```
import bioblend.galaxy
url = 'http://localhost:8080/'
api_key = 'admin_api_key'
tool_id = 'CONVERTER_fasta_to_2bit'
endpoint = "api/tools/%s/install_dependencies" % tool_id
gi = bioblend.galaxy.GalaxyInstance(url, api_key)
gi.make_post_request("/".join((url, endpoint)), payload={'id': tool_id})
```
If `use_cached_dependency_manager` is activated in the galaxy.ini,
a cached environment can be built like this:
```
endpoint = "api/tools/%s/build_dependency_cache" % tool_id
gi.make_post_request("/".join((url, endpoint)), payload={'id': tool_id})
```
This will remove a pre-existing cached dependency directory.
Contributor
|
(I would love this too 😄 ) |
Member
|
+1 Alright - it is marked as beta I guess anyway and certainly main could use it to eliminate the expensive copying @natefoo is doing right now. |
Member
|
Yeah I see this as a bug fix since w/o it, many deployments will be broken (and since it's optional it won't break any existing working deployments). +1 |
Member
|
Are you considering updating conda version with this fix? |
nsoranzo
reviewed
Dec 6, 2016
| def hash_requirements(self, resolved_dependencies): | ||
| """Return hash for requirements""" | ||
| resolved_dependencies = [[(dep.name, dep.version, dep.exact, dep.dependency_type) for dep in resolved_dependencies]] | ||
| hash_str = json.dumps(sorted([resolved_dependencies])) |
Member
There was a problem hiding this comment.
@mvdbeek There are a few extra brackets here:
resolved_dependencies = [(dep.name, dep.version, dep.exact, dep.dependency_type) for dep in resolved_dependencies]
hash_str = json.dumps(sorted(resolved_dependencies))
But this change may break previously created cached environments.
Member
Author
There was a problem hiding this comment.
Yes, how did I miss that!? :(.
Let's change it while it' not in wide use.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport #3106 and #3222.
It appears that building conda environments in the job directory is problematic in some larger production environments (true for @lecorguille @lparsons and myself ).
Since that seriously cripples Conda usage I think we should backport this to
release_16.10.Ping @bgruening @jmchilton