Make Galaxy (almost) packageable/installable by natefoo · Pull Request #921 · galaxyproject/galaxy

natefoo · 2015-10-15T03:32:53Z

2019 Update

This won't be entirely ready for pip install galaxy but does what's needed to quickly finish it up. The plan is for the galaxy-web-apps package package essentially a full install of Galaxy with its web stack, and the galaxy metapackage will be pinned to stable releases. Because the packagification of Galaxy began just before updating this PR happened, the method originally used here (one big package) is no longer valid, and the metapackage isn't working yet. That will happen in a subsequent PR.

This PR includes some important useful pieces:

GalaxyConfiguration.config_dir is dirname(config_file) (or . if there is no config file)
GalaxyConfiguration.data_dir is ./data
No assumption that cwd is galaxy_root
No assumption that configs are in ./config/
Sample configs are no longer required
Galaxy now finds templates, static, and tools via symlinks in the galaxy lib dir (but these are not packageable yet)
Sample configs moved to lib/galaxy/config/sample

The exception to the changes above: if running from cwd = galaxy_root, then config_dir and data_dir are <cwd>/config and <cwd>/database as before.

Updated TODO list for the next PR

Finish the metapackage
Update galaxy-config for YAML configs
Add galaxy-uwsgi or galaxy -m uwsgi
Package static/
Other directories: external_services/, tool-data/, ...
Create a galaxy-set-metadata command to replace the tempfile method currently used
Move galaxy-manage-db to galaxy-app after package-ify top-level Python files in lib/galaxy. #8293 I think
Fix tool migrations
Other things I'm not thinking of

Original PR

Building on #428, this attempts to make Galaxy installable via the standard Pythonic mechanism. To use:

Make a virtualenv and activate it
PYTHONPATH= pip install --upgrade pip wheel
make wheels
PYTHONPATH= pip install -i https://wheels.galaxyproject.org/simple/ 'dist/galaxy-16.7.dev0-py2-none-any.whl[postgresql]'
Create a directory somewhere, copy config/galaxy.ini.sample to it as galaxy.ini
galaxy-paster serve galaxy.ini

Alternatively:

Make a virtualenv and activate it
PYTHONPATH= pip install --upgrade pip
make client
PYTHONPATH= pip install -i https://wheels.galaxyproject.org/simple '.[postgresql]' (add -e for editable/develop mode)
As above

TODO list

mr-c · 2015-10-15T09:36:00Z

💯 !!

martenson · 2015-10-15T16:36:05Z

api tests on Jenkins report

Traceback (most recent call last):
  File "./scripts/manage_db.py", line 11, in <module>
    from migrate.versioning.shell import main
ImportError: No module named migrate.versioning.shell

martenson · 2015-10-19T16:07:12Z

this needs to be rebased or dev merged into it before jenkins can test

natefoo · 2015-10-20T14:10:23Z

@martenson Waiting for the tests to pass on #428 first.

jmchilton · 2016-05-16T19:52:43Z

setup.py

+if GALAXY_PACKAGE == "galaxy-lib":
+    requirements = []
+    base_packages = [
+        'galaxy.datatypes',


Datatypes depends on models so it shouldn't live in galaxy-lib in my opinion.

jmchilton · 2016-05-16T20:00:39Z

What made you decide what to put in galaxy-app and what to put into galaxy? I feel like we should just figure out galaxy-lib first - and probably after this effort? After that I'd like to aim toward a design of:

galaxy-lib (minimal required dependencies)
galaxy-models (depends on sqlalchemy and galaxy-lib)
galaxy-app (galaxy-models but no web stuff - someday it would be nice to run a webless galaxy process - e.g. job or workflow handlers using just this)
galaxy-web (galaxy-app + controllers)

I would start from the bottom and introduce one or so a year as we refactor around these new distinctions? There are a think both practical and abstract design goals with each of these distinctions that I could go into in depth. None of this feels like something that should be rushed though.

natefoo · 2016-05-17T15:21:04Z

What made you decide what to put in galaxy-app and what to put into galaxy?

"This seems like a web package", "This doesn't". Which is why it's almost certainly wrong.

So for now perhaps we should just have galaxy-lib and galaxy.

nsoranzo · 2016-05-17T20:32:48Z

It would be nice to make the final tool migration (xref. #636) before this PR is merged.

natefoo · 2016-05-19T18:58:54Z

Per conversation with @jmchilton we are going to add a config_dir option to galaxy.ini so that we can stop assuming the location of everything is relative to the current working directory. Is there a sensible default for this? I suggest the directory containing galaxy.ini.

if you install it and then run `galaxy` from the root of a Galaxy clone, after installing all the dependencies (see the end of ./scripts/common_startup.sh), it should start.

another story), fix classifiers.

tool-shed packages (see galaxyproject#1396). Lots of path fixing to be done still.

wheels and packages.

"installed Galaxy" world is not the current working directory.

live in the `galaxy` package. Install deps at install time.

lib/galaxy/config/sample/

Fix comparisons of relative and absolute paths like the one described at galaxyproject#921 (comment)

lib/galaxy/tools/data_manager/manager.py

nsoranzo · 2019-08-08T14:18:41Z

lib/galaxy/config/__init__.py

+        return os.path.join(self.sample_config_dir, path)
+
+    def _parse_config_file_options(self, defaults, listify_defaults, config_kwargs):
+        for var, defaults in defaults.items():


Probably it's not a bug, but reusing defaults variable name as loop variable is surely confusing.

Yes!! I noticed it and hated it on first site.. I'll submit a quick PR (didn't want to trigger a code review over such a minor thing - but since you mention this.. so thanks :)

@ic4f thanks!

@nsoranzo How about defaults >> vals?

for var, vals in defaults.items(): if config_kwargs.get(var, None) is not None: path = config_kwargs.get(var) setattr(self, var + '_set', True) else: for val in vals: if os.path.exists(resolve_path(val, self.root)): path = val break else: path = vals[-1] setattr(self, var + '_set', False) setattr(self, var, resolve_path(path, self.root))

Maybe values/value to make it a bit more different from var.

nsoranzo · 2019-08-08T14:29:09Z

lib/galaxy/config/__init__.py

+                setattr(self, var + '_set', False)
+            setattr(self, var, resolve_path(path, self.root))
+
+        for var, defaults in listify_defaults.items():


martenson · 2019-08-27T21:09:55Z

lib/galaxy/tools/toolbox/base.py

+        try:
+            tool_conf_source = get_toolbox_parser(config_filename)
+        except (OSError, IOError) as exc:
+            for opt in ('shed_tool_conf', 'migrated_tools_config'):


fresh on dev this is called twice with config_filename=/Users/marten/devel/git/temp/galaxy/config/migrated_tools_conf.xml but once it has getattr(self.app.config, opt) equal to /Users/marten/devel/git/temp/galaxy/config/shed_tool_conf.xml.

so the _dynamic_tool_confs end up being

[{ 'tool_path': '/Users/marten/devel/git/temp/galaxy/database/shed_tools', 'create': '<?xml version="1.0"?>\n<toolbox tool_path="/Users/marten/devel/git/temp/galaxy/database/shed_tools">\n</toolbox>\n', 'config_filename': '/Users/marten/devel/git/temp/galaxy/config/migrated_tools_conf.xml', 'config_elems': []}]

which seems wrong?

Can you create an issue and assign me?

The issue is here: #8484 and you are assigned to it.

Ah, didn't realize this was the same thing, thanks.

for config files. Broken in galaxyproject#921 . For context, see galaxyproject#8639 (comment) Includes also elimination of `resolve_path` function as in galaxyproject#8646

Addresses this comment: galaxyproject#921 (comment)

Backport galaxyproject#8894 Context: galaxyproject#921 (comment)

Fix comparisons of relative and absolute paths like the one described at galaxyproject#921 (comment)

1. Split method body in two 2. Change assignment of config_dir when running from source 3. Ensure value of config_dir is an absolute path I think splitting the method imporoves readability: its functionality can be easily grouped into 2 parts: (1) finding and reading the main config file (I've removed the comment because it was not quite in place and, I think, not essential to understanding the code); and (2) setting paths for core config properties based on part (1). The change to config_dir addresses previous discussions: galaxyproject#8894 (comment) galaxyproject#921 (comment) I could've changed the line in question: galaxyproject#921 (comment) to this: `if self.config_file is None and `config_dir` is not in config_kwargs:` However, in my opinion, such an edit would further obfuscate the logic of assigning that value. The proposed solution, I think, although more verbose, makes the code more straightforward: it is immediately clear what values are assigned under what circumstances. Finally, in the previous version `config_dir` can be a relative path if `config_file` exists, but `config_dir` is set by the user (to a relative path). That, I believe, was a bug.

natefoo added kind/enhancement wip labels Oct 15, 2015

natefoo mentioned this pull request Oct 29, 2015

Merge changes to common modules shared between Galaxy, Planemo, and Pulsar. #1005

Merged

This was referenced Nov 20, 2015

Improve wheel/venv-related flags to common_startup.sh #1150

Closed

Support pip install galaxy with dependencies #1152

Open

natefoo mentioned this pull request Jan 11, 2016

Distribute Galaxy packages #1472

Open

natefoo force-pushed the installable branch from eed66e8 to 8c9642b Compare May 13, 2016 18:14

jmchilton reviewed May 16, 2016
View reviewed changes

natefoo force-pushed the installable branch 3 times, most recently from 99faf51 to 88c01d7 Compare June 16, 2016 19:23

natefoo force-pushed the installable branch from 9041218 to e5b8f1f Compare July 5, 2016 21:45

natefoo added 10 commits July 7, 2016 13:49

Make Galaxy installable the Pythonic way via setup.py. Incomplete, but

1e312fa

if you install it and then run `galaxy` from the root of a Galaxy clone, after installing all the dependencies (see the end of ./scripts/common_startup.sh), it should start.

Add some missing package data (although whether Galaxy can find it is

362b27e

another story), fix classifiers.

Initial work toward separate galaxy-lib, galaxy-app, galaxy, and

cfc3bee

tool-shed packages (see galaxyproject#1396). Lots of path fixing to be done still.

Move tools, images, and templates in to lib/galaxy/ for inclusion in

77c2d15

wheels and packages.

Update client build for moved static directories

e67e7fd

Improvements to distribution building

787b4d4

Fixes for paths expected to be relative to "galaxy root", which in the

6518e64

"installed Galaxy" world is not the current working directory.

Remove the galaxy-app package for now, and PEP 8 fixes

fbb6eea

Remove the namespacing of galaxy packages for now, everything will just

8772297

live in the `galaxy` package. Install deps at install time.

Move some things from scripts/ to lib/ and some sample configs to

42cf61f

lib/galaxy/config/sample/

nsoranzo added a commit to nsoranzo/galaxy that referenced this pull request Jul 17, 2019

Make self.root an absolute path

397be57

Fix comparisons of relative and absolute paths like the one described at galaxyproject#921 (comment)

nsoranzo mentioned this pull request Jul 17, 2019

Startup fixes #8346

Merged

natefoo mentioned this pull request Aug 7, 2019

Fix config file finding for the manage_db script #8431

Merged

jdavcs mentioned this pull request Aug 8, 2019

Do not override None with empty string galaxyproject/planemo#950

Merged

nsoranzo reviewed Aug 8, 2019

View reviewed changes

lib/galaxy/tools/data_manager/manager.py Show resolved Hide resolved

nsoranzo reviewed Aug 8, 2019

View reviewed changes

jdavcs mentioned this pull request Aug 8, 2019

Improve variable naming #8433

Merged

jmchilton mentioned this pull request Aug 9, 2019

Mutable Configs Broken in Dev (Installable Galaxy) galaxyproject/ansible-galaxy#81

Closed

blankenberg mentioned this pull request Aug 20, 2019

Cannot install from Toolshed in fresh checkout #8484

Closed

jmchilton mentioned this pull request Aug 24, 2019

Fix default job metrics in dev/. #8510

Merged

martenson reviewed Aug 27, 2019

View reviewed changes

blankenberg mentioned this pull request Aug 28, 2019

Default tool data tables are not loaded #8534

Closed

natefoo mentioned this pull request Sep 10, 2019

[19.09] Fix autocreation of shed_*_conf.xml #8610

Merged

nsoranzo mentioned this pull request Sep 18, 2019

Modify schema defaults and their loading #8639

Merged

jdavcs added the area/configuration Galaxy's configuration system label Sep 19, 2019

nsoranzo mentioned this pull request Sep 20, 2019

[19.09] Restore resolution of relative paths w.r.t. Galaxy root #8687

Merged

natefoo mentioned this pull request Oct 15, 2019

Galaxy is reading from data_manager_conf.xml.sample #8811

Closed

nsoranzo mentioned this pull request Oct 16, 2019

[19.09] Fix workflow scheduler conf default path. #8812

Merged

jdavcs added a commit to jdavcs/galaxy that referenced this pull request Oct 28, 2019

If config_file is None, use cwd, not its parent

8d1ecc8

Addresses this comment: galaxyproject#921 (comment)

jdavcs mentioned this pull request Oct 28, 2019

If config_file is None, use cwd, not its parent #8894

Merged

jdavcs added a commit to jdavcs/galaxy that referenced this pull request Oct 28, 2019

If config_file is None, use cwd, not its parent

bc611c1

Backport galaxyproject#8894 Context: galaxyproject#921 (comment)

jdavcs mentioned this pull request Oct 28, 2019

[19.09] If config_file is None, use cwd, not its parent #8896

Merged

HadleyKing pushed a commit to biocompute-objects/galaxy that referenced this pull request Oct 30, 2019

Make self.root an absolute path

7ee0d45

Fix comparisons of relative and absolute paths like the one described at galaxyproject#921 (comment)

jdavcs mentioned this pull request Oct 30, 2019

Refactor _set_config_base; fix config_dir value #8915

Merged

natefoo mentioned this pull request Apr 2, 2021

Fixes for installed Galaxy #11779

Merged

9 tasks

natefoo mentioned this pull request May 6, 2025

[24.1] Add galaxy metapackage for installing the latest stable version of Galaxy with pip install galaxy #19051

Closed

2 tasks

Conversation

natefoo commented Oct 15, 2015 • edited by nsoranzo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

2019 Update

Updated TODO list for the next PR

Original PR

TODO list

Uh oh!

mr-c commented Oct 15, 2015

Uh oh!

martenson commented Oct 15, 2015

Uh oh!

martenson commented Oct 19, 2015

Uh oh!

natefoo commented Oct 20, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmchilton commented May 16, 2016

Uh oh!

natefoo commented May 17, 2016

Uh oh!

nsoranzo commented May 17, 2016

Uh oh!

natefoo commented May 19, 2016

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martenson Aug 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

natefoo commented Oct 15, 2015 •

edited by nsoranzo

Loading

martenson Aug 27, 2019 •

edited

Loading