Configuration & Server Overhaul - Tool Shed and Reports by jmchilton · Pull Request #3179 · galaxyproject/galaxy

jmchilton · 2016-11-16T15:15:32Z

Bring startup scripts and config processing inline between Galaxy, Reports, and the Tool Shed (retry of Bring Galaxy, reports, and shed run scripts in sync. #2837).
Define a schema format and tooling (derived from pykwalify) to define web server and application option documentation and validation.
Update Galaxy common startup functions to allow starting up uwsgi.
Update Galaxy apps to allow YAML-based configuration files.
Switch out-of-the box server for use by tool shed and reports to uwsgi.
Add schema validation for uwsgi options (derived from uwsgi docs).
Add validation/documentation schema for tool shed including
- Utilities to convert of .ini configs to new YAML ones and switch paste to uwsgi (make tool-shed-config-convert and make tool-shed-config-convert-dry-run).
- Build a new YAML sample that targets uWSGI out of the box.
- Utility to validate a tool shed configuration file in the new YAML format (make tool-shed-config-validate).
- Utility to check best practice config option for tool shed (make tool-shed-config-lint).
Add validation/documentation schema for reports including
- Utilities to convert of .ini configs to new YAML ones and switch paste to uWSGI (make reports-config-convert and make reports-config-convert-dry-run).
- Build a new YAML sample that targets uWSGI out of the box.
- Utility to validate a tool shed configuration file in the new YAML format (make reports-config-validate).
- Utility to check best practice config option for reports (make reports-config-lint).
Add reports documentation to the admin section and documentation of the configuration options derived from the schema.

There is no additional duplication between various items (before a .ini.sample had a default and description) now the schema defines these items and everything else (docs, validation, sample YAML) are derived from that. The new validation format should be more usable from within the actual applications themselves to replace existing type conversion and default handling logic - an exciting future direction for this work.

This is hopefully a test bed to bring these concepts quickly to Galaxy - I think all of the infrastructure needed to do this is now readily available after this PR.

This is another attempt at #2866. This extends that work with more unit tests, new linting options, and correctly handles proxy prefix and gzip filters previously expressed in Paste.

natefoo · 2016-11-16T19:39:00Z

Grr Jenkins... I wanted to merge this.

martenson · 2016-11-16T19:45:51Z

@natefoo it targets 17.01, also please let me retest.

nsoranzo · 2016-11-16T20:01:38Z

lib/galaxy/dependencies/pinned-requirements.txt

 mercurial==3.7.3
 numpy==1.9.2
 pycrypto==2.6.1
+pyuwsgi==2.1.dev0+gx1.c6ee1f6


Shouldn't this be named uwsgi as in PyPi? https://pypi.python.org/pypi/uWSGI

I believe the naming difference was necessary to use the custom (sadface) version we had to build to make it manylinux wheel-able. More in galaxyproject/starforge#95 and galaxyproject/starforge#114.

https://github.com/galaxyproject/starforge/pull/114/files#diff-2fbe7e1ff37f22c3de4b47d45e196667R101 is using the uWSGI name, should that be changed then?

Hrm, yeah, possibly. Since @jxtx's original work in galaxyproject/starforge#95 may have been destroyed I'm not 100% clear.

@nsoranzo I appreciate you bringing this and I don't know the answer to the naming question. Given the difficulty of this - do you think resolving that is important prior to merging this? I think we can just create an issue to track this and go with what is here for now - but obviously we will wait if you think we need to.

@jmchilton Not a blocker for me, a tracking issue seems fair!

Not a blocker for me either.

AFAIK the way @jxtx is building it is the only way a wheeled uWSGI will work at all and I believe it also needs to be the development version (2.1). Which means that galaxyproject/starforge#114 needs to have both the name and version updated? Unless the changes needed were merged back to 2.0.

martenson · 2016-12-02T23:25:44Z

closing and reopening to trigger selenium tests

jmchilton · 2016-12-03T02:42:11Z

@galaxybot test this

natefoo · 2016-12-07T20:22:12Z

Ok to merge @jmchilton?

jmchilton · 2016-12-07T20:38:00Z

Of course

martenson · 2016-12-07T20:51:08Z

Am I missing something that has to be done after the conversion to yml for uwsgi to work (reports)?

Activating virtualenv at .venv
[uWSGI] getting YAML configuration from config/reports.yml
*** Starting uWSGI 2.1-dev (64bit) on [Wed Dec  7 15:49:14 2016] ***
compiled with version: 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.72) on 28 June 2016 12:17:28
os: Darwin-15.6.0 Darwin Kernel Version 15.6.0: Mon Aug 29 20:21:34 PDT 2016; root:xnu-3248.60.11~1/RELEASE_X86_64
nodename: martenson.bx.psu.edu
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /Users/marten/devel/git/galaxy
detected binary path: /Users/marten/devel/git/galaxy/.venv/bin/python
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 4864
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
bind(): Can't assign requested address [core/socket.c line 771]

jmchilton · 2016-12-07T21:37:17Z

@martenson Can I see the config file?

martenson · 2016-12-07T21:38:05Z

$cat reports.yml
uwsgi:

  http: '127.0.0.1:9001'
  threads: 10

  http-raw-body: true

  offload-threads: 8

  module: 'galaxy.webapps.reports.buildapp:uwsgi_app()'

reports:

  database_connection: 'postgres://usr:pswd@localhost:5432/galaxy-git'

jmchilton · 2016-12-07T21:46:12Z

@martenson

You already have that address in use?

If not, can you replace http: '127.0.0.1:9001' with http: 'localhost:9001' and let me know if that helps?

martenson · 2016-12-07T21:48:05Z

@jmchilton shouldn't be in use; after the change to localhost it works

jmchilton · 2016-12-07T21:49:35Z

Is 127.0.0.1 not a thing on Macs? I don't get it. @natefoo should this be localhost instead?

martenson · 2016-12-07T22:36:33Z

127.0.0.1 is a thing, I use it (even with Galaxy) normally

- They now uniformly handle server option processing. - Reports and Tool Shed now respect many new and important options such as --skip_venv. - Reports and Tool Shed now do extra checking around Python version that Galaxy does. In addition to improving the reports and tool shed startup scripts and reducing cognative load with respect to option handling, this is an important pre-condition to us being able us to switch on uwsgi by default for all three services simultaneously without a bunch of copy and pasting.

Initial outline of allowing launch via uwsgi.

This has been rebased several times and includes a variety of fixes (some thanks to martenson).

- Implement kwalify schema to validate/define the schema. - Replace ini configuration with YAML (completely backward compatible). - Rebuild a sample configuration from the schema (config/tool_shed.yml.sample).

- Implement kwalify schema to validate/define the schema. - Replace ini configuration with YAML (completely backward compatible). - Rebuild a sample configuration from the schema (config/reports.yml.sample).

- Automatically generate RST docs from config schema. - Describe older functionality as well as new YAML-based configuration. - Replace https://wiki.galaxyproject.org/Admin/UsageReports.

Required for the new out-of-the-box configuration of tool shed and reports.

xref galaxyproject/starforge#110

…alaxyproject.org.

…ckage.

jmchilton · 2017-08-10T17:16:59Z

Rebased... again...

bgruening · 2017-08-10T21:00:16Z

restarted the tests, now everything is green
ping @natefoo and @erasche for review

bgruening · 2017-08-11T15:20:00Z

@jmchilton the only thing I have found so far is a not so nice warning if you only have sample yml files.

make tool-shed-config-lint   
if [ -f .venv/bin/activate ]; then . .venv/bin/activate; fi; python lib/galaxy/webapps/config_manage.py lint tool_shed
WARNING: Path [None] not a file, using sample.

I think this can be improved in a later PR and we should get this in asap. Will merge after dinner if no one beats me to it.

hexylena · 2017-08-11T08:46:54Z

Makefile

+uwsgi-rebuild-validation: ## rebuild uwsgi_config.yml kwalify schema against latest uwsgi master.
+	$(CONFIG_MANAGE) build_uwsgi_yaml
+
+tool-shed-config-validate: ## validate tool shed YAML configuration file


since there are a number of lint / validation could we have a target for all of these? e.g. config-validate: uwsgi-rebuild-vlaidation toolshed-config-validate ...

uwsgi-rebuild-validation isn't something I'd expect deployers to run that is something we build and ship to them. I guess . a config validate that checked Galaxy, Reports, and Tool Shed configs if they are present would be cool though - I'd hope not too many people have tool shed configurations though 😓.

hexylena · 2017-08-11T08:48:12Z

doc/source/admin/reports.rst

+Configuration Options
+----------------------------
+
+.. include:: reports_options.rst


quite like this :D

Want to help me get this done for Galaxy for this release 😄?

bgruening · 2017-08-12T08:36:16Z

I talked to @erasche and his comments are not critical but nice to have. Would be great if this can be added in follow-up PR.
Tracking issue here: #4419

bgruening · 2017-08-12T08:37:02Z

Thanks @jmchilton - this enables some nice additions in the future!

martenson · 2017-08-12T15:26:58Z

Glad this got finally merged, thanks @jmchilton @bgruening @natefoo @erasche 🥇

nsoranzo · 2018-06-01T16:45:50Z

run_tool_shed.sh


-python ./scripts/paster.py serve $TOOL_SHED_CONFIG_FILE --pid-file=tool_shed_webapp.pid --log-file=tool_shed_webapp.log $args
+find_server $TOOL_SHED_CONFIG_FILE
+$run_server $server_args


@martenson @jmchilton After these changes we are setting but not using any more the $args variable, is this breaking the TS bootstrap?

I doubt I tested this so I don't know, no one has complained though 😅. I wouldn't worry too much about that functionality unless someone really intends to use it. I just double checked a fresh Planemo build and its bootstrapped toolshed is working - but it is probably called in a fairly custom way (xref https://github.com/galaxyproject/ansible-galaxy-toolshed/blob/d1355719f431f9228952d57c6c73b47394dba07f/tasks/database.yml#L29).

jmchilton added this to the 17.01 milestone Nov 16, 2016

jmchilton added area/admin area/cleanup area/documentation area/framework area/reports area/scripts area/toolshed kind/enhancement area/testing labels Nov 16, 2016

martenson self-assigned this Nov 16, 2016

jmchilton force-pushed the web_config_overhaul_0 branch from 788a9ed to 01539d6 Compare November 16, 2016 16:18

nsoranzo reviewed Nov 16, 2016

View reviewed changes

natefoo mentioned this pull request Nov 18, 2016

New uswgi build galaxyproject/starforge#114

Closed

martenson closed this Dec 2, 2016

martenson reopened this Dec 2, 2016

jmchilton mentioned this pull request May 8, 2017

New check_galaxy_config command to help upgrade a galaxy instance config galaxyproject/ephemeris#40

Open

jmchilton added 15 commits August 10, 2017 13:14

Centralize launching server for reports and tool shed.

72e5a00

Initial outline of allowing launch via uwsgi.

Update galaxy.util.properties to allow YAML.

1bdb99b

Scripts to manage configuration infrastructure.

8189a1f

This has been rebased several times and includes a variety of fixes (some thanks to martenson).

Add uwsgi_schema.yml.

7973a91

Update tool shed to use newer configuration schema.

90e899d

- Implement kwalify schema to validate/define the schema. - Replace ini configuration with YAML (completely backward compatible). - Rebuild a sample configuration from the schema (config/tool_shed.yml.sample).

Update Galaxy Reports to use newer configuration schema.

102566d

- Implement kwalify schema to validate/define the schema. - Replace ini configuration with YAML (completely backward compatible). - Rebuild a sample configuration from the schema (config/reports.yml.sample).

Add reports documentation to doc/.

50d426e

- Automatically generate RST docs from config schema. - Describe older functionality as well as new YAML-based configuration. - Replace https://wiki.galaxyproject.org/Admin/UsageReports.

Makefile shortcuts for configuration script.

4945f55

Add required dependency on uwsgi.

ef46a60

Required for the new out-of-the-box configuration of tool shed and reports.

Add pykwalify as dependency for the new config validation stuff.

8191fd4

xref galaxyproject/starforge#110

Fix Python 3 linting problem with test_config_manage.py.

ffe295e

Pin version of uWSGI to freshly released 2.0.15 available on wheels.g…

1ce93cc

…alaxyproject.org.

Fix ORM scripts for galaxyproject#3179.

539a62f

Update config overhaul branch for introduction of galaxy.web.stack pa…

98ca8ce

…ckage.

jmchilton force-pushed the web_config_overhaul_0 branch from 1f9f5b7 to 98ca8ce Compare August 10, 2017 17:15

hexylena approved these changes Aug 12, 2017

View reviewed changes

bgruening mentioned this pull request Aug 12, 2017

Adding lint/validation Makefile target #4419

Open

bgruening merged commit 331851b into galaxyproject:dev Aug 12, 2017

natefoo mentioned this pull request Aug 22, 2017

Run Galaxy fully under uWSGI, including job handlers #4475

Merged

14 tasks

jmchilton mentioned this pull request Dec 1, 2017

Convert options file to YAML and use uWSGI by default #5105

Merged

11 tasks

nsoranzo reviewed Jun 1, 2018

View reviewed changes

nsoranzo deleted the web_config_overhaul_0 branch February 6, 2020 20:00

Conversation

jmchilton commented Nov 16, 2016

Uh oh!

natefoo commented Nov 16, 2016

Uh oh!

martenson commented Nov 16, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsoranzo Nov 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martenson commented Dec 2, 2016

Uh oh!

jmchilton commented Dec 3, 2016

Uh oh!

natefoo commented Dec 7, 2016

Uh oh!

jmchilton commented Dec 7, 2016

Uh oh!

martenson commented Dec 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmchilton commented Dec 7, 2016

Uh oh!

martenson commented Dec 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmchilton commented Dec 7, 2016

Uh oh!

martenson commented Dec 7, 2016

Uh oh!

jmchilton commented Dec 7, 2016

Uh oh!

martenson commented Dec 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmchilton commented Aug 10, 2017

Uh oh!

bgruening commented Aug 10, 2017

Uh oh!

bgruening commented Aug 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmchilton Aug 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bgruening commented Aug 12, 2017

Uh oh!

bgruening commented Aug 12, 2017

Uh oh!

martenson commented Aug 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

nsoranzo Nov 21, 2016 •

edited

Loading

martenson commented Dec 7, 2016 •

edited

Loading

martenson commented Dec 7, 2016 •

edited

Loading

martenson commented Dec 7, 2016 •

edited

Loading

jmchilton Aug 12, 2017 •

edited

Loading

martenson commented Aug 12, 2017 •

edited

Loading