Skip to content

New uswgi build#114

Closed
bgruening wants to merge 6 commits intomasterfrom
uwsgi
Closed

New uswgi build#114
bgruening wants to merge 6 commits intomasterfrom
uwsgi

Conversation

@bgruening
Copy link
Member

Try to push #95 forward.

Talking to many people at the ELIXIR conference many do not use uwsgi and seems to have problems with performance. We should really do a better shop in setting our defaults and leverage the knowledge from @natefoo and Co.

ping @jxtx

@bgruening
Copy link
Member Author

Despite the fact that starforge is not using the latest starforge code, it is unable to determine root directory for whatever reason :(

@natefoo
Copy link
Member

natefoo commented Nov 15, 2016

@bgruening any hints on what the issues folks were encountering with uWSGI? There are two issues I can think of:

  1. Older (pre-16.07) releases of Galaxy had threads starting before uWSGI forking, which could cause workers to deadlock and require a kill -9.
  2. If you don't configure dedicated job handlers, uWSGI will attempt to run jobs and create chaos.

I'll get the Starforge build images updated ASAP, #109 needs to be addressed.

@natefoo
Copy link
Member

natefoo commented Nov 18, 2016

@jxtx: @nsoranzo brought the name change up in galaxyproject/galaxy#3179 and I can no longer remember exactly why this was done. My recollection is because you're building it as an extension class rather than the standard manner, so if someone got uWSGI from PyPI instead of us, it wouldn't work with our out-of-the-box config. Can you comment?

@jxtx
Copy link
Contributor

jxtx commented Nov 18, 2016

Because there are already uwsgi wheels on pypi, and this approach is
completely incompatible with them.

On Fri, Nov 18, 2016 at 11:33 AM Nate Coraor notifications@github.com
wrote:

@jxtx https://github.com/jxtx: @nsoranzo https://github.com/nsoranzo
brought the name change up in galaxyproject/galaxy#3179
galaxyproject/galaxy#3179 and I can no longer
remember why this was done. Can you comment?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#114 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAE4ZUtqBysGm0gNsfambrv2_7LEDkOoks5q_dNigaJpZM4KkFGY
.

@natefoo
Copy link
Member

natefoo commented Nov 18, 2016

Yeah, ok, that's what I figured. If we used the same package name as upstream, would it be possible to support both styles in our out-of-the-box config? i.e. if the extension class build is available, use it, but if not, use command-line uwsgi?

Or was the problem not with starting the server, but starting the embedded server for functional tests?

@jxtx
Copy link
Contributor

jxtx commented Nov 18, 2016

The latter is where we stalled.
On Fri, Nov 18, 2016 at 11:41 AM Nate Coraor notifications@github.com
wrote:

Yeah, ok, that's what I figured. If we used the same package name as
upstream, would it be possible to support both styles in our out-of-the-box
config? i.e. if the extension class build is available, use it, but if not,
use command-line uwsgi?

Or was the problem not with starting the server, but starting the embedded
server for functional tests?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#114 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAE4Zfs0i149__NQK4oHNdUgMxbkuQLRks5q_dUfgaJpZM4KkFGY
.

@natefoo
Copy link
Member

natefoo commented Nov 18, 2016

Might want to support the former then because in the majority of cases, people installing won't care about running tests but may want to pip install w/o using our wheels server (or more likely, forget to use our wheels server).

@jxtx
Copy link
Contributor

jxtx commented Nov 18, 2016

What I mean is that the former already worked. The latter was the problem.
On Fri, Nov 18, 2016 at 12:00 PM Nate Coraor notifications@github.com
wrote:

Might want to support the former then because in the majority of cases,
people installing won't care about running tests but may want to pip
install w/o using our wheels server (or more likely, forget to use our
wheels server).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#114 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAE4ZaO6EAj49jRhlDblAS5XwEBnlTdZks5q_dm0gaJpZM4KkFGY
.

@natefoo
Copy link
Member

natefoo commented Nov 18, 2016

Right, two things:

  1. Does our default config work with both?
  2. This implies that perhaps we should keep the package name the same.

@jxtx
Copy link
Contributor

jxtx commented Nov 18, 2016

The config should work with both, but how you run it differs. I believe I
worked around this in run.sh so both would work. The name really should be
different.

-- jt

On Fri, Nov 18, 2016 at 12:12 PM, Nate Coraor notifications@github.com
wrote:

Right, two things:

  1. Does our default config work with both?
  2. This implies that perhaps we should keep the package name the same.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#114 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAE4ZduiPmYOe8cjcbrjxUmooBVJj2Kjks5q_dyEgaJpZM4KkFGY
.

@jmchilton
Copy link
Member

@natefoo I'm happy to help work on this - but I don't understand much of it.

  • I'd say we should call it galaxy-uwsgi instead of pyuwsgi to make it clear this is a local modification and publish it both to wheels.galaxyproject.org and PyPI?
  • I wouldn't hold up anything based on the functional tests yet - testing is still working in Configuration & Server Overhaul - Tool Shed and Reports galaxy#3179. I don't think we need an embedded server for most tests, if some tests do need it at some point - that is fine - some core portion of the core API should not require uWSGI at any point right? Fancy UI stuff like websockets will require it and the real functional web tests will not require an embedded server.
  • I can try uninstall pyuwsgi in Configuration & Server Overhaul - Tool Shed and Reports galaxy#3179 and then retesting everything with a pip install from pypi - is that needed?

@jxtx
Copy link
Contributor

jxtx commented Nov 21, 2016

I'd say we should call it galaxy-uwsgi instead of pyuwsgi to make it clear this is a local modification and publish it both to wheels.galaxyproject.org and PyPI?

I called it pyuwsgi because that is what upstream calls it in their build configuration, see: https://github.com/unbit/uwsgi/blob/master/buildconf/pyuwsgi.ini

I don't think we should be publishing this to pypi. Installing Galaxy from pypi or packages should use upstream uwsgi packages.

I don't think we need an embedded server for most tests

I agree. I think killing the embedded server was the plan to move this forward. However the embedded server was/is still the default.

@natefoo
Copy link
Member

natefoo commented Nov 21, 2016

I can get in touch with the maintainers and discuss wheel options with them, if they're receptive to shipping wheels then we wouldn't need to maintain anything ourselves (once this version of uWSGI is released) and could just use whichever upstream name they used.

natefoo added a commit that referenced this pull request Jan 11, 2017
can still be built, this functionality is now part of the "lionshead"
library and some wheel monkeypatches in starforge.interface.wheel.

Overhaul wheel Docker image building to push more to the Dockerfile to
better use caching during the build process. Also, install Starforge
in docker from the local source rather than PyPI or Github.

Fixes #109, should allow uWSGI to build (related to #114).
natefoo added a commit that referenced this pull request Jan 11, 2017
can still be built, this functionality is now part of the "lionshead"
library and some wheel monkeypatches in starforge.interface.wheel.

Overhaul wheel Docker image building to push more to the Dockerfile to
better use caching during the build process. Also, install Starforge
in docker from the local source rather than PyPI or Github.

Fixes #109, should allow uWSGI to build (related to #114).
@natefoo natefoo mentioned this pull request Jan 11, 2017
@bgruening bgruening closed this Apr 14, 2017
@bgruening bgruening deleted the uwsgi branch April 14, 2017 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants