Skip to content

[wip] Dynamic proxy using uWSGI. #2385

Closed
jxtx wants to merge 2 commits intogalaxyproject:devfrom
jxtx:uwsgi-dynamic-proxy
Closed

[wip] Dynamic proxy using uWSGI. #2385
jxtx wants to merge 2 commits intogalaxyproject:devfrom
jxtx:uwsgi-dynamic-proxy

Conversation

@jxtx
Copy link
Contributor

@jxtx jxtx commented May 19, 2016

A dynamic proxy implementation using uWSGI. A RPC function written in python is used to map the session to the appropriate backend.

Why?

  1. We use uwsgi for Galaxy main. With this configuration we can proxy directly to interactive environments, no double proxy needed.
  2. We've talked about making uwsgi the default webserver for all Galaxy's. If we do that we can proxy directly without needing additional processes or ports.
  3. uwsgi is fast, flexible, and well tested.
  4. I was having trouble getting the node based proxy running.

ping @jmchilton @natefoo @bgruening @erasche

jxtx added 2 commits May 19, 2016 18:03
…on could also be used in the same uwsgi process that runs Galaxy
@bgruening
Copy link
Member

Wow this looks very elegant. I had no clue that this is possible.
Do we plan to switch to uwsgi for 16.07?

@hexylena
Copy link
Member

hexylena commented May 19, 2016

@jxtx the golang proxy was intended to solve deployment / config issues, but this is very interesting as well. We have some plans for the GIE stuff that you should be aware of while adding another proxy in the mix, namely #2085.

  1. We use uwsgi for Galaxy main. With this configuration we can proxy directly to interactive environments, no double proxy needed.
  2. We've talked about making uwsgi the default webserver for all Galaxy's. If we do that we can proxy directly without needing additional processes or ports.

I'm not seeing 1/2 in the current code? Do you have plans here?

  1. uwsgi is fast, flexible, and well tested.

No contest.

  1. I was having trouble getting the node based proxy running.

Yep. Huge part of my motivation for golang proxy which was a fast, efficient, single binary to drop in and provide GIE proxying.

@jxtx
Copy link
Contributor Author

jxtx commented May 20, 2016

@bgruening I don't have the code to make a case for it yet, but I think switching to uwsgi makes a lot of sense. It would give us a way to start introducing websockets into Galaxy itself.

@jxtx
Copy link
Contributor Author

jxtx commented May 20, 2016

@erasche

the golang proxy was intended to solve deployment / config issues

Yes, I had read that thread. I thought I would give this a try and it worked better than I expected. I appreciate what you've done with the golang proxy, but I have some concerns similar to what @jmchilton expressed in that issue. I'm not fond of golang and if we don't need to introduce it into Galaxy I would prefer not to. This solution is much simpler than either of the existing proxies in terms of code and configuration, but leverages uwsgi so it has a lot of flexibility. I think this may be a better way to go for the way to do dynamic proxies in Galaxy.

We use uwsgi for Galaxy main. With this configuration we can proxy directly to interactive environments, no double proxy needed.
We've talked about making uwsgi the default webserver for all Galaxy's. If we do that we can proxy directly without needing additional processes or ports.
I'm not seeing 1/2 in the current code? Do you have plans here?

So 1 is just configuration, no code to show. Add something similar to what is in uswgi_proxy.ini to your uwsgi config for Galaxy, set Galaxy not to manage the proxy, and things will just work.

For 2 I don't have code yet, but I'll try to get something together.

@hexylena
Copy link
Member

Yes, I had read that thread. I thought I would give this a try and it worked better than I expected. I appreciate what you've done with the golang proxy, but I have some concerns similar to what @jmchilton expressed in that issue. I'm not fond of golang and if we don't need to introduce it into Galaxy I would prefer not to. This solution is much simpler than either of the existing proxies in terms of code and configuration, but leverages uwsgi so it has a lot of flexibility. I think this may be a better way to go for the way to do dynamic proxies in Galaxy.

I don't disagree with any of this. I was just using the tools I knew were available at the time. When we were looking at moving container launching + killing into the proxy, it made a lot more sense. Docker is written in it, a proxy written in it exposing an HTTP api wouldn't be a bad thing.

I'm 100% on board with scrapping my work if it's going to be as simple as this leads me to believe. :)

So 1 is just configuration, no code to show

I was referring more to

With this configuration we can proxy directly to interactive environments, no double proxy needed

since you were launching a separate process, hence grouping it with 2. That's all.

Thanks for the feedback!

@jxtx jxtx mentioned this pull request May 20, 2016
2 tasks
@jmchilton
Copy link
Member

Thanks for starting the process on this @jxtx - I did look at uwsgi (and a few other Python options) for a long time before starting work on the javascript solution but I couldn't get it work. If uwsgi provides a path forward that allows us to just dynamically manage proxies from within the Galaxy process - so many limitations or complications due to configuration options, cookies, timeouts, etc... would be minimized to an amazing degree - but even if it is just a separate Python process that still makes dependencies easier at least and this implementation is obviously very clean.

@jxtx
Copy link
Contributor Author

jxtx commented May 25, 2016

Thanks @jmchilton. My plan is to propose making uwsgi the default web process, and then implement proxying there. What I'm not sure about is whether this PR should be merged or if we should just wait for than and then get rid of proxy entirely.

@bgruening
Copy link
Member

@jxtx I think it's fine to merge this if the intention is to get uwsgi as the default web process for the next release. This would help testing the proxy for those that have uwsgi already running.

@natefoo natefoo removed this from the 17.09 milestone Sep 7, 2017
@natefoo natefoo self-assigned this Sep 7, 2017
@martenson
Copy link
Member

This is marked as WIP so I am pushing it to 18.05.

@martenson martenson modified the milestones: 18.01, 18.05 Jan 2, 2018
@jmchilton jmchilton modified the milestones: 18.05, 18.09 Apr 19, 2018
@dannon dannon modified the milestones: 18.09, 19.01 Sep 6, 2018
@jmchilton jmchilton modified the milestones: 19.01, 19.05 Dec 12, 2018
@jmchilton jmchilton modified the milestones: 19.05, 19.09 Apr 9, 2019
@jmchilton jmchilton modified the milestones: 19.09, 20.01 Aug 22, 2019
@mvdbeek mvdbeek modified the milestones: 20.01, 20.05 Dec 17, 2019
@mvdbeek mvdbeek modified the milestones: 20.05, 20.09 Apr 20, 2020
@mvdbeek mvdbeek removed this from the 20.09 milestone Sep 16, 2020
@mvdbeek mvdbeek closed this Apr 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants