Replace eggs with wheels by natefoo · Pull Request #428 · galaxyproject/galaxy

natefoo · 2015-07-05T09:47:33Z

Smash Eggs

More importantly, replace all of our custom egg installation code with a very slightly modified version of pip. The modification is necessary if/until wheel is updated to include support for Linux wheels.

Other changes include

Removal of the awful lib/pkg_resources.py
Relocated the bit of eggs code that checked the Galaxy config to determine what dependencies are needed to a wheels module.
galaxyproject/docker-build has been updated with scripts and images to support building wheels for distribution on 32 and 64 bit Linux.

On virtualenvs

scripts/common_startup.sh will create a virtualenv at $galaxy_root/.venv if one is not already active and wheels will be installed to here. Otherwise, Galaxy will use whatever virtualenv is currently active (in $VIRTUAL_ENV).

Whatever virtualenv Galaxy is started under will be passed to jobs if a virtualenv is not already active in the job environment (and it exists at the same path on the compute resource). You can force jobs to use a different virtualenv using the <env> tag in a job destination, e.g.:

<destination id="foo">
    <env file="/path/to/venv/bin/activate"/>
</destination>

Galaxy needs to become installable in the traditional python setup.py install manner which will change some things about how virtualenvs are handled in the future (tl;dr, we don't need to "handle" them all unless you are running directly from source), but that is the next project. This was all recently done for Pulsar so Galaxy can steal from there. =)

TODO

Questions

Should we pin exact versions of dependencies? I have a strong preference to say no, and as a concession say we should pin to "less than the next major version" e.g. SQLAlchemy<1.1. Although this does present one small problem - we will have to ensure that we always have wheels built for every version published on PyPI (we can certainly automate this) or else pip will bypass the wheels and install from source, unless we use pip install --only-binary. However, if we use --only-binary, then we cannot trivially fall back to compiling from source on platforms that we do not have wheels for. I was won over by the argument that we should pin exact versions ("exact set of versioned dependencies for an exact version of Galaxy), but that we will:
- Fetch new dependency versions as they are released
- Build wheels for them
- Test Galaxy against these new wheel versions
- If the tests pass, automatically issue a PR against Galaxy to bump the version
~~Should the existence of $galaxy_root/.venv take precedence over $VIRTUAL_ENV (since I keep installing all of Galaxy's wheels in my tox venv =P)?~~use --skip-venv if you don't want .venv to take precedence.

wheels.galaxyproject.org

mr-c · 2015-07-05T10:28:56Z

Fantastic work, 👍

dannon · 2015-07-05T12:32:18Z

Regarding dependency pinning, +1 to SQLAlchemy<1.1. We don't want automatic major version updates to happen without testing, but it'd definitely be nice to make minor updates less onerous.

natefoo · 2015-07-05T12:43:26Z

@jxtx converted bx-python back to setuptools, so that one's done.

nsoranzo · 2015-07-05T12:52:29Z

This is going to be awesome!

jmchilton · 2015-07-05T13:05:58Z

135 additions and 5,471 deletions. 😈

natefoo · 2015-07-05T14:21:21Z

OS X wheels built with python.org builds of Python (which are multi-arch Intel builds using the 10.6 SDK) work on all later versions of OS X. I am starting to upload these to https://wheels.galaxyproject.org/

jxtx · 2015-07-05T14:40:45Z

scripts/common_startup.sh

This currently checks out the repository again every time I run it. Changing to pip install -e git+https://github.com/natefoo/pip@linux-wheels#egg=pip allows it just to update.

@jxtx

every time (courtesy @jxtx).

nsoranzo · 2015-07-12T17:53:21Z

Remove all eggs imports and eggs require()s

@natefoo: Will it be possible to change all pkg_resources.require()s to imports?

natefoo · 2015-07-13T19:13:22Z

@nsoranzo In theory it should be possible to simply remove them. They should precede the name they are importing, which should now succeed since everything should be accessible on sys.path from the beginning.

`{eggs,pkg_resources}.require`. Probably broke some stuff where we were trying to be smart about whether a particular egg was available, but Galaxy does at least start.

natefoo · 2015-07-15T16:43:55Z

I thought I wrote this down somewhere but now I can't find it...

The suggestion from @jxtx regarding pinning was to pin to exact versions but to automate checking for minor version updates, automatically build wheels, and then PR with the new versions so that tests are run. If tests pass, then we can merge the PR and have updated versions.

This ensures that each stable release of Galaxy will have an exact set of versioned dependencies.

natefoo · 2015-07-15T16:44:06Z

@martenson Are our modifications to Whoosh forked somewhere?

natefoo · 2015-07-15T18:10:55Z

bx wheel fixed with galaxyproject/starforge@bcd773f

wheels.

natefoo · 2015-07-15T19:39:31Z

Also, if anyone has any ideas on how to automate building wheels on OS X, I'd appreciate them since obviously we can't use docker-build there. @jxtx claims it's possible to virtualize OS X on non-Apple hardware these days?

jxtx · 2015-07-15T19:41:51Z

Definitely possible. Questionably legal. http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/

natefoo · 2015-08-17T22:00:01Z

I've been working on the "figure out how to build psycopg2" option, and the plan that I like the best goes like this:

Using galaxyproject/starforge#27 (and some stuff I still need to push to that PR) it'll be possible to build wheels on many different distros using docker. Right now I plan to build for both 32- and 64-bit Debian 6, 7, and 8, Ubuntu 12.04 and 14.04, and CentOS/RHEL 6 and 7. Also if anyone wants to add SLES/openSUSE support that would be good too. This means we get to dynamically link the vendor-provided libpq rather than our own statically-compiled and probably out-of-date-and-buggy version.

There'll also be a fallback generic linux-x86_64 wheel that will be statically compiled against psycopg2. Maybe.

hexylena · 2015-08-18T00:03:53Z

@natefoo :D

natefoo · 2015-09-17T16:21:48Z

SVGFig is now available here, and versioned tarballs are still accessible (as of now) from Google Code via this wiki page. So I downloaded the upstream 1.1.6 tarball, fixed the version number in setup.py, created an sdist, uploaded that to wheels.galaxyproject.org, and then built a (generic py2) wheel from that in docker-build (and uploaded it to wheels.galaxyproject.org).

natefoo · 2015-09-18T15:47:58Z

@dannon can you re-enlighten me as to what version of sqlalchemy-migrate we need to use? IIRC it was just that we needed to use the version from git as opposed to the published version in PyPI? Because the version in PyPI is only a couple weeks old now so maybe it's good.

whole reason we use a modified pysam is for the index_filename param...

natefoo · 2015-10-22T16:30:02Z

martenson · 2015-10-22T16:38:09Z

damn @natefoo ; I think @jxtx does not approve of that emoticon :D

gratz on the tests though, wheels are coming!

hexylena · 2015-10-22T16:40:44Z

@natefoo I can't find a reaction gif for how excited I am for this. Well done.

natefoo · 2015-10-22T16:47:16Z

So on a scale of -1 to +1, how do people feel about this? =D

jxtx · 2015-10-22T16:50:49Z

martenson · 2015-10-22T16:56:06Z

👍

jmchilton · 2015-10-22T16:59:38Z

This is going to break a lot of stuff, I would suggest we get drunk tonight
and merge this at Zeno's. #mergethebigonesdrunk
On Oct 22, 2015 12:53 PM, "Eric Rasche" notifications@github.com wrote:

@natefoo https://github.com/natefoo I can't find a reaction gif for how
excited I am for this. Well done.

—
Reply to this email directly or view it on GitHub
#428 (comment).

hexylena · 2015-10-22T17:00:00Z

Works for me on:

pre-existing with eggs
intside a venv and not

Doesn't work for me on:

virtualenv using --skip-venv

(asdf)hxr@leda:~/work/galaxy$ sh run.sh --skip-venv
Activating virtualenv at /home/hxr/work/galaxy/.venv
Obtaining pip from git+https://github.com/natefoo/pip@linux-wheels#egg=pip
Updating ./.venv/src/pip clone (to linux-wheels)

that flag seems to only be available on run_tests.sh

Installing own wheels to a venv using pip install...

(asdf)hxr@leda:~/work/galaxy$ pip install --index-url=https://wheels.galxayproject.org/ -r requirements.txt
Downloading/unpacking Cheetah==2.4.4 (from -r requirements.txt (line 2))
Cannot fetch index base URL https://wheels.galxayproject.org/
Could not find any downloads that satisfy the requirement Cheetah==2.4.4 (from -r requirements.txt (line 2))
Cleaning up...
No distributions at all found for Cheetah==2.4.4 (from -r requirements.txt (line 2))
Storing debug log for failure in /home/hxr/.pip/pip.log

Untested:

Non-Debianish Linuxes
Macs of various flavors
Galaxy servers using PostgreSQL on a variety of platforms
Upgrading a CloudMan instance
Using a non-standard Python (compiled from source)
Run Galaxy on your PDP-11

hexylena · 2015-10-22T17:01:03Z

That said, neither of those cases are things which should prevent this merge and can't be solved in the next two months... so 👍

natefoo · 2015-10-22T17:05:18Z

@erasche Why fix later what I can fix now before I forget how everything works?

@jmchilton +:100:

dannon · 2015-10-22T17:05:52Z

Super exciting, but, for me on OSX, upgrading from existing install:

Found existing installation: mercurial 3.5.2
Detected a distutils installed project ('mercurial') which we cannot uninstall. The metadata provided by distutils does not contain a list of files which have been installed, so pip does not know which files to uninstall.
You must give at least one requirement to install (see "pip help install")
Activating virtualenv at /Users/yoplait/work/galaxy/.venv
Traceback (most recent call last):
  File "./scripts/paster.py", line 26, in <module>
    from galaxy.util.pastescript import serve
  File "/Users/yoplait/work/galaxy/lib/galaxy/util/__init__.py", line 36, in <module>
    import docutils.core
ImportError: No module named docutils.core

Wiped out pre-existing .venv, same thing.

natefoo · 2015-10-22T17:07:47Z

@dannon What's your $PYTHONPATH look like? virtualenv-burrito screws everything up.

dannon · 2015-10-22T17:08:55Z

[yoplait@malleus:~/work/galaxy] echo $PYTHONPATH                               smash_eggs ✭
/usr/local/lib/python2.7/site-packages

(brew installed python on El Capitan!)

hexylena · 2015-10-22T17:10:09Z

@natefoo

Why fix later what I can fix now before I forget how everything works?

fair enough. :)

natefoo · 2015-10-22T18:59:12Z

@erasche Well why'd you type wheels.galxay instead of wheels.galaxy... oh... nevermind. (fixed my comment above where I typo'd it). Also, --skip-venv support coming shortly.

@dannon What if you start with PYTHONPATH unset?

If it works, that's gonna be a fun one to deal with. Perhaps we should unset PYTHONPATH in the event that --skip-venv is not used.

hexylena · 2015-10-22T19:01:06Z

@natefoo tested again with a correct URL, works. :)

Edit:

Downloading/unpacking python-lzo==1.8 (from -r requirements.txt (line 10))
  Could not find any downloads that satisfy the requirement python-lzo==1.8 (from -r requirements.txt (line 10))
Cleaning up...
No distributions at all found for python-lzo==1.8 (from -r requirements.txt (line 10))
Storing debug log for failure in /home/hxr/.pip/pip.log

than an editable directly from github. Also, use `set -e` in scripts/common_startup.sh to fail if anything fails, rather than chaining a bunch of things with `&&` but not properly handling failure cases.

dannon · 2015-10-22T20:02:22Z

@natefoo With an unset pythonpath, there is no mercurial issue, but then I hit:

  Could not find a version that satisfies the requirement pysam==0.8.3gx1 (from -r requirements.txt (line 13)) (from versions: 0.8.3+gx1)
No matching distribution found for pysam==0.8.3gx1 (from -r requirements.txt (line 13))
You must give at least one requirement to install (see "pip help install")
Activating virtualenv at /Users/yoplait/work/galaxy/.venv
Traceback (most recent call last):
  File "./scripts/paster.py", line 26, in <module>
    from galaxy.util.pastescript import serve
  File "/Users/yoplait/work/galaxy/lib/galaxy/util/__init__.py", line 36, in <module>
    import docutils.core
ImportError: No module named docutils.core

nturaga · 2015-10-22T21:25:10Z

Activating virtualenv at /Users/nturaga/Documents/PR_testing/galaxy/.venv Ignoring indexes: https://pypi.python.org/simple Requirement already up-to-date: pip in ./.venv/lib/python2.7/site-packages Requirement already satisfied (use --upgrade to upgrade): Cheetah==2.4.4 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 2)) ... serving on http://127.0.0.1:8080

Works for me...

natefoo · 2015-10-22T22:38:10Z

For the record, @erasche's problems were fixed with the addition of installing the Galaxy-modified pip.

provided.

dannon · 2015-10-22T22:44:32Z

+1

Replace eggs with wheels

natefoo added 3 commits July 4, 2015 15:18

WIP: Stop using eggs, start using prebuilt wheels.

633061f

Install modified pip for fetching Linux wheels, fetch wheels from

a24134c

wheels.galaxyproject.org

Remove (dist-)eggs.ini.

0de7451

natefoo added kind/enhancement wip labels Jul 5, 2015

Fix problems in wheels.py found by flake8

3309531

jxtx reviewed Jul 5, 2015
View reviewed changes

natefoo added 2 commits July 5, 2015 10:50

Install modified pip in develop/edit mode so it won't be reinstalled

5501917

every time (courtesy @jxtx).

Add nose to requirements.txt.

e6446b0

natefoo mentioned this pull request Jul 7, 2015

Automatically build and test new versions of wheels #437

Closed

Remove all imports of galaxy.eggs and all calls to

033b1e8

`{eggs,pkg_resources}.require`. Probably broke some stuff where we were trying to be smart about whether a particular egg was available, but Galaxy does at least start.

Updating requirements.txt as I get wheels ready. TODO: automate OS X

431ebf9

wheels.

dannon mentioned this pull request Aug 27, 2015

Galaxy image metadata framework needs enhancements #659

Closed

natefoo added 2 commits October 22, 2015 11:23

Talked to Dan, who corrected me on a few things. Also, forgot that the

048f0be

whole reason we use a modified pysam is for the index_filename param...

Updates to vis data providers for pysam API changes

7e372d3

natefoo added 3 commits October 22, 2015 15:01

Use proper PEP-440 local version identifiers for modified dependencies.

0cbaf3b

Install our modified pip as a wheel from wheels.galaxyproject.org rather

4a5f5f4

than an editable directly from github. Also, use `set -e` in scripts/common_startup.sh to fail if anything fails, rather than chaining a bunch of things with `&&` but not properly handling failure cases.

Pass relevant args from run.sh to common_startup.sh.

7375964

Unset $PYTHONPATH in run.sh and common_startup.sh unless --skip-env is

88683c3

provided.

dannon added a commit that referenced this pull request Oct 22, 2015

Merge pull request #428 from natefoo/smash_eggs

e423ab6

Replace eggs with wheels

dannon merged commit e423ab6 into galaxyproject:dev Oct 22, 2015

peterjc mentioned this pull request Nov 6, 2015

scripts/fetch_eggs.py removed #1030

Closed

nsoranzo mentioned this pull request Nov 27, 2015

Update to Mercurial 3.5 breaks hg push of Tool Shed repositories #1183

Closed

Conversation

natefoo commented Jul 5, 2015

Smash Eggs

Other changes include

On virtualenvs

TODO

Questions

Uh oh!

mr-c commented Jul 5, 2015

Uh oh!

dannon commented Jul 5, 2015

Uh oh!

natefoo commented Jul 5, 2015

Uh oh!

nsoranzo commented Jul 5, 2015

Uh oh!

jmchilton commented Jul 5, 2015

Uh oh!

natefoo commented Jul 5, 2015

Uh oh!

jxtx Jul 5, 2015

Choose a reason for hiding this comment

Uh oh!

nsoranzo commented Jul 12, 2015

Uh oh!

natefoo commented Jul 13, 2015

Uh oh!

natefoo commented Jul 15, 2015

Uh oh!

natefoo commented Jul 15, 2015

Uh oh!

natefoo commented Jul 15, 2015

Uh oh!

natefoo commented Jul 15, 2015

Uh oh!

jxtx commented Jul 15, 2015

Uh oh!

natefoo commented Aug 17, 2015

Uh oh!

hexylena commented Aug 18, 2015

Uh oh!

natefoo commented Sep 17, 2015

Uh oh!

natefoo commented Sep 18, 2015

Uh oh!

natefoo commented Oct 22, 2015

Uh oh!

martenson commented Oct 22, 2015

Uh oh!

hexylena commented Oct 22, 2015

Uh oh!

natefoo commented Oct 22, 2015

Uh oh!

jxtx commented Oct 22, 2015

Uh oh!

martenson commented Oct 22, 2015

Uh oh!

jmchilton commented Oct 22, 2015

Uh oh!

hexylena commented Oct 22, 2015

Uh oh!

hexylena commented Oct 22, 2015

Uh oh!

natefoo commented Oct 22, 2015

Uh oh!

dannon commented Oct 22, 2015

Uh oh!

natefoo commented Oct 22, 2015

Uh oh!

dannon commented Oct 22, 2015

Uh oh!

hexylena commented Oct 22, 2015

Uh oh!

natefoo commented Oct 22, 2015

Uh oh!

hexylena commented Oct 22, 2015

Uh oh!

dannon commented Oct 22, 2015

Uh oh!

nturaga commented Oct 22, 2015