Skip to content

Add new versions of several Python packages#3195

Merged
tgamblin merged 1 commit intospack:developfrom
adamjstewart:features/new-python-versions
Feb 20, 2017
Merged

Add new versions of several Python packages#3195
tgamblin merged 1 commit intospack:developfrom
adamjstewart:features/new-python-versions

Conversation

@adamjstewart
Copy link
Copy Markdown
Member

Many of these come from #2869. I figured it would be easier to review #2869 with unrelated changes in a separate PR.

url = "https://pypi.io/packages/source/n/numpy/numpy-1.9.1.tar.gz"

version('1.12.0', '33e5a84579f31829bbbba084fe0a4300',
url="https://pypi.io/packages/source/n/numpy/numpy-1.12.0.zip")
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest numpy release only comes with .zip files. I'm not sure if this will always be the case from now on, but do we want to switch all of the numpy URLs (and checksums) to .zip?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the extrapolator (at least my one) will base the URL on the most recent URL in the list. So maybe there is no need? It's generally good not to do this too frequently, as it means people will need to re-download (and maybe re-sync mirrors) but I don't have a problem with doing it every once in a while for consistency.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll wait for one more .zip exclusive release before I make the call and switch it all to .zip. Good point about making people re-download/re-sync mirrors, I didn't think of that.

I believe the extrapolator (at least my one) will base the URL on the most recent URL in the list.

See #2737, which is actually explained in #2725. Basically, if I add a newer version to this package and don't specify a URL, it will extrapolate from the url for 1.12.0, not the package-level url. This might trip up new users. It sure tripped me up!

version('25.2.0', 'a0dbb65889c46214c691f6c516cf959c',
url="https://pypi.python.org/packages/9f/32/81c324675725d78e7f6da777483a3453611a427db0145dfb878940469692/setuptools-25.2.0.tar.gz")
version('34.2.0', '41b630da4ea6cfa5894d9eb3142922be',
url="https://pypi.io/packages/source/s/setuptools/setuptools-34.2.0.zip")
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest release only comes in a zipball. Do we want to convert all of the setuptools URLs (and checksums) to .zip?

Copy link
Copy Markdown
Member Author

@adamjstewart adamjstewart Feb 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In an interesting turn of events, the latest version of py-setuptools now depends on py-six, py-packaging, and py-appdirs. The latter 2 aren't in Spack yet, but the first definitely depends on py-setuptools. Not sure what the best course of action here is...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

py-six and py-packaging can both be bootstrapped by using distutils.core instead of setuptools, but py-appdirs has a hard setuptools import. Let me contact the developers and see what is up.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed an issue: pypa/setuptools#980

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pretty sure this is actually a py-appdirs problem. wait for the setuptools people to get back to you, but my understanding is by them depending on appdirs, they've implicitly required that appdirs supports distutils. As long as appdirs changes their setup.py to support this, it should be solved?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's the way things will have to end up going. py-six and py-packaging have that exact same try-except import statement. If the setuptools devs don't get back to me, I'll reach out to appdirs and see what they think. We can add a patch for now that replaces the import with a try-except import. Shouldn't be hard to test.

Alternatively, we can stop updating setuptools, but no one wants that 😆

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol. as if the python packaging system wasn't already confusing enough 🙃

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Started digging through the CHANGES.rst for setuptools:

v34.0.0
-------

* #581: Instead of vendoring the growing list of
  dependencies that Setuptools requires to function,
  Setuptools now requires these dependencies just like
  any other project. Unlike other projects, however,
  Setuptools cannot rely on ``setup_requires`` to
  demand the dependencies it needs to install because
  its own machinery would be necessary to pull those
  dependencies if not present (a bootstrapping problem).
  As a result, Setuptools no longer supports self upgrade or
  installation in the general case. Instead, users are
  directed to use pip to install and upgrade using the
  ``wheel`` distributions of setuptools.

  Users are welcome to contrive other means to install
  or upgrade Setuptools using other means, such as
  pre-installing the Setuptools dependencies with pip
  or a bespoke bootstrap tool, but such usage is not
  recommended and is not supported.

  As discovered in #940, not all versions of pip will
  successfully install Setuptools from its pre-built
  wheel. If you encounter issues with "No module named
  six" or "No module named packaging", especially
  following a line "Running setup.py egg_info for package
  setuptools", then your pip is not new enough.

  There's an additional issue in pip where setuptools
  is upgraded concurrently with other source packages,
  described in pip #4253. The proposed workaround is to
  always upgrade Setuptools first prior to upgrading
  other packages that would upgrade Setuptools.

From what I understand, this means that setuptools doesn't care whether or not we can bootstrap it. I think I can hack appdirs to make the installation work, but let's hope this behavior doesn't continue.

variant('lapack', default=True, description='Build with LAPACK support')

depends_on('python@2.6:2.8,3.2:')
depends_on('py-nose', type='build')
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nose is not needed for installation. It is only used for testing purposes. We can add this back as a test deptype after #1279 and #2869 are complete.

version('0.15.0', '639112f077f0aeb6d80718dc5019dc7a')

depends_on('python@2.6:2.8,3.2:')
depends_on('py-nose', type='build')
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as for numpy, nose is not needed or used in any way during installation.

depends_on('python@2.6:2.8,3.2:')
depends_on('py-nose', type='build')
# Known not to work with 2.23, 2.25
depends_on('binutils@2.26:', type='build')
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't a dependency of scipy any more than curl or gcc are. Removed as suggested in http://spack.readthedocs.io/en/latest/getting_started.html#binutils

@tgamblin tgamblin merged commit 4de85ab into spack:develop Feb 20, 2017
@adamjstewart adamjstewart added WIP and removed WIP labels Feb 20, 2017
@adamjstewart
Copy link
Copy Markdown
Member Author

@tgamblin Oops, this wasn't ready. The latest version of setuptools requires additional dependencies (which depend on setuptools of course). Let me put together a quick fix.

@tgamblin
Copy link
Copy Markdown
Member

@adamjstewart Ok! Thanks!

@adamjstewart adamjstewart deleted the features/new-python-versions branch March 24, 2017 22:10
diaena pushed a commit to diaena/spack that referenced this pull request May 26, 2017
xavierandrade pushed a commit to xavierandrade/spack that referenced this pull request Jun 16, 2017
healther pushed a commit to electronicvisions/spack that referenced this pull request Jul 26, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants