Skip to content

build abi3 wheels#162

Merged
sethmlarson merged 5 commits intopython-hyper:masterfrom
asottile:abi3
Nov 3, 2020
Merged

build abi3 wheels#162
sethmlarson merged 5 commits intopython-hyper:masterfrom
asottile:abi3

Conversation

@asottile
Copy link
Copy Markdown
Contributor

@asottile asottile commented Nov 3, 2020

sorry for the drive by, noticed this tweet and figured I could save you some future pain :)

cffi natively builds abi3-tagged .so files so this just extends it to the wheels

the neat thing about abi3 is it means you don't need to:

  • produce wheels for every version of cpython 3.x
  • produce new wheels when a new version of python comes out

this drastically reduces the pain when a new version is released as it can just reuse the existing wheels

you'll probably want to change whatever build infra you have to just build the oldest version of cpython you support

my PR on this is actually a day early, I've got a video about the specifics of this technique going up tomorrow on my youtube channel

here's an example:

$ virtualenv venv -ppython3.6
created virtual environment CPython3.6.12.final.0-64 in 238ms
  creator CPython3Posix(dest=/tmp/brotlipy/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/asottile/.local/share/virtualenv)
    added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
$ ./venv/bin/pip wheel . --no-deps
Processing /tmp/brotlipy
Building wheels for collected packages: brotlipy
  Building wheel for brotlipy (setup.py) ... done
  Created wheel for brotlipy: filename=brotlipy-0.7.0-cp36-abi3-linux_x86_64.whl size=1007729 sha256=c56ce8972b973afb724300ed5f416efe04dde4bf167fe4cc8614d2c18720d7f4
  Stored in directory: /tmp/pip-ephem-wheel-cache-tnu52eiq/wheels/f2/e8/0b/cf0d3ca022916c8d034f0d83ed5d7b567f189a7a19de9853f5
Successfully built brotlipy
$ virtualenv venv39 -ppython3.9
created virtual environment CPython3.9.0.final.0-64 in 1241ms
  creator CPython3Posix(dest=/tmp/brotlipy/venv39, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/asottile/.local/share/virtualenv)
    added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
$ venv39/bin/pip install brotlipy-0.7.0-cp36-abi3-linux_x86_64.whl 
Processing ./brotlipy-0.7.0-cp36-abi3-linux_x86_64.whl
Collecting cffi>=1.0.0
  Downloading cffi-1.14.3-cp39-cp39-manylinux1_x86_64.whl (405 kB)
     |████████████████████████████████| 405 kB 2.3 MB/s 
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Installing collected packages: pycparser, cffi, brotlipy
Successfully installed brotlipy-0.7.0 cffi-1.14.3 pycparser-2.20
$ venv39/bin/python -c 'import brotli'
$

for an example of another package which does this, check out https://github.com/asottile/onigurumacffi

@sethmlarson
Copy link
Copy Markdown
Member

sethmlarson commented Nov 3, 2020

A very welcome drive-by, thank you :) My knowledge of binary wheels and CFFI is limited, so your expertise is great here!

I'm currently migrating the test suite off of Travis+AppVeyor to GHA here #161 so that's why the CI likely wasn't working before.

@asottile
Copy link
Copy Markdown
Contributor Author

asottile commented Nov 3, 2020 •
edited
Loading

ah I see, master is broken, I thought I broke it but I guess I was just confused

sethmlarson
sethmlarson reviewed Nov 3, 2020
View reviewed changes
setup.py
else:
class BDistWheel(wheel.bdist_wheel.bdist_wheel):
def finalize_options(self):
self.py_limited_api = "cp3{}".format(sys.version_info[1])
Copy link
Copy Markdown
Member

@sethmlarson sethmlarson Nov 3, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this work in 3.10 as well? Not sure where things are at wrt 310 vs 3_10?

Copy link
Copy Markdown
Contributor Author

@asottile asottile Nov 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure either, I don't know if it has been decided yet. wheel and pip currently disagree on what it should be

Copy link
Copy Markdown
Contributor Author

@asottile asottile Nov 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but it doesn't really matter until python3.10 is your minimum supported version

sethmlarson
sethmlarson approved these changes Nov 3, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@sethmlarson sethmlarson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just the one outstanding question from me.

@sethmlarson sethmlarson merged commit 9b27f5d into python-hyper:master Nov 3, 2020
@sethmlarson
Copy link
Copy Markdown
Member

sethmlarson commented Nov 3, 2020

Thanks!

@asottile asottile deleted the abi3 branch November 3, 2020 22:58
@nateprewitt nateprewitt mentioned this pull request Sep 22, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sethmlarson sethmlarson sethmlarson approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@asottile @sethmlarson