Support CPython 3.11, 3.12, and aarch64 processors#2331
Support CPython 3.11, 3.12, and aarch64 processors#2331maartenbreddels merged 91 commits intovaexio:masterfrom
Conversation
|
Hoi 👋 exciting, will take a look early next week!
that worries me a bit.. :) groeten, Maarten |
|
here are all timings: https://github.com/ddelange/vaex/actions/runs/3965720337/usage depending on how often a month you release vaex, this could eat into the 2k free minutes of GH... as the parallelization is maximised and they're pushed to PyPI as soon as they're built, most of the wheels will be available soon upon release regardless here are all the wheels: distributions.zip |
|
interestingly, that was 8260 minutes ^ apparently that's OK? then I don't understand their explanation 🤔 https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#included-storage-and-minutes |
|
ah there is a fair amount of duplication in that usage table for whatever reason 🤯 |
|
a diff of current PyPI vs the zip above: vaex_core-4.16.1-cp310-cp310-macosx_10_9_x86_64.whl
vaex_core-4.16.1-cp310-cp310-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp310-cp310-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp310-cp310-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp310-cp310-musllinux_1_1_aarch64.whl
vaex_core-4.16.1-cp310-cp310-musllinux_1_1_x86_64.whl
vaex_core-4.16.1-cp310-cp310-win_amd64.whl
+vaex_core-4.16.1-cp311-cp311-macosx_10_9_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-macosx_11_0_arm64.whl
+vaex_core-4.16.1-cp311-cp311-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp311-cp311-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-musllinux_1_1_aarch64.whl
+vaex_core-4.16.1-cp311-cp311-musllinux_1_1_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-win_amd64.whl
vaex_core-4.16.1-cp36-cp36m-macosx_10_9_x86_64.whl
-vaex_core-4.16.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp36-cp36m-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp36-cp36m-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp36-cp36m-musllinux_1_1_aarch64.whl
vaex_core-4.16.1-cp36-cp36m-musllinux_1_1_x86_64.whl
vaex_core-4.16.1-cp36-cp36m-win_amd64.whl
vaex_core-4.16.1-cp37-cp37m-macosx_10_9_x86_64.whl
-vaex_core-4.16.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp37-cp37m-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp37-cp37m-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp37-cp37m-musllinux_1_1_aarch64.whl
vaex_core-4.16.1-cp37-cp37m-musllinux_1_1_x86_64.whl
vaex_core-4.16.1-cp37-cp37m-win_amd64.whl
vaex_core-4.16.1-cp38-cp38-macosx_10_9_x86_64.whl
vaex_core-4.16.1-cp38-cp38-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp38-cp38-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp38-cp38-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp38-cp38-musllinux_1_1_aarch64.whl
vaex_core-4.16.1-cp38-cp38-musllinux_1_1_x86_64.whl
vaex_core-4.16.1-cp38-cp38-win_amd64.whl
vaex_core-4.16.1-cp39-cp39-macosx_10_9_x86_64.whl
vaex_core-4.16.1-cp39-cp39-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp39-cp39-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp39-cp39-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp39-cp39-musllinux_1_1_aarch64.whl
vaex_core-4.16.1-cp39-cp39-musllinux_1_1_x86_64.whl
vaex_core-4.16.1-cp39-cp39-win_amd64.whl |
| namespace std { | ||
| template<> | ||
| struct hash<PyObject*> { | ||
| size_t operator()(const PyObject *const &o) const { | ||
| return PyObject_Hash((PyObject*)o); | ||
| } | ||
| }; | ||
| } |
There was a problem hiding this comment.
This was my best guess...
pybind11 ref https://github.com/pybind/pybind11/blob/769fd3b889fef6cddb060f2a0be26aee62b4da05/include/pybind11/pytypes.h#L859
failure log https://github.com/ddelange/vaex/actions/runs/3965609112/jobs/6795506653#step:6:2110
There was a problem hiding this comment.
@maartenbreddels any thoughts on this (incl me updating the pybind11 submodule)?
packages/vaex-core/setup.py
Outdated
| include_package_data=True, | ||
| ext_modules=([extension_vaexfast] if on_rtd else [extension_vaexfast, extension_strings, extension_superutils, extension_superagg]) if not use_skbuild else [], | ||
| zip_safe=False, | ||
| python_requires=">=3.6", |
There was a problem hiding this comment.
cibuildwheel parses this to determine which wheels to build
beff33a to
7bf26cc
Compare
|
I'm guessing this is blocked by #2339 |
|
Just letting you know i'm very busy and had a vacation. |
|
fwiw there are now third party free minutes on native arm64 machines, to get rid of the slow qemu builds |
|
Could you try rebasing this? |
|
@maartenbreddels already merged in master 👍 |
|
|
Yeah, a bug/artifact or our release script. Should be good now. |
|
hoi @maartenbreddels 👋 I pulled master and fixed merge conflicts, but it looks like CI is still not very happy. Seeing errors like hdf file missing on disk, and Do you think it might be related to this PR? |
|
Just wondering here on the Python packaging. Python 3.6 and 3.7 are now deprecated on the other hand we can bump to 3.10 and 3.11? |
|
Do we have any updates on this MR? |
|
HI @maartenbreddels 👋 Was your s3 account deleted by any chance? raises 
|
5680eb9 to
2136629
Compare
|
that did it. we're green 🏁 fwiw, pandas deprecated python 3.8 in August 2023 (v2.1.0) |
|
Whow, amazing. You did most of the work @ddelange thanks a lot for the work and support ! I'll set up the trusted publisher (probably next week) and see if we can make a release 🥳 |
|
Wow it got merged! Congratulations!! |
|
Well done @ddelange @maartenbreddels @EwoutH ; that's a great news for our codes! |
|
Great news! Thanks a lot. @ddelange @maartenbreddels @EwoutH |
|
@Ben-Epstein wake up it's christmas |
|
fyi followup PR #2434 should land soon, so release is imminent:) |
| @@ -1,19 +1,19 @@ | |||
| import os | |||
| import imp | |||
| from importlib.machinery import SourceFileLoader | |||
There was a problem hiding this comment.
@maartenbreddels imp was removed and will fail to import on cp312. so this PR touches all the universal packages to achieve cp312 support.
it might make sense to bump the minimum required versions accordingly in vaex: so cp312 users don't end up with older (broken) versions of the universal packages in their env
There was a problem hiding this comment.
I'm not sure if I should take action here. Can you elaborate?
|
Now that this upgrade process is still fresh in everybody’s mind, would it make sense to directly push to Python 3.13? |
|
I doubt the dependencies (e.g. arrow) are already shipping, but I think we should indeed do this soon! |
|
Hello Vaex team Python 3.12 compatibility is merged (congrats) but it does not seem to be released on Conda, can you confirm? |
|
Hi @erwanp 👋
|
|
sorry to bump this, but currently this breaking colab. I think the pre release version that was yanked was working colab with 3.11 / 3.12 @maartenbreddels |
|
This package looks awesome, and I'd love to use it; unfortunately I can't get it to install, on either Python3.11 or Python3.12 via pip/pypi in a venv. Py3.11 fails on my Linux laptop with: Let me know if I can provide more of the logs, or if you there are specific versions (ideally Py3.11/Py3.12, but I could use something older) known to work together with Linux, it would be great to know! Many thanks for all your work on this, excited to use it. :) |
|
Hoi 👋
linux-aarch64 makes up for almost 10% of all platforms ref giampaolo/psutil#2103
manylinux_2_28).the wheels from this PR can be installed with:
fixes #2366, fixes #2368, fixes #2397, closes #2427, fixes #2384