Skip to content

Add numpy2 support#2449

Merged
maartenbreddels merged 48 commits intovaexio:masterfrom
2maz:numpy-v2
Aug 19, 2025
Merged

Add numpy2 support#2449
maartenbreddels merged 48 commits intovaexio:masterfrom
2maz:numpy-v2

Conversation

@2maz
Copy link
Contributor

@2maz 2maz commented Jan 16, 2025

This pull request intends to enable the use of numpy2 (#2446 #2425) with vaex.
In addition it applies some other fixes.

An (incomplete) summary:

Note, that I encountered some broken test(s) in cmodule.py test_interpolate and those related to Webserver:

  • this looked like it was in a broken state already, so left it for now
    Some format changes from tab to space indent have been applied.

Please let me know if (further) changes are required - only validated on Linux system right now.
I ran tests in vaex-core/vaex/test - any more instructions on how to run you full test-suite (apart from ci)?

Thx.

@maartenbreddels @ddelange

@2maz
Copy link
Contributor Author

2maz commented Jan 17, 2025

Copy link
Contributor

@ddelange ddelange left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome, thanks for the quick action!

@setu4993
Copy link

setu4993 commented Feb 6, 2025

This looks great!

@maartenbreddels : Thoughts on getting this in?

@ddelange
Copy link
Contributor

@maartenbreddels kind reminder:)

@setu4993
Copy link

setu4993 commented Mar 6, 2025

@maartenbreddels : Realize there's a lot going on but would really be helpful if you could help with getting this in. It closes a bunch of open issues and vaex-core is still blocked on Python 3.10+.

@ddelange
Copy link
Contributor

@maartenbreddels kind reminder :)

@ddelange
Copy link
Contributor

@maartenbreddels kind reminder here. I've reviewed and this looks good to me.

@ddelange
Copy link
Contributor

@maartenbreddels can you please merge this PR?

@maartenbreddels maartenbreddels merged commit 17e9763 into vaexio:master Aug 19, 2025
43 checks passed
@maartenbreddels
Copy link
Member

Looks good! Lets see if it's still green when merged :)

@ddelange
Copy link
Contributor

ddelange commented Aug 20, 2025

looks like only some rot in test_catboost.py

FAILED tests/ml/catboost_test.py::test_catboost_validation_set - AttributeError: 'CatBoostModel' object has no attribute 'fit'

801 commits since the last release... including numpy v2 support!

@ddelange
Copy link
Contributor

ddelange commented Aug 20, 2025

@2maz can 25cf41a be reverted?

now that catboost 1.2.8 has been released, vaex's ci is allowed to install numpy v2 (catboost/catboost@b6fd136) hence causing the crash as per 25cf41a

@2maz
Copy link
Contributor Author

2maz commented Aug 21, 2025

@ddelange yes, we can now operate with catboost>=1.2.8
There are, however, some other issues as well with the pipeline which i am currently looking at: master...2maz:vaex:master

One new windows issue seems to have appeared https://github.com/2maz/vaex/actions/runs/17110604988/job/48530848114#step:11:880

@ddelange
Copy link
Contributor

awesome, thanks for looking into this!

super interesting that such a simple sum can consistently give a wrong result on windows only, especially because it was green before 🤔

@2maz
Copy link
Contributor Author

2maz commented Aug 22, 2025

super interesting that such a simple sum can consistently give a wrong result on windows only, especially because it was green before

FYI: this particular one turned out to be a pyarrow 21.0.0 issue in fill_null - until fixed we need to restrict to pyarrow<21.0.0 for windows

@minouHub
Copy link

Thank you so much for solving this issue and merging this guys!

@2maz
Copy link
Contributor Author

2maz commented Aug 26, 2025

Final touches in #2463 @ddelange @maartenbreddels

@ddelange
Copy link
Contributor

super interesting that such a simple sum can consistently give a wrong result on windows only, especially because it was green before

FYI: this particular one turned out to be a pyarrow 21.0.0 issue in fill_null - until fixed we need to restrict to pyarrow<21.0.0 for windows

update: this will be fixed in arrow v22.

Dev-iL added a commit to SummitSG-LLC/hamilton that referenced this pull request Feb 14, 2026
vaex - vaexio/vaex#2449
spark - supports numpy 2 as of version 4.0.0
Dev-iL added a commit to SummitSG-LLC/hamilton that referenced this pull request Feb 14, 2026
vaex - vaexio/vaex#2449
spark - supports numpy 2 as of version 4.0.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants