Skip to content

Update for pyarrow 21.0.0#8

Merged
pitrou merged 3 commits into
pitrou:mainfrom
AlenkaF:update-for-pyarrow-21
Apr 25, 2025
Merged

Update for pyarrow 21.0.0#8
pitrou merged 3 commits into
pitrou:mainfrom
AlenkaF:update-for-pyarrow-21

Conversation

@AlenkaF

@AlenkaF AlenkaF commented Apr 23, 2025

Copy link
Copy Markdown
Contributor

In PyArrow version 21.0.0 PyExtensionType will be removed, see apache/arrow#46199. We need to update the checks in __init__ so that the hotfix doesn't break with future pyarrow versions.

@AlenkaF AlenkaF mentioned this pull request Apr 23, 2025
Merged
@pitrou

pitrou commented Apr 23, 2025

Copy link
Copy Markdown
Owner

Hi @AlenkaF , two things:

  1. can you merge from main to fix existing CI tests (I had to update them to account for GitHub Actions runner updates)
  2. can you check the hotfix tests run with PyArrow 21?

@AlenkaF AlenkaF force-pushed the update-for-pyarrow-21 branch from 89315f9 to 28b8b25 Compare April 23, 2025 12:22
@AlenkaF

AlenkaF commented Apr 23, 2025

Copy link
Copy Markdown
Contributor Author

can you merge from main to fix existing CI tests (I had to update them to account for GitHub Actions runner updates)

Done.

can you check the hotfix tests run with PyArrow 21?

With the last commit and hacky diff for local dev setup:

diff --git a/tests/test_in_process.py b/tests/test_in_process.py
index 19bf894..0f87759 100644
--- a/tests/test_in_process.py
+++ b/tests/test_in_process.py
@@ -9,7 +9,7 @@ import pyarrow.parquet as pq
 import pyarrow_hotfix
 
 
-pa_version = tuple(map(int, pa.__version__.split('.')))
+pa_version = tuple(map(int, ("21.0.0").split('.')))

the tests passed.

> pytest tests
========================================================================== test session starts ==========================================================================
platform darwin -- Python 3.12.10, pytest-8.3.5, pluggy-1.5.0
rootdir: /Users/alenkafrim/Repos/pyarrow-hotfix
configfile: pyproject.toml
plugins: anyio-4.9.0, hypothesis-6.127.4
collected 14 items                                                                                                                                                      

tests/test_in_process.py .......                                                                                                                                  [ 50%]
tests/test_subprocess.py .......                                                                                                                                  [100%]

========================================================================== 14 passed in 2.00s ===========================================================================

Adding package info, just in case:

>>> pip list
...
pyarrow                       20.0.0.dev327+gb4b3728ff4 /Users/alenkafrim/Repos/arrow/python
pyarrow-hotfix                0.6                       /Users/alenkafrim/Repos/pyarrow-hotfix

@AlenkaF AlenkaF marked this pull request as ready for review April 23, 2025 12:28
@pitrou

pitrou commented Apr 23, 2025

Copy link
Copy Markdown
Owner

It seems this needs to be fixed for PyArrow 0.14, see CI failures.

@AlenkaF

AlenkaF commented Apr 23, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Plan to have a look at it tomorrow/Friday. Thanks!

@AlenkaF

AlenkaF commented Apr 24, 2025

Copy link
Copy Markdown
Contributor Author

@pitrou pinging for CI approval.

raulcd
raulcd approved these changes Apr 25, 2025
View reviewed changes

@raulcd raulcd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @AlenkaF !

@pitrou pitrou merged commit 8d44d72 into pitrou:main Apr 25, 2025
@pitrou

pitrou commented Apr 25, 2025

Copy link
Copy Markdown
Owner

I'm gonna make a new release soon.

@pitrou

pitrou commented Apr 25, 2025

Copy link
Copy Markdown
Owner

The release is now made: https://pypi.org/project/pyarrow-hotfix/

@AlenkaF AlenkaF deleted the update-for-pyarrow-21 branch April 25, 2025 12:48
@raulchen raulchen mentioned this pull request May 1, 2025
Merged
raulchen added a commit to ray-project/ray that referenced this pull request May 6, 2025
@raulchen
upgrade pyarrow-hotfix to 0.7 (#52730)
9aa3f7b 
Upgrade `pyarrow-hotfix` to fix compatibility issue with the latest
pyarrow nightly. See pitrou/pyarrow-hotfix#8

Signed-off-by: Hao Chen <chenh1024@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@raulcd raulcd raulcd approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@AlenkaF @pitrou @raulcd