Skip to content

fix: removal of pyarrow.lib.PyExtensionType in pyarrow 21#3581

Merged
ikrommyd merged 5 commits intomainfrom
ikrommyd/pyarrow-21-support
Jul 19, 2025
Merged

fix: removal of pyarrow.lib.PyExtensionType in pyarrow 21#3581
ikrommyd merged 5 commits intomainfrom
ikrommyd/pyarrow-21-support

Conversation

@ikrommyd
Copy link
Copy Markdown
Collaborator

Fixes #3579

@ikrommyd ikrommyd requested review from henryiii and ianna July 18, 2025 19:04
@ikrommyd
Copy link
Copy Markdown
Collaborator Author

ikrommyd commented Jul 18, 2025

@ianna @henryiii Is this the right fix here? PyExtensionType has been removed in pyarrow 21 and the recommendation in their docs is to use ExtensionType. We deal with ExtensionType right below the lines I changed. The original code is

    if isinstance(storage_type, pyarrow.lib.PyExtensionType):
        raise ValueError(
            "Arrow arrays containing pickled Python objects can't be converted into Awkward Arrays"
        )
    elif isinstance(storage_type, pyarrow.lib.ExtensionType):
        ...
)

My idea was to do the first isinstance check only if pyarrow < 21

Copy link
Copy Markdown
Member

@ianna ianna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ikrommyd - thanks for looking into it. I think a better way would be to wrap it in a try block.

try:
    # If PyExtensionType exists and matches, raise immediately
    pyext_type = pyarrow.lib.PyExtensionType  # May trigger AttributeError
    if isinstance(storage_type, pyext_type):
        raise ValueError(
            "Arrow arrays containing pickled Python objects can't be converted into Awkward Arrays"
        )
except AttributeError:
    # PyExtensionType doesn't exist (likely newer PyArrow)
    pass

#  Handle remaining ExtensionType cases
if isinstance(storage_type, pyarrow.lib.ExtensionType):
    assert not isinstance(storage_type, AwkwardArrowType)
    # Logic for generic ExtensionType fallback
    return

@ikrommyd
Copy link
Copy Markdown
Collaborator Author

@ikrommyd - thanks for looking into it. I think a better way would be to wrap it in a try block.

try:
    # If PyExtensionType exists and matches, raise immediately
    pyext_type = pyarrow.lib.PyExtensionType  # May trigger AttributeError
    if isinstance(storage_type, pyext_type):
        raise ValueError(
            "Arrow arrays containing pickled Python objects can't be converted into Awkward Arrays"
        )
except AttributeError:
    # PyExtensionType doesn't exist (likely newer PyArrow)
    pass

#  Handle remaining ExtensionType cases
if isinstance(storage_type, pyarrow.lib.ExtensionType):
    assert not isinstance(storage_type, AwkwardArrowType)
    # Logic for generic ExtensionType fallback
    return

Ah yeah that's possible since an AttributeError cannot come from somewhere else since we're not running any code in the try block.

Copy link
Copy Markdown
Member

@ianna ianna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ikrommyd - thanks for making this change. Please see minor changes to the comments.

@ikrommyd
Copy link
Copy Markdown
Collaborator Author

@ikrommyd - thanks for making this change. Please see minor changes to the comments.

Done.

Copy link
Copy Markdown
Member

@ianna ianna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ikrommyd - this looks great! Thanks for fixing it! Please merge it if you are done with it. Thanks

@ikrommyd ikrommyd merged commit ba736d9 into main Jul 19, 2025
43 checks passed
@ikrommyd ikrommyd deleted the ikrommyd/pyarrow-21-support branch July 19, 2025 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pyarrow.lib.PyExtensionType has been removed in pyarrow 21.0.0

2 participants