-
Notifications
You must be signed in to change notification settings - Fork 7.4k
[Data] Failure when converting to Arrow from TensorArray #59445
Copy link
Copy link
Closed
Labels
P0Issues that should be fixed in short orderIssues that should be fixed in short orderdataRay Data-related issuesRay Data-related issues
Description
What happened + What you expected to happen
Context
- Pandas uses custom extension types (like Int64Dtype, etc) to support nullable values.
- Ray Data's
TensorArrayrepresents tensors as Numpy tensors and therefore can't support nullable numeric values - When converting from Pandas
Seriesinto Numpy tensor, it becomes a tensor with dtype="object" - When we try to convert it into Arrow's tensor calling
pyarrow.from_numpy_dtypefails withUnsupported numpy type 17
Versions / Dependencies
2.52
Reproduction script
NA
Issue Severity
None
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
P0Issues that should be fixed in short orderIssues that should be fixed in short orderdataRay Data-related issuesRay Data-related issues