-
Notifications
You must be signed in to change notification settings - Fork 4k
Closed
Description
I'm starting to appriciate Arrow more and more, and many new ideas I have with vaex fully align with arrow. I've recently been adding in string support in vaex and I am trying to follow the arrow specs, however have a bit of an issue with the int32 limitation of ListArray
This limits each element in the list to be ~2GB in size (which is not crazy), but also complicates matters when dealing with string concatenation, having to switch to a ChunkedArray. The memory saving by doing 32bit is not huge, and I am not sure the penalty of code complexity is worth it.
I was wondering if the 32bit limitation is set in stone, or is something that could change, and if maybe other people have strong opinions about.
cc @xhochy
Metadata
Metadata
Assignees
Labels
No labels