ARROW-3919: [Python] Support 64 bit indices for pyarrow.serialize and pyarrow.deserialize#3062
Closed
pcmoritz wants to merge 9 commits intoapache:masterfrom
Closed
ARROW-3919: [Python] Support 64 bit indices for pyarrow.serialize and pyarrow.deserialize#3062pcmoritz wants to merge 9 commits intoapache:masterfrom
pcmoritz wants to merge 9 commits intoapache:masterfrom
Conversation
Member
|
@pcmoritz can you rebase? would you like me to review this? |
e0036b7 to
a1a86b6
Compare
Contributor
Author
|
@wesm This is not done yet, but do you want to take a brief look if that's the direction we want to go in (make ListArray offsets int64_t)? |
Contributor
Author
|
After looking into this a bit more, the alternative route would be to use ChunkedArrays and the new ChunkedBuilders etc. Which wouldn't blow up everybody's ListArray offsets by 2x. We would need a ChunkedUnionArrayBuilder to implement this I think. |
Member
|
@pcmoritz this isn't consistent with the columnar format specification (list offsets are 32 bits). We'll have to find another way |
Member
|
Closing per discussions on mailing list |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.