Change default parquet statistics truncation to be 64 bytes #7578
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mbrobbel
approved these changes
Jun 2, 2025
Contributor
|
@alamb would you like me to start merging the deferred to 56.0.0 PRs, or do you prefer to do that yourself? |
Contributor
Author
Please do! |
…ics_truncate_length
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
max_statistics_truncate_length#7490Rationale for this change
Statistics for large columns (e.g. large strings) are typically not useful for min/max value pruning.
However, the current defaults in parquet-rs will store the entire min and max value.
For large binary/string columns (think JSON blobs), this means that two (a min and a max) potentially large values will be stored in both the file level metadata as well as in each data page header
What changes are included in this PR?
Change default statistics truncation size to be 64 to match the default for truncating PageIndex statistics
Are there any user-facing changes?
This is a user facing change -- I expect users will see:
It is an API change, so we should wait to merge this until the next major release