Skip to content

PARQUET-2414: Add test file for additional BYTE_STREAM_SPLIT types#46

Merged
pitrou merged 2 commits intoapache:masterfrom
pitrou:bss-extended-file
Mar 18, 2024
Merged

PARQUET-2414: Add test file for additional BYTE_STREAM_SPLIT types#46
pitrou merged 2 commits intoapache:masterfrom
pitrou:bss-extended-file

Conversation

@pitrou
Copy link
Copy Markdown
Member

@pitrou pitrou commented Feb 19, 2024

Add a new data file that allows exercising BYTE_STREAM_SPLIT for all supported types: FLOAT, DOUBLE, INT32, INT64, FIXED_LEN_BYTE_ARRAY (the latter with several widths and logical types).

For each type, two columns are provided with the same values: one PLAIN-encoded, the other BYTE_STREAM_SPLIT-encoded.

Add a new data file that allows exercising BYTE_STREAM_SPLIT for all supported types:
FLOAT, DOUBLE, INT32, INT64, FIXED_LEN_BYTE_ARRAY (the latter with several widths and logical types).

For each type, two columns are provided with the same values: one PLAIN-encoded, the other BYTE_STREAM_SPLIT-encoded.
@pitrou
Copy link
Copy Markdown
Member Author

pitrou commented Feb 19, 2024

This should not be merged until PARQUET-2414 is voted on.

@pitrou pitrou requested a review from wgtmac February 20, 2024 10:16
It contains 7 pairs of columns, each in two variants containing the same
values: one `PLAIN`-encoded and one `BYTE_STREAM_SPLIT`-encoded:
```
Version: 2.6
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm... it seems we need to fix the version so here can correctly displays version 2.11

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That will have to be part of the final C++ PR. We do not need the C++ changes to fix this README if we want to, though.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(also see apache/arrow#40096)

@pitrou pitrou changed the title [DO NOT MERGE] PARQUET-2414: Add test file for additional BYTE_STREAM_SPLIT types PARQUET-2414: Add test file for additional BYTE_STREAM_SPLIT types Mar 18, 2024
@pitrou pitrou merged commit 74278bc into apache:master Mar 18, 2024
@pitrou pitrou deleted the bss-extended-file branch March 18, 2024 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants