Skip to content

feat(parquet): Add Int64 Timestamp in reader#11530

Closed
zuyu wants to merge 1 commit intofacebookincubator:mainfrom
zuyu:ts-int64
Closed

feat(parquet): Add Int64 Timestamp in reader#11530
zuyu wants to merge 1 commit intofacebookincubator:mainfrom
zuyu:ts-int64

Conversation

@zuyu
Copy link
Copy Markdown
Contributor

@zuyu zuyu commented Nov 13, 2024

w/ both plain and dictionary encoding support.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 13, 2024
@netlify
Copy link
Copy Markdown

netlify Bot commented Nov 13, 2024

Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit 06f1ed1
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/67645b8eb0339100083dcd8c

Comment thread velox/dwio/common/IntDecoder.h Outdated
@zuyu zuyu changed the title Add Int64 Timestamp support in Parquet reader <feat>(parquet): Add Int64 Timestamp support in Parquet reader Nov 14, 2024
@zuyu zuyu changed the title <feat>(parquet): Add Int64 Timestamp support in Parquet reader <feat>(parquet): Add Int64 Timestamp in reader Nov 14, 2024
@zuyu zuyu changed the title <feat>(parquet): Add Int64 Timestamp in reader feat(parquet): Add Int64 Timestamp in reader Nov 14, 2024
@liujiayi771
Copy link
Copy Markdown
Contributor

Could you add some unit tests?

Copy link
Copy Markdown
Contributor

@yingsu00 yingsu00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zuyu Can you please add a commit and PR message describing the change?

Comment thread velox/dwio/common/IntDecoder.h Outdated
Comment thread velox/dwio/common/IntDecoder.h Outdated
Comment thread velox/dwio/parquet/reader/TimestampColumnReader.h Outdated
@surnaik
Copy link
Copy Markdown

surnaik commented Nov 20, 2024

I am assuming you're also planning to add support for Dictionary encoding and range filters, currently the range filter is always casted to INT96 range filter.

@majetideepak
Copy link
Copy Markdown
Collaborator

@rui-mo, @surnaik can you help review this change? Thanks.

Copy link
Copy Markdown
Contributor

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I added some comments.

Comment thread velox/dwio/parquet/reader/PageReader.cpp Outdated
Comment thread velox/dwio/parquet/reader/TimestampColumnReader.h Outdated
Comment thread velox/dwio/parquet/reader/TimestampColumnReader.h Outdated
@zuyu zuyu requested a review from rui-mo December 17, 2024 07:44
@zuyu
Copy link
Copy Markdown
Contributor Author

zuyu commented Dec 17, 2024

@rui-mo Addressed ur comments. Could u take another look? The test failures are the same as the main branch. Ty!

Copy link
Copy Markdown
Contributor

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Added some nits and overall looks good to me.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in nano precision

nit: is other precision other than nano possible?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, other precisions handle separately: see #11600 (comment)

Comment thread velox/dwio/parquet/reader/TimestampColumnReader.h Outdated
Comment thread velox/dwio/parquet/reader/TimestampColumnReader.h Outdated
Comment thread velox/dwio/parquet/reader/TimestampColumnReader.h Outdated
@facebook-github-bot
Copy link
Copy Markdown
Contributor

@zacw7 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

@zacw7 merged this pull request in c9dee4e.

@zuyu zuyu deleted the ts-int64 branch December 20, 2024 01:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants