Skip to content

Fix for Parquet v2.0 being deprecated#12300

Merged
nstarman merged 2 commits into
astropy:mainfrom
WilliamJamieson:bugfix/parquet
Oct 27, 2021
Merged

Fix for Parquet v2.0 being deprecated#12300
nstarman merged 2 commits into
astropy:mainfrom
WilliamJamieson:bugfix/parquet

Conversation

@WilliamJamieson

@WilliamJamieson WilliamJamieson commented Oct 26, 2021 •
edited by nstarman
Loading

Copy link
Copy Markdown
Contributor

Description

This is a possible fix for Parquet v2.0 being depreciated.

Fixes #12298

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see "When to rebase and squash commits".
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the Extra CI label.
  • Is a change log needed? If yes, did the change log check pass? If no, add the no-changelog-entry-needed label. If this is a manual backport, use the skip-changelog-checks label unless special changelog handling is necessary.
  • Is a milestone set? Milestone must be set but astropy-bot check might be missing; do not let the green checkmark fool you.
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate backport-X.Y.x label(s) before merge.

@nstarman

nstarman commented Oct 26, 2021

Copy link
Copy Markdown
Member

cc @erykoff for review.
@pllim, who's the appropriate team member for getting @erykoff on astropy-contributors for requesting review in future pyarrow related PRs?

@pllim

pllim commented Oct 26, 2021

Copy link
Copy Markdown
Member

I am not sure. Maybe @astrofrog or @astropy/coordinators ?

@github-actions

github-actions Bot commented Oct 26, 2021

Copy link
Copy Markdown
Contributor

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

@WilliamJamieson WilliamJamieson mentioned this pull request Oct 26, 2021
Closed
@pllim pllim added this to the v5.0 milestone Oct 26, 2021
@pllim pllim added Affects-dev PRs and issues that do not impact an existing Astropy release no-changelog-entry-needed labels Oct 26, 2021
@WilliamJamieson WilliamJamieson marked this pull request as ready for review October 26, 2021 19:10
@nstarman nstarman self-requested a review October 26, 2021 19:11
@astrofrog

astrofrog commented Oct 26, 2021

Copy link
Copy Markdown
Member

To request adding a person to an astropy team open an issue over at https://github.com/astropy/astropy-project - there is a template for requesting GitHub admin action items I think.

@nstarman

nstarman commented Oct 26, 2021

Copy link
Copy Markdown
Member

Thanks @astrofrog
astropy/astropy-project#219

erykoff
erykoff suggested changes Oct 26, 2021
View reviewed changes

@erykoff erykoff left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick PR! I think we need to make sure that the version checking is deferred, and not done on module load (which will slow down import astropy, which nobody wants).

Comment thread astropy/io/misc/parquet.py Outdated
Comment thread astropy/io/misc/parquet.py

# We use version='2.0' for full support of datatypes including uint32.
with parquet.ParquetWriter(output, schema, version='2.0') as writer:
with parquet.ParquetWriter(output, schema, version=writer_version) as writer:

@erykoff erykoff Oct 26, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have we checked that we can write files with pyarrow 6 and read them with pyarrow 5?

@WilliamJamieson WilliamJamieson Oct 26, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not, my work has been focused on trying to solve the CI issues that we are experiencing. Please test this and report your results.

@nstarman nstarman Oct 27, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any conclusion? It would seem odd that schema 2.4 is not compatible with 2.0, but I don't see any documentation about the change.

@nstarman nstarman Oct 27, 2021 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/apache/arrow/pull/11031/files indicates there are genuine differences between 2.0 and 2.4.
But I'm not sure there's really anything we can do about this. Up to users to maintain a stable pyarrow version or to manually specify the version as 2.0 when reading a file with astropy.

@dhomeier dhomeier mentioned this pull request Oct 26, 2021
Merged
9 tasks
@nstarman

nstarman commented Oct 27, 2021

Copy link
Copy Markdown
Member

Thanks @WilliamJamieson and @erykoff for the diagnosis and fix!

@nstarman nstarman merged commit e6d19a9 into astropy:main Oct 27, 2021
@nstarman nstarman mentioned this pull request Oct 27, 2021
Merged
15 tasks
@pllim

pllim commented Oct 27, 2021

Copy link
Copy Markdown
Member

Cool that we got away without version pinning. Thanks!

@pllim pllim mentioned this pull request Oct 27, 2021
Merged
11 tasks
@WilliamJamieson WilliamJamieson deleted the bugfix/parquet branch October 27, 2021 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nstarman nstarman nstarman approved these changes
+1 more reviewer
@erykoff erykoff erykoff requested changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Affects-dev PRs and issues that do not impact an existing Astropy release io.misc no-changelog-entry-needed

Projects

None yet

Milestone

v5.0

Development

Successfully merging this pull request may close these issues.

Parquet v2.0 deprecated

5 participants

@WilliamJamieson @nstarman @pllim @astrofrog @erykoff