Conversation
jrbourbeau
left a comment
There was a problem hiding this comment.
Thanks @rjzamora! Glad to hear the upcoming pyarrow release has fixes : )
Could you update the pyarrow version constraint over in dask/continuous_integration/hdfs/install.sh?
| # Remove <2 version constraint once we've added pyarrow 2.0 compatibility | ||
| # xref https://github.com/dask/dask/issues/6754 | ||
| - pyarrow>=0.14.0,<2 | ||
| - pyarrow>=0.14.0,!=2.0.0 |
There was a problem hiding this comment.
Alternative could be to (temporarily) install the nightly wheel in this dev environment? (https://arrow.apache.org/docs/python/install.html#installing-nightly-packages)
Then we would already have coverage for the new parquet dataset engine with pyarrow > 2.0 as well
There was a problem hiding this comment.
That's a good idea. I'm trying out CI with the nightly version of pyarrow over in #6936
There was a problem hiding this comment.
Another option could also be to skip the specific failing tests for pyrrow 2.0.0, but still install it in some CI environments (then at least we test the other (passing) tests with pyarrow 2.0)
Closes #6754 - Fix for pyarrow 2.0.0 errors will be fixed in the next release, so we will need to avoid 2.0.0
black dask/flake8 dask