CI: test pyarrow 2.0 and nightly#7030
Conversation
.github/workflows/ci-upstream.yml
Outdated
| build: | ||
| runs-on: "ubuntu-latest" | ||
| if: "contains(github.event.head_commit.message, 'test-upstream') || (github.repository == 'dask/dask' && github.event_name != 'pull_request')" | ||
| # if: "contains(github.event.head_commit.message, 'test-upstream') || (github.repository == 'dask/dask' && github.event_name != 'pull_request')" |
There was a problem hiding this comment.
TODO: undo this change at the end
.github/workflows/ci.yml
Outdated
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| fail-fast: true | ||
| fail-fast: false |
There was a problem hiding this comment.
TODO: undo this change at the end
| - tiledb-py | ||
| - sqlalchemy | ||
| - pyarrow>=0.14.0,!=2.0.0 | ||
| - pyarrow=0.14.* |
There was a problem hiding this comment.
I pinned this to 0.14, because that is what in practice for all platforms gets used (and by putting it in the env file, it's a bit easier to see this)
| - fsspec>=0.6.0 | ||
| - sqlalchemy | ||
| - pyarrow>=0.14.0,!=2.0.0 | ||
| - pyarrow=1.0.* |
There was a problem hiding this comment.
same here, this is the version that got used in practice
| # For the dependencies. These will be re-installed later. | ||
| - numpy | ||
| - pandas | ||
| - pandas=1.0.* |
There was a problem hiding this comment.
pinned here to older pandas, since otherwise I get the failures that are being handled by #6996
| - fsspec>=0.6.0 | ||
| - sqlalchemy | ||
| - pyarrow>=0.14.0,!=2.0.0 | ||
| - pyarrow>=0.14.0 |
There was a problem hiding this comment.
Here just leave the "larger than" constraint, to get the last released version of pyarrow, which at the moment is 2.0.0 (after removing the !=2.0.0 pin)
setup.cfg
Outdated
| # error:::dask[.*] | ||
| # error:::pandas[.*] | ||
| # error:::numpy[.*] | ||
| # error:::distributed[.*]] |
There was a problem hiding this comment.
TODO: undo this change in the end
|
cc @jsignell apart from the graphviz edit, and some edits I need to do undo at the end (if all is OK otherwise), I think this should be ready In the upstream build, remaining failure is a |
|
Great!
Yeah I think I'd done some fixes around that one in the past. Let |
|
OK, everything seems to be passing here (except the 1 sparse failure), will revert the temporary edits |
| # conda uninstall --force numpy pandas | ||
| # python -m pip install --no-deps --pre \ | ||
| # -i https://pypi.anaconda.org/scipy-wheels-nightly/simple \ | ||
| # numpy | ||
| # python -m pip install --pre pandas==1.1.0rc0 |
There was a problem hiding this comment.
Apologies for not catching this earlier, but I think we want to keep this for testing nightly versions of numpy and pandas
Though I just noticed we're pinning pandas==1.1.0rc0 which might be an oversight. Instead I would have expected us to install both numpy and pandas from https://pypi.anaconda.org/scipy-wheels-nightly/simple
There was a problem hiding this comment.
Yes, we certainly want to keep that, but the thing is that they were currently failing a lot, and thus not very useful (hard to see what errors still need to be fixed for a given dependency). The pandas ones are being fixed in #6996 (and the pandas nightly install is added back there), and the numpy ones in #6896 I think
* CI: test pyarrow 2.0 and nightly * temp disable warning->error * try to force install pyarrow nightly * try conda update * temp pin graphviz * add skip for pyarrow test with pyarrow==2.0.0 on windows * also skip pyarrow-dataset engine on windows * clean-up / undo temp edits
Alternative to #6936 (cc @jrbourbeau): since there is an upstream-dev build, we can actually install pyarrow nightly in there (instead of only doing it in a test PR as #6936 does).
Additionally, I also removed the
!=2.0.0pin (which will require some skips in the tests (TODO), but I think it's still worth the run the rest of the tests with the latest released pyarrow version).