[REVIEW] Reduce code duplication for dask & distributed nightly/stable installs#11565
Conversation
dask & distributed nightly/stable based on flag
dask & distributed nightly/stable based on flagdask & distributed nightly/stable based on INSTALL_DASK_MAIN
Dask-cudf groupby tests *should* be failing as a result of dask/dask#9302 (see [failures](https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cudf/job/prb/job/cudf-gpu-test/CUDA=11.5,GPU_LABEL=driver-495,LINUX_VER=ubuntu20.04,PYTHON=3.9/9946/) in #11565 is merged - where dask/main is being installed correctly). This PR updates the dask_cudf groupby code to fix these failures. Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: #11561
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## branch-22.10 #11565 +/- ##
===============================================
Coverage ? 87.51%
===============================================
Files ? 133
Lines ? 21798
Branches ? 0
===============================================
Hits ? 19077
Misses ? 2721
Partials ? 0 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
|
Thanks for taking care of this @galipremsagar ! |
dask & distributed nightly/stable based on INSTALL_DASK_MAINdask & distributed nightly/stable correctly
ci/benchmark/build.sh
Outdated
| export INSTALL_DASK_MAIN=1 | ||
|
|
||
| # Dask version to install when `INSTALL_DASK_MAIN=0` | ||
| export DASK_STABLE_VERSION="2022.8.0" |
There was a problem hiding this comment.
Want to note that the relevant fixes for the mixed stable/nightly issue here are in conda-forge/dask-feedstock#191 and conda-forge/distributed-feedstock#218, since our problem is that when installing conda-forge dask we ended up pulling in nightly dask-core/distributed.
With this context, conda-forge/conda-forge-repodata-patches-feedstock#312 applies this change across all stable dask/distributed packages that would've had this issue, so once that's in we shouldn't have to bump the stable version here
There was a problem hiding this comment.
Awesome, I'll hold off until conda-forge/conda-forge-repodata-patches-feedstock#312 is merged and will revert back to the prev stable version that we were pointing to as a minimum.
Co-authored-by: Charles Blackmon-Luca <20627856+charlesbluca@users.noreply.github.com>
charlesbluca
left a comment
There was a problem hiding this comment.
A couple suggestions so we can verify that the new stable install command works:
|
After doing some more local testing of conda-forge/conda-forge-repodata-patches-feedstock#312 I realize there's still issues that need to be resolved 🤦🏽 |
|
This PR has been labeled |
Co-authored-by: Charles Blackmon-Luca <20627856+charlesbluca@users.noreply.github.com>
|
@charlesbluca could you review this PR? I know this is not solving the original problem we were trying to address and it has to be done at dask side, but this does remove a lot of version number duplication and will be helpful while we pin & unpin for release. |
dask & distributed nightly/stable correctlydask & distributed nightly/stable installs
Co-authored-by: Charles Blackmon-Luca <20627856+charlesbluca@users.noreply.github.com>
Co-authored-by: Charles Blackmon-Luca <20627856+charlesbluca@users.noreply.github.com>
|
@gpucibot merge |
Description
After dask/dask#9367 was fixed in dask upstream we had to bump the minimum version of dask to 2022.8.0 to correctly fetch nightly(if channel exists) or stable (if
dask/devlabel doesn't exist). Without this fix, conda builds were always picking up2022.7.1only and/or there would be a mix of nightly & stable packages in an env.This PR also does some cleanup and makes the
build.shscript easy to maintain.Checklist