Errors being caused by allowing conda-forge & dask/label/dev packages to co-exist #9367
Description
What happened:
Looks like the way currently dask, dask-core & distributed are being packaged in conda-forge & dask/label/dev to be able to use interchangeably is leading to issues when being used with dask-cuda-worker.
What you expected to happen:
Ideally, I'd expect dask/label/dev packages to not be compatible with conda-forge packages, and vice-versa.
Minimal Complete Verifiable Example:
sample_error.yml.txt
Error scenario:
(myenv) pgali@dt07:/nvme/0/pgali/cudf$ mamba env create --prefix /nvme/0/pgali/envs/cudfdev --file sample_error.yml
(myenv) pgali@dt07:/nvme/0/pgali/cudf$ conda activate /nvme/0/pgali/envs/cudfdev
(cudfdev) pgali@dt07:/nvme/0/pgali/cudf$ conda list | grep "dask"
dask 2022.7.1 pyhd8ed1ab_0 conda-forge
dask-core 2022.7.1a220722 py_gedfa91e22_16 dask/label/dev
dask-cuda 22.08.00a220808 py39_g9f46a1a_35 rapidsai-nightly
(cudfdev) pgali@dt07:/nvme/0/pgali/cudf$ conda list | grep "distributed"
distributed 2022.7.1 pyhd8ed1ab_0 conda-forge
(cudfdev) pgali@dt07:/nvme/0/pgali/cudf$ ipython
iPython 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:56:21)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import dask
In [2]: dask.__version__
Out[2]: '2022.7.0+16.gedfa91e22'
(cudfdev) pgali@dt07:/nvme/0/pgali/cudf$ dask-cuda-worker
Traceback (most recent call last):
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 581, in _build_master
ws.require(__requires__)
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 909, in require
needed = self.resolve(parse_requirements(requirements))
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 800, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (dask 2022.7.0+16.gedfa91e22 (/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages), Requirement.parse('dask==2022.7.1'), {'dask-cuda', 'distributed'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/nvme/0/pgali/envs/cudfdev/bin/dask-cuda-worker", line 33, in <module>
sys.exit(load_entry_point('dask-cuda==22.8.0a0', 'console_scripts', 'dask-cuda-worker')())
File "/nvme/0/pgali/envs/cudfdev/bin/dask-cuda-worker", line 25, in importlib_load_entry_point
return next(matches).load()
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/importlib/metadata.py", line 86, in load
module = import_module(match.group('module'))
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/dask_cuda/__init__.py", line 12, in <module>
from .cuda_worker import CUDAWorker
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/dask_cuda/cuda_worker.py", line 22, in <module>
from .initialize import initialize
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/dask_cuda/initialize.py", line 5, in <module>
import numba.cuda
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/numba/__init__.py", line 19, in <module>
from numba.core import config
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/numba/core/config.py", line 15, in <module>
import llvmlite.binding as ll
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/llvmlite/binding/__init__.py", line 4, in <module>
from .dylib import *
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/llvmlite/binding/dylib.py", line 3, in <module>
from llvmlite.binding import ffi
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/llvmlite/binding/ffi.py", line 175, in <module>
from pkg_resources import resource_filename
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3260, in <module>
def _initialize_master_working_set():
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3234, in _call_aside
f(*args, **kwargs)
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3272, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 583, in _build_master
return cls._build_from_requirements(__requires__)
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 596, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/nvme/0/pgali/envs/cudfdev/lib/python3.9/site-packages/pkg_resources/__init__.py", line 795, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'dask==2022.7.1' distribution was not found and is required by dask-cuda, distributed
Working scenario when dask/label/dev is not present in the environment:
sample_working.yml.txt
(myenv) pgali@dt07:/nvme/0/pgali/cudf$ mamba env create --prefix /nvme/0/pgali/envs/cudfdev --file sample_working.yml
(myenv) pgali@dt07:/nvme/0/pgali/cudf$ conda activate /nvme/0/pgali/envs/cudfdev
(cudfdev) pgali@dt07:/nvme/0/pgali/cudf$ dask-cuda-worker
# Works fine without any errors
(cudfdev) pgali@dt07:/nvme/0/pgali/cudf$ conda list | grep "dask"
dask 2022.7.1 pyhd8ed1ab_0 conda-forge
dask-core 2022.7.1 pyhd8ed1ab_0 conda-forge
dask-cuda 22.08.00a220808 py39_g9f46a1a_35 rapidsai-nightly
(cudfdev) pgali@dt07:/nvme/0/pgali/cudf$ conda list | grep "distributed"
distributed 2022.7.1 pyhd8ed1ab_0 conda-forge
(cudfdev) pgali@dt07:/nvme/0/pgali/cudf$ ipython
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import dask
In [2]: dask.__version__
Out[2]: '2022.7.1'