Since __dask_distributed_unpack__() is a classmethod, it can be moved out of the class and isolated in its own module -- it would ugly but work in this case. However, the implementation of __dask_distributed_unpack__() might itself import external modules, which would trigger the problem again.

After a quick look through the existing problem areas, it seems like we will need to move all __dask_distributed_unpack__ definitions (and releated graph-materialization functions) into a dedicated layers.py file (or something) within each module. We will also need to revise the graph-materialization functions to avoid the use of utilities that rely on "superfluous" dependencies (numpy, pandas, etc.). One way to make this easier would be to move all "light-weight" utilites into a light_utils.py file (or something) for each module. This approach would produce an organization like:

dask/
 |
 |— array/

 |      |— layers.py
 |      |— light_utils.py
 |
 |— dataframe/
 |      |— layers.py
 |      |— light_utils.py
 |
 |— layers.py
 |— light_utils.py

EDIT: It seems that the above directory structure will not work, since importing the dask.dataframe.layers module (for example) also imports the dask.dataframe imports.

@jrbourbeau - I think you suggested the use of a full HLG/Layer directory structure last week (in an offline discussion). I think that ideam may be the best way forward. Perhaps we can create a highlevelgraph/ directory, and use that umbrella to store any function/class that may need to run on the scheduler? For example:

dask/
 |— array/
 |      |— # API and modules requiring typical Array imports
 |— dataframe/
 |      |— # API and modules requiring typical DataFrame imports
 |— highlevelgraph/
 |      |— utils.py # Lightweight utils
 |      |— blockwise.py
 |      |— highlevelgraph.py
 |      |— array/
 |      |      |— utils.py # Lightweight array-focused utils
 |      |      |— shuffle.py
 |      |      |— …
 |      |— dataframe/
 |      |      |— utils.py # Lightweight dataframe-focused utils
 |      |      |— …

Thinking about it a bit more I wonder if starting out with a single layers.py would work. If that's too unruly then the structure you outlined seems sensible to me.

Additionally, whatever module structure we go with, we should make use of our CI import build

to ensure that we can always import, for example dask.layers, without any additional dependencies

Thinking about it a bit more I wonder if starting out with a single layers.py would work. If that's too unruly then the structure you outlined seems sensible to me.

#7381 is going in this direction for shuffle, but that PR is also attempting to minimize the amount of necessary code to include in layers.py (by effectively splitting the Layer definitions a into the unpack/materialization code and "everything else"). Do you think that is a reasonable approach, or should we plan to move entire Layer subclass definitions into layers.py? The latter approach will keep the definitions in one place (which is nice), but it will limit the kind of logic we can do (directly) in __init__, cull and __dask_distributed_pack__.

Let's include this test over in #7381 -- thanks @rjzamora!