FYI: I have been debugging using HighLevelGraph.validate() and I have traced must of the bugs to __dask_layers__() not returning layers. For instance, Delayed.__dask_layers__() and Scalar.__dask_layers__() returns low level keys.
I have fixed Scalar but I don't know what we should do with Delayed. See #6507

Good to know, and thank you for organizing things as we proceed down this path. HLGs are relatively new in the codebase and not as well worn as other parts. I hope that this PR helps to expose additional errors. My experience is that placing checks into assert_eq functions is a good way to get a lot of coverage throughout the codebase.

I'm not sure what RAPIDS folks are up to these days, but this seems like a task taht @rjzamora could probably knock out pretty well.

It looks like #6507 fixes all failures besides dask/tests/test_delayed.py::test_keys_from_array, which (I think) is due to the Delayed.__dask_layers__() problem described by @madsbk in that PR. If we do want a dask_layers definitiion for Delayed, perhaps it does make sense to to treat it as a "simple" collection?

If we do want a dask_layers definitiion for Delayed, perhaps it does make sense to to treat it as a "simple" collection?

My guess is that the smallest delta of a change would be to keep Delayed using HighLevelGraphs. Naively I would expect that using the keyname as the layer name would be fine. I'm curious what it isn't working out well.

It also looks like we didn't wire everything up properly in the svd code, which is not surprising, it's one of the more complicated multi-layered functions we have.

Is this still needed now that PR ( #6588 ) is in?

Nope, we're good. Thanks for flagging this @jakirkham