Skip to content

Document dev process around high level graphs #7755

@mrocklin

Description

@mrocklin

We would like to enable other people to write high level layers. For example some of the imaging crowd is trying to accomplish this for Map Overlap.

Currently it's not clear what contract needs to be filled to do this well, particularly around efficient culling (it is common in imaging to construct very large dask arrays and then slide very small pieces out of them, so there are great potentials for benefit here). The current cull method has a brief docstring on what is needed, but it's not super clear. In particular, I think that it would be helpful to explain that the result Layer should probably have changed state so that it only generates the correct keys. Also, we should define and give examples for the resulting deps mapping. The current doctest example doesn't include this, and it's not clear what the expected format is.

cc @madsbk @rjzamora if either of you have the time.

@GenevieveBuckley is working on map-overlap and is somewhat blocked on trying to follow the high level graph layer flow

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprove or add to documentationhighlevelgraphIssues relating to HighLevelGraphs.needs attentionIt's been a while since this was pushed on. Needs attention from the owner or a maintainer.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions