Skip to content

Clarification of the task graph: are futures allowed in tuples that aren't tasks? #4183

@madsbk

Description

@madsbk

As reported by #4177, custreamz CI fails because the task graph contains tuples with future objects.

The question is if this is even allowed? Dask don't allow keys in tuples that aren't tasks. E.g. in the following ("x",) is interpreted as a literal with no dependencies:

import dask.core
dsk = {
    "x": 1,
    "a": ("x",),
    "b": ["x"]
}
assert dask.core.get_dependencies(dsk, "a") == set()
assert dask.core.get_dependencies(dsk, "b") == {"x"}

But what about futures? Conceptually, futures are remote graph keys so I am inclined to think that they should have the same semantic as regular keys.

What are you thoughts, should the following be allowed?

from distributed import LocalCluster, Client

if __name__ == '__main__':
    client = Client(LocalCluster(n_workers=1))

    def getfirst(x):
        return x[0]

    x = client.submit(sum, [1,2])
    y = client.submit(getfirst, (x,))
    y.result()

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions