Skip to content

Allow workers to cancel running task? #4694

@martindurant

Description

@martindurant

Currently, if a task is already running on a thread of a worker, it will always run to completion/exception even if the work is no longer required (e.g., the future was explicitly cancelled). We usually say "you cannot interrupt a python thread", so Dask has no option here. However, you can set the thread state to exception using ctypes.pythonapi.PyThreadState_SetAsyncExc. Yes, this is a C-api call, but it is public and stable. Is this worth pursuing? It would allow not only early cancellation of unneeded work, but "interrupts" and timeouts from the worker itself, perhaps even better debugging if it can break deadlocks with a useful traceback.

(Note that the setting of an exception only has effect at the next python bytecode instruction, so some long-running C-level function or even wait(big) will not raise immediately)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions