-
-
Notifications
You must be signed in to change notification settings - Fork 757
Description
Currently, if a task is already running on a thread of a worker, it will always run to completion/exception even if the work is no longer required (e.g., the future was explicitly cancelled). We usually say "you cannot interrupt a python thread", so Dask has no option here. However, you can set the thread state to exception using ctypes.pythonapi.PyThreadState_SetAsyncExc. Yes, this is a C-api call, but it is public and stable. Is this worth pursuing? It would allow not only early cancellation of unneeded work, but "interrupts" and timeouts from the worker itself, perhaps even better debugging if it can break deadlocks with a useful traceback.
(Note that the setting of an exception only has effect at the next python bytecode instruction, so some long-running C-level function or even wait(big) will not raise immediately)