dask worker: daemonic processes are not allowed to have children #2142
Description
I have a package that I have written that used multiprocessing and some complex communication with multiprocessing.Pipe. I run into an issue when trying to using multiprocessing within dask workers. Is there any workaround that I could do? Is there a way to run a dask worker not as a daemon?
Minimum working example.
dask-schedulerdask-worker <scheduler-ip:scheduler-port>def test_process():
import multiprocessing
def adder(a, b):
return a + b
p = multiprocessing.Process(target=adder, args=(1, 3))
p.start()
p.join()
if __name__ == "__main__":
from dask.distributed import Client
client = Client('<scheduler-ip:scheduler-port>')
f = client.submit(test_process)
f.result()Error message
distributed.worker - WARNING - Compute Failed
Function: test_process
args: ()
kwargs: {}
Exception: AssertionError('daemonic processes are not allowed to have children',)
Traceback (most recent call last):
File "test.py", line 16, in <module>
f.result()
File "/home/costrouc/.virtualenvs/dask-6qByegfR/lib/python3.6/site-packages/distributed/client.py", line 192, in result
six.reraise(*result)
File "/home/costrouc/.virtualenvs/dask-6qByegfR/lib/python3.6/site-packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "test.py", line 8, in test_process
p.start()
File "/usr/lib/python3.6/multiprocessing/process.py", line 103, in start
'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children